Virtual and Augmented Reality Analytics Digility

Virtual and AugmentedVirtual and Augmented

Reality AnalyticsReality Analytics

Virtual and AugmentedVirtual and Augmented

Reality AnalyticsReality Analytics

Frog #84027132 © julien tromeur – fotolia.com - extended licenseFrog #84027132 © julien tromeur – fotolia.com - extended licensetransformed into VR Frog by Joerg Osarektransformed into VR Frog by Joerg Osarek

Joerg OsarekJoerg OsarekIT-avant-gardist, Business-IT-ArchitectIT-avant-gardist, Business-IT-Architect

Skilltower InstituteSkilltower Institute

Partner, its-peoplePartner, its-people

DigilityDigilitySeptember 2016September 2016

Cologne, GermanyCologne, Germany

© Skilltower Institute, 20162

Intro: Joerg Osarek – vr.skilltower.com

ManagementManagement

IT-avant-gardeIT-avant-garde

Virtualization (VMware)Virtualization (VMware)

Oracle, LinuxOracle, Linux

Virtual Reality AnalyticsVirtual Reality Analytics

Cinematic VRCinematic VR

VR BusinessVR Business

Co-founder:Co-founder:www.its-people.dewww.its-people.de


Reports from the future – my passion ever since

Always some years ahead of Mainstream (Years Ahead of Mainstream = YAM)

1989: HP: Computer Viruses: 10 YAM

1998: DOAG: VRML – The future of the Internet: 9 YAM (2ndLife 2007)

2007: DOAG: How does SOA influence Business Intelligence?: 3 YAM

2010: DOAG: Cloud Computing and the Technology-Monster: 2 YAM

2011: DOAG: The Organic Computing Age: 9 YAM?

2012: DOAG: SoMoLo (Social, Mobile, Local) / Big Data: 3 YAM

2015: DOAG: Virtual Reality Analytics: 3-5 YAM?


Reports from the future – my passion ever since

1998: DOAG: VRML – The futureof the internet

Many thanks to Wolfgang Taschnerfor his archived copy


VR/AR Analytics

How VR and AR change business intelligence and how you can integrate VR/AR Analytics in your BI-systems and data warehouses

I. Introduction into VR/AR today and tomorrow

II. VR/AR Analytics: Your new business intelligence gold mine

II.a Integrate VR/AR Sources into your Big Data Systems

II.b Chances and Challenges with VR/AR Analytics

III. The 3 waves – from VR to AR to USEMIR: What future awaits us?


I. Introduction into VR/AR today and tomorrow


Virtual Reality: The power of Immersion

VR puts tough guys under a spell unlike any other medium.Reactions are ranging

from delight to pure horror

Try it yourselfor visit our VR Meetup in Frankfurt.


Display Technologie

2015: Samsung Gear, Oculus DK2, 2015: Samsung Gear, Oculus DK2, Mobile VR, Google Cardboard, +++Mobile VR, Google Cardboard, +++

2016: Oculus Rift Consumer Version2016: Oculus Rift Consumer VersionPlaystation VR (Morpheus)Playstation VR (Morpheus)HTC Vive, FOVE (Eye Tracking)HTC Vive, FOVE (Eye Tracking)Microsoft Hololens für EntwicklerMicrosoft Hololens für Entwickler

What is happening at VR / MR? - VR is not just headsets!


Display Technology



FulldomesFulldomes

Leia HolodisplayLeia Holodisplay

Magic LeapMagic Leap



Display Technology Recording



FulldomesFulldomes



Google Jump Google Jump + Youtube VR+ Youtube VR

JauntVR NeoJauntVR NeoLightfield CamLightfield Cam

Lytro ImmergeLytro Immerge



Display Technology Recording

Interaction 2015: Samsung Gear, Oculus DK2, 2015: Samsung Gear, Oculus DK2,

Mobile VR, Google Cardboard, +++Mobile VR, Google Cardboard, +++


FulldomesFulldomes



Google Jump Google Jump + Youtube VR+ Youtube VR

JauntVR NeoJauntVR NeoLightfield CamLightfield Cam

LeapMotionLeapMotion

Fairy LightsFairy LightsHologrammHologrammzum Anfassenzum Anfassen

VR SculptingVR Sculpting

Google SoliGoogle SoliRadar ChipRadar Chip

Disney DISHDisney DISH

ILM X-LabsILM X-Labs

Lytro ImmergeLytro Immerge



What is happening next?

Voice Mood Analysis (e.g. cogito App)

Brain Wave interfaces (e.g. NeuroSky Neuro Headset)

Nerve Induction? (e.g. direct projection into retina or nerves)

Mark Zuckerberg: „we'll be able to send emotions to each other soon“*

* Article at theverge.com from Rich McCormick on June 30, 2015.* http://www.theverge.com/2015/6/30/8873819/mark-zuckerberg-facebook-sending-emotions-soon



Quelle: Woman and pan: combined of two public domain images from www.pixabay.com

Mark Zuckerberg: „we'll be able to send emotions to each other soon“… could look like this:



Quelle: Woman and pan: combined of two public domain images from www.pixabay.com , Quelle: Angry driver: Vladimir Mucibabic/123rf.com #4554701_l

Mark Zuckerberg: „we'll be able to send emotions to each other soon“… could look like this:

LIKE ?LIKE ?


II. VR/AR Analytics: Your new business intelligence gold mine


VR Analytics comes in two flavours

a) Visualization of analytics data in VR


VR Analytics comes in two flavours

a) Visualization of analytics data in VR

We don't cover this today.

b) Analyze VR experiences in your DWH


Let's Play


II.a Integrate VR/AR Sources into your Big Data Systems


Level 1: Your Mission

BUILD ME THE PERFECTVR ANALYTICS PLATFORM.

MUHAHAHA...


Level 2: Make or buy? - Status VR Analytics vendorsLevel 2: Make or buy? - Status VR Analytics vendors cognitiveVRcognitiveVR: : http://cognitivevr.co/http://cognitivevr.co/ Analytics platform and plugins for Unity and Unreal Game Engines Analytics platform and plugins for Unity and Unreal Game Engines

retinadVR.comretinadVR.com (Unity Plugin) → joined (Unity Plugin) → joined rothenbergventures.comrothenbergventures.com (investing in frontier tech startups)(investing in frontier tech startups)

http://ghostline.xyz/http://ghostline.xyz/ VR Analytics tool PSVR, Vive, Oculus (pc based) VR Analytics tool PSVR, Vive, Oculus (pc based)

http://www.viarbox.com/2016/05/02/virtual_reality_analytics/http://www.viarbox.com/2016/05/02/virtual_reality_analytics/ - end to end Platform - end to end Platform

http://www.finwe.mobi/main/liveheat/http://www.finwe.mobi/main/liveheat/ - 360 video heat maps - 360 video heat maps

Election.tvElection.tv Engages Kosher Virtual Reality Analytics For Exclusive 2016 US Election Tech Polling Engages Kosher Virtual Reality Analytics For Exclusive 2016 US Election Tech Polling Partnership (Partnership (Kosher.tvKosher.tv, a media ventures company), a media ventures company)

Some BI / DWH experts start to broaden their portfolio Some BI / DWH experts start to broaden their portfolio (like (like its-peopleits-people and and skilltower instituteskilltower institute))

It is quite early for the large BI / DWH software vendors but we should watchIt is quite early for the large BI / DWH software vendors but we should watch trendsetterstrendsetters like: like: IBM, SAP, Qlik, Oracle, SAS, Information Builders, IBM, SAP, Qlik, Oracle, SAS, Information Builders, MicrostrategyMicrostrategy and and challengerschallengers like: like: Microsoft, Tableau, Tibco, Infor, BoardMicrosoft, Tableau, Tibco, Infor, Board and some more.and some more.

http://cognitivevr.co/

http://retinadVR.com/

http://rothenbergventures.com/

http://ghostline.xyz/

http://www.viarbox.com/2016/05/02/virtual_reality_analytics/

http://www.finwe.mobi/main/liveheat/

http://Election.tv/

http://www.kosher.tv/

http://www.its-people.de/

http://www.skilltower.com/


2USEMIR:VR/AR Analytics

links oben:www.retinad.io

Mitte: cognitivevr.co

Demo Demo timetime

http://www.retinad.io/

http://cognitivevr.co/


CognitiveVR scene explorer in ordinary web browser







Robert Merki, CognitiveVR:Robert Merki, CognitiveVR:

We also provide users with a We also provide users with a multitude of pre-built analytics calls. multitude of pre-built analytics calls.

These include algorithms for These include algorithms for detecting comfort, detecting comfort,

controller/HMD occlsion, controller/HMD occlsion, VR performance, VR performance,

and even arm length of users.and even arm length of users.

Robert Merki from cognitiveVR.co (Canada) holding my book – Selfie Image from Robert Sept.19th.2016



Robert Merki, CognitiveVR:Robert Merki, CognitiveVR:

The user who bought the game The user who bought the game could not get passed a certain level.could not get passed a certain level.

Upon analysis of the session, Upon analysis of the session, it was clear that it was clear that this particular this particular

user was too short to see the button user was too short to see the button that needed to be press[ed] that needed to be press[ed]

to open the next area!to open the next area!



Level 3: Approaches for displaying VR in headsets

Hybrid Rendering

Nozon.com Presenz

some light field tech

Live 3D in Game engine

(Unity3D, CryengineUnreal Engine, ...).

Pre-rendered

YouTube VR or 360°

VR Player apps, ...


measurement data, e.g.:measurement data, e.g.:Position 5,3,1Position 5,3,1Rotation 23°left, 18°upRotation 23°left, 18°upSpeed of movementSpeed of movement(eye position, parallax)(eye position, parallax)......

Mapping:Mapping:Position in VRPosition in VRObjects watchedObjects watchedInteractionsInteractions……

Merging with otherMerging with other

Data WarehouseData WarehouseContentContent

Enriched AnalysisEnriched Analysis

determinedeterminemeaningmeaning

MeasureMeasureAnalyzeAnalyze

← ← (Interactive live return path) ←(Interactive live return path) ←

Level 4: How to build a semanticLevel 4: How to build a semantic Meta Layer for VR Analytics? Meta Layer for VR Analytics?


depends on the particular VR/MR experience. depends on the particular VR/MR experience.

time matters as objects move over time!time matters as objects move over time!

meaning must be mapped manually or... meaning must be mapped manually or...

(soon) can be mapped automatically by autonomous analysis (soon) can be mapped automatically by autonomous analysis algorithms (Google/Facebook recognize objects, text and algorithms (Google/Facebook recognize objects, text and speech in images, videos, audio) e.g. Google Translate speech in images, videos, audio) e.g. Google Translate Augmented Reality or Youtube TranscriptionAugmented Reality or Youtube Transcription

Level 4: How to build a semanticLevel 4: How to build a semantic Meta Layer for VR Analytics? Meta Layer for VR Analytics?


2USEMIR: Neural Talk 2 – free image analysis with artificial intelligence (AI)

https://github.com/karpathy/neuraltalk2https://github.com/karpathy/neuraltalk2http://cs.stanford.edu/people/karpathy/neuraltalk2/demo.htmlhttp://cs.stanford.edu/people/karpathy/neuraltalk2/demo.html

https://github.com/karpathy/neuraltalk2

http://cs.stanford.edu/people/karpathy/neuraltalk2/demo.html


Level 5: data interchange between VR/AR and our reality

SmartWatch:SmartWatch:heart rate, ...heart rate, ...

voice- and voice- and mood analysismood analysis

facial expression facial expression analysisanalysis

NLP* NLP* eye movementseye movements

*NLP = Neuro Linguistic Programming:*NLP = Neuro Linguistic Programming:an approach to communication, personal development, and psychotherapyan approach to communication, personal development, and psychotherapy

created by Richard Bandler and John Grinder in California, United States in the 1970s.created by Richard Bandler and John Grinder in California, United States in the 1970s.


II.b Chances and Challenges with VR/AR Analytics


Challenges of the emerging VR market: Mistakes to avoid with VR/AR Analytics

The even Bigger Data challenge

technical fragmentation challenge

The challenge of Standards and content vs. technology improvements

The challenge of Data privacy and Security

The challenge of patent wars



BigBigDataData



BigBigDataData

GiganDataGiganData

VR/ARVR/ARturnturn

intointo


solution approaches: interpolate Data reduction – example from Blender – simplify F Curves addon.

For a VR Analytics system this has to be done by an automated algorithm.


Herausforderungen im VR Markt und Lösungsansätze

Challenges:

Big Data → GiganData

Technical Fragmentation

Standards vs. Innovation

Data Privacy and Security

Patent Wars

Approaches for solutions

interpolate, translate to meaning

Include intermediate results layer and abstraction tiers

Include intermediate results layer

open dialogue / co creation,security concept for VR/AR similar to BYOD

...


Level 5: facial expression analysis

Science Fiction? Only for Headsets?

predicted in 2012 at DOAG for 2017, predicted in 2012 at DOAG for 2017, already reality in 2015already reality in 2015



Emotion Analyticswith PC or smartphone www.affectiva.comwww.affectiva.com

The video, that watches you.The video, that watches you.analysis of facial expressions and your moodanalysis of facial expressions and your moodwith a simple webcam → Emotion Database with APIwith a simple webcam → Emotion Database with API

http://www.affectiva.com/



Yesterday:Yesterday:What did the user do while interacting?What did the user do while interacting?

Today:Today:What is the user doing while interacting?What is the user doing while interacting?

Tomorrow:Tomorrow:What is the user going to do What is the user going to do while interacting?while interacting?(interactive and responsive, blending worlds)(interactive and responsive, blending worlds)


Mission accomplishedMission accomplished

measurement data → semantic layer

combination withadditional data(heart rate, facial expressions,eye movement)

combination withclassic data warehousedata streams

enables (NLP)conclusionsclose to thought reading andprediction of behaviour

YES. YES.MUHAHAHA...


III. The 3 waves – from VR to AR to USEMIR: What future awaits us?*

*which has a huge impact onour business intelligence systems

and our culture

Level up!Level up!which will lead us to:which will lead us to:

USEMIRUSEMIR UUbiquitious biquitious SESEnsory nsory MIMIxed xed RRealityeality

*Quantum*Quantum TechTech

GeneticsGenetics RoboticsRobotics ITIT Nano-TechNano-Tech Space-TechSpace-Tech

accellerated by Q-GRINS*accellerated by Q-GRINS*


2USEMIR: Next generation ofmobile 3D devices

Google builds VR Unit+ Google Cardboard+ Google Jump / Daydream+ YouTube VR / Spotlight Stories+ ca.1.4 Billion Investment in MagicLeap 2014, 2015 Google + others+ self driving cars

Apple seems to build VR Unit+ acquired primesens+ acquired faceshift + hired Doug Bowman+ plans for self driving cars

Bildquelle: Project Tango, Google.com

First Google Tango Phone: First Google Tango Phone: Lenovo PHAB2 Pro announced Lenovo PHAB2 Pro announced for September 2016 for 499,-€for September 2016 for 499,-€


2USEMIR: MSQRD (simple smartphone camera – real time)

erhältlich im iStore und Playstore: https://play.google.com/store/apps/details?id=me.msqrd.android&hl=de


2USEMIR: Real-Eyes lightfield display without headset

www.real-eyes.eu – own photos by Joerg Osarek.


11 Stereo-photoswith photogrammetryturned back into3D model.

2USEMIR: Real-Eyes lightfield display without headset

www.real-eyes.eu – own photos by Joerg Osarek.


USEMIRUSEMIR tomorrow – or in the far future?tomorrow – or in the far future?

We will be surrounded byU.SE.MIRU.SE.MIR

UUbiquitious biquitious SESEnsory nsory MIMIxed xed RRealityeality

==bidirectional intelligent bidirectional intelligent

interactioninteraction

only linear progress only linear progress necessary!necessary!

Images: presskit from Microsoft.com hololens, affectiva.com , presskit fromhttp://digitalnature.slis.tsukuba.ac.jp/2015/06/fairy-lights-in-femtoseconds/

http://digitalnature.slis.tsukuba.ac.jp/2015/06/fairy-lights-in-femtoseconds/


I have an idea.

Let's provide themwith technology

to read our thoughts.

Quelle: Wikimedia.org – Erstellt 1590 – Public Domain

Who ought to know what about us? - a comparable encounterWho ought to know what about us? - a comparable encounter


What choices do we have?

back to the caves?back to the caves?

prepare to be terminated?prepare to be terminated?


What choices do we have?

Wrong question!Wrong question!Why do we fear a Terminator-Scenario?Why do we fear a Terminator-Scenario?

Why don't we consider ourselves asWhy don't we consider ourselves asself-confident honourable partners?self-confident honourable partners?

Is there a third choice?Is there a third choice?


The third choice:We can qualify for the 22nd century by cultivating chivalry and wisdom.

Q22=C+W

humbleness, honour, kindness, confidence,

politeness, courage and bravery, clemency, moderate behaviour ,

continuity, loyalty, dignity, discipline and decency


The third choice:We can qualify for the 22nd century by cultivating chivalry and wisdom.

Q22=C+W

self confidence by acting

honourably

working togetheras civilizationto boost our development

EARNING BONUS POINTS:EARNING BONUS POINTS:

appearing as self confident

and honourable examples and

partners


LEVEL UP!LEVEL UP! INSERT COINSINSERT COINSTO CONTINUETO CONTINUE

The rest is up to us now.

We shape our future together: with our systems and with our actions.

Remember:"with great powercomes greatresponsibility" (Stan Lee)


additional informationon VR/AR Analytics

read additional read additional information information in the book:in the book:



additional informationon VR/AR Analytics

3 video 3 video TrainingsTrainingsavailable soon atavailable soon attraining.vr.skilltower.comtraining.vr.skilltower.com



Contact and further information on Business VR / AR

Skilltower Institute

Joerg OsarekFounder and CEO

Triftstr. 30D-61350 Bad HomburgGermany

mailto:[email protected].: +49(0)151-23 0 24 333

vr.skilltower.com

mailto:[email protected]

http://vr.skilltower.com/

Data & Analytics

Virtual and Augmented Reality Analytics Digility