JASS `04 Benjamin Fingerle, Christian Wachinger1 2nd Joined Advanced Student School Ubiquitous Tracking based AR-Setups Benjamin Fingerle Christian Wachinger

JASS `04JASS `04

Benjamin Fingerle, Christian Wachinger 1

2nd Joined Advanced Student School

Ubiquitous Tracking based

AR-SetupsBenjamin Fingerle

Christian Wachinger

JASS `04JASS `04


Procedure

• We will present a scenario in which a user - Gerhard - gets into the benefits of an AR-enabled Environment

• First the scenario will be presented from the user’s perspective

• In a 2nd stage we’ll look behind the surface and see how the AR-environment can be modeled using the DWARF Service concept

• Finally we’ll go even deeper into detail and present how spatial relationships of objects can be represented using the Ubiquitous Tracking Framework

JASS `04JASS `04


The User’s Equipment

• Gerhard wears an Optical See Through Head Mounted Display (OST-HMD)

• Up on this HMD two 6DOF markers - one optical, one magnetic - are rigidly mounted

• Additionally a stereo-vision camera is installed on top of his head

• Gerhard wears special gloves with attached optical 6DOF markers

JASS `04JASS `04


A Day in Gerhard’s Life …

• Gerhard strolls down the TUM hallway greeting colleagues of his through closed doors

• Eventually he reaches his own office and steps in• Sitting down in front of the desk he reads a virtual

message of his Russian friend Vladimir from St. Petersburg asking for a chess game

• He starts the RemoteChess application whereupon a virtual chess board appears aligned on the desk and the virtual counterpart of Vladimir takes a seat across from him

JASS `04JASS `04


… A Day in Gerhard’s Life

• Suddenly an emergency request for instant help by his befriended Spanish colleague José shows up

• José asks Gerhard - a renowned surgeon - for advises regarding a difficult surgery José is currently conducting

• Just where a minute ago the chess board was visible a 3D - Model showing the patient crops up

• Gerhard studies the patient while having a look from different sides and getting computer tomography images registered on the patient on demand

• In vivid discussion with José a life is saved

JASS `04JASS `04


The Scenario Can Be Modeled With Services

• Services– Have needs

• Refined by predicates

– Offer abilities• Refined by attributes

• Connectors– Offer interfaces for data

exchange

• Service Managers– One for each network

node– Detect mutually

satisfying services– Provide services with

connectors

<service name=“OpticalTracker”><need name=“video” type=“VideoStream”>

<connector protocol=“sharedMemory”></need>

<need name=“marker” type=“MarkerData”><connector

protocol=“ObjectReference”></need>

<ability name=“markerPose” type=“PoseData”><attribute name=“location”

value=“$(marker.location)”><attribute name=“identity”

value=“$(marker.identity)”><connector

protocol=“NotificationPush”></ability>

</service>

Context = {location, identity, activity, time} can be modeled using Predicates and Attributes

JASS `04JASS `04


Services are Specified Using XML

<service name=“HMDOpticalTracker”><need name=“

...</need>

<ability name=“HeadPose” type=“PoseData”><attribute name=“location”

value=“$(landmark.location)”><attribute name=“identity”

value=“Gerhard”><connector protocol=“NotificationPush”>

</ability>

<ability name=“markerPose” type=“PoseData”>...

</ability></service>

JASS `04JASS `04


Hallway Services

Service Name Needs Abilities

GerhardConfigData -- landmarkDescription

HallwayConfigData -- landmarkDescription

HMDCamera -- videostream

HMDOpticalTracker videostream markerPose, HMDPose

HMDVideoShow videostream --

WhatsBehind pose videostream

ContextEstimator landmark context

JASS `04JASS `04


Office Services

Service Name Needs Abilities

GerhardConfigData -- landmarkDescription

HMDCamera -- videostream

HMDOpticalTracker videostream markerPose, HMDPose

HMDVideoShow videostream --

ContextEstimator landmark context

RoomConfigData -- landmarkDescription

RoomCamera -- videostream

roomMagneticTracker landmark markerPose

roomOpticalTracker videostream markerPose

Table 3D-content --

VirtualChess chessPartner 3D-content

VirtualSurgery handPose 3D-content

VirtualCommunication communicationPartner 3D-content

JASS `04JASS `04


Mutually Satisfying Services are Found by ServiceManager

• For each network node (e.g. Room, Hall, …) one ServiceManager exists

• This ServiceManager observes needs and abilities of those services belonging to his particular network node

• When a match is found the ServiceManager provides all involved parties with connector-objects

• These connectors offer push-, pull-, or shared memory access to abilities via various interfaces

JASS `04JASS `04


Matching of Mutually Satisfying Services

JASS `04JASS `04


Ubiquitous Tracking - The Formal Layer

• The crucial problem of Augmented Reality applications is the correct tracking of objects

• It is common to integrate the tracking procedure into the application

• Different tracking technologies are combined to get better results

• To release the AR application of tracking and to enable a seamless integration of new tracking devices a formal layer is introduced

• The formal framework, called UbiquitousTracking, forms the formal layer

JASS `04JASS `04


Ubiquitous Tracking – The Formal Framework

• Requests to the framework about the spatial relationship of objects can be send

• The answer delivers the optimal relationship available

Graph-model:• nodes == objects• edges == spatial relationships

JASS `04JASS `04


Ubiquitous Tracking - The Underlying Graph Model

• Properties of a spatial relationship– Represents the transformation and translation of the source

coordinate system to the target coordinate system– Attributes characterizing the quality of the relationship

• Three different types of graphs– Real relationship graph– Measured relationship graph– Inferred Relationship

JASS `04JASS `04


Real Relationship Graph

Each pair of objects has at every point of time a geometric relationship

JASS `04JASS `04


Measured Relationship Graph

• Estimates of relationships are just available for certain objects for discrete points of time

• Additionally Attributes characterizing the quality of the measurement exist

• Error function describing

the quality of a relationship

JASS `04JASS `04


Inferred Relationship Graph

• Knowledge about spatial relationships not just for discrete points of time is necessary

• Knowledge has to be inferred about the relationship of objects• Error functions help to find optimal inferences

JASS `04JASS `04


What Is a Good Error Function?

Aim: Finding optimal paths in the graph!

= { Latency,

Update frequency

Confidence value

Pose accuray

Time to live }

JASS `04JASS `04


Optimisation of Graph Algorithms

• Precomputing the Data Flow Graphs– Infrequently changing structure of the spatial relationships– Infrequently changing Attributes– No dependency on the pose measurements

• Grouping Nodes– Several nodes can be represented by a single supernode– Enabling level of detail hierarchies Faster graph search

What is the coordinate system of the new supernode?

JASS `04JASS `04


Security & Safety Aspects

• What if the “WhatsBehind” Service can be used in front of every door?

• What if your boss knows what you have done this weekend?

• What if other users know about every place where you have been this week?

• What if Indra the intern uses the AR setup of Gerhard?• What if Eve corrupts the data showing you the false

distance to the oncoming bus?• What if you can’t see the car because the latest sports

news occluded it?

Privacy

Restrictions

Authenticity

Privacy

Restrictions

Authenticity

JASS `04JASS `04


Finding Matches Forms Crucial Performance Issue

Matching problem of high computational complexity• One Service Manager for each Network Node• Number of potential matchings grows exponentially with

the number of services available inhibiting scalability

Different strategies for coping with this issue could include• Heuristics, based on context information, realized as

graph searches on the spatial relationship graph• Interpretation of matching problem as predicate logic

formula, applying specialized algorithms known from model checking

• …

JASS `04JASS `04


Issues Concerning Graph Representation

Possible ways of storing the spatial relationship graph include

• One graph-service holding the complete Graph

+ all graph algorithms applicable- contradicts the distributed computing paradigm

• Each Service knows its adjacency

+ complies with distributed computing paradigm

- not all graph algorithms applicable

JASS `04JASS `04


Issues Concerning Access to the Graph Information

Possible ways of accessing spatial relationships include• Request formulated as need for spatialRelationship with predicates

source and target, abilities offered by – Certain GraphInformationServices

+ Graph can be split up between different such services

- different graph search strategies for the same relation cannot coexist

– One service for each relationship instantiated by special relationShipGenerators

+ relationships instantly available

- different graph search strategies for the same relation cannot coexist

- number of services raises dramatically

• Requesting a graphInformerObject providing an interface getSpatialRelationShip(source, target)+ Objects implementing different graph searches can coexist

- centralization

JASS `04JASS `04


Open Questions

• How much information about the world do we have to put in the error function?

• Which graph algorithms have to be applicable?• Which graph representation allows theses algorithms?• How to perform context changes?• How to enable new users to enter AR environments?• What has then to be calibrated?

JASS `04JASS `04


Critical success factors

• Number of AR ready buildings• Development and acceptance of standards• Prices and convenience• Legal issues• Products and applications urging the user to buy

Documents

JASS `04 Benjamin Fingerle, Christian Wachinger1 2nd Joined Advanced Student School Ubiquitous Tracking based AR-Setups Benjamin Fingerle Christian Wachinger