Model-based Design of Multimodal Interaction for Augmented...

. 1

22.06.2015

Sebastian Feuerstack, Állan C. M. de Oliveira, Mauro dos Santos Anjo, Regina B.Araujo, and Ednaldo B. Pizzolato

Model-based Design of Multimodal

Interaction for

Augmented Reality Web Applications

The Augmented “Drag-and-Drop”

Overall Research Question

2

22.06.2015

How to model

interactions that

span Modes and

Media ?

http://www.multi-access.de

Multimodal Interfaces

22.06.2015

Which Media ?

http://flic.kr/p/6eeLwg

Which Modes ?

http://flic.kr/p/6eeLwg http://flic.kr/p/7sZ6ho

3

Interaction Resources

22.06.2015

How ? Interaction Resources

4

Why modeling?

22.06.2015

Why modeling?

One specification that

is declarative and exact

discuss & execute & store & share

add new forms of interaction

no code but model changes

5

Related Work 6

22.06.2015

Use Case:

Augmented Drag and Drop

7

22.06.2015

Buying Furniture

Shopping Cart

Drag and Drop

Into my environment

How to model?

22.06.2015

8

Cameleon Reference Framework

W3C MBUI

Task Model

Abstract UI

Custom

Interaction Resources

Mappings

Abstract User Interface Model

9

22.06.2015

Abstract User Interface Model

22. Juni 2015 The Augmented “Drag-and-Drop

10

Abstract Behavior Model

22. Juni 2015

AUI:AIO:AIChoiceElement:

AISingleChoiceElement

initialized

listed

focus

suspended

organize

defocus

organized

suspend

focused

organize

Presenting

chosen

unchosen

drag

dragging

unchoose

drop

H

[in(focused)] choose

/ aios=find(parent.childs.chosen);

aios.all.unchoose

next||prev||parent

/aio=find(act);

aio.focus

Selectionpresent

11

Multimodal Mapping: The Mouse-based Drag-and-Drop

22.06.2015

C

dst=SingleChoice.focusedC

Tw<0,3s

sc.dragdst.drop(aios)

sc=SingleChoiceElement.focused

Tw<0,3sLeftButton.pressed

LeftButton.released

sc.parent.dropfail

12

IR:IN:Pointer

stopped

move stop

moving

IR:IN:Button

released

press release

pressed

IR:IN:Mouse Mouse

ButtonWheel Pointer

LeftButton RightButton

- x,y- z

Hand Gesture Resource Interactor

22.06.2015

IR:IN:HandGesture:PointingSelect

left_hand_appeared

thumb_down

Left Hand

left_hand_disappeared

selected

released

thumb_up

Command

Right Hand

NoLeftHandBothHands

NoRightHand

right_hand_disappearedright_hand_appeared

Pointing

move

moving

stopped

stop

move

NoHands

left_hand_appeared right_hand_appeared

left_hand_disappearedright_hand_disappeared

Cdst=SingleChoice.focused

C

Tw<0,3s

aois.dragdst.drop(aios)

sc=SingleChoice.focused

Tw<0,3sRightHand.cmd.selected

RightHand.cmd.released

Video: Connecting the Web with AR

14

22.06.2015

Cam

Cam Pointing

Command

Monitor

Jessica

Conclusions & Future Work

22.06.2015

• Detailed, declarative modeling of multimodal interactions

• We can combine modes and media by mappings and

bridge realities

• All models are interpreted and can be changed while

running

• What`s still missing is a tool / editor

• We intend to contribute to

• W3C-MMI WG

• W3C Ubiquitous Application Design CG

15

Questions ?

22.06.2015

Thanks for your attention

Sebastian Feuerstack

feuerstack@offis.de

16