Interfaces for human-centered production and use of computer … Cannavò_presentation.pdf · Prof. Fabrizio Lamberti, Politecnico di Torino. INTRODUCTION Computer Graphics Core enabling

Interfaces for human-centered production and use of computer graphics assets

Alberto Cannavò

XXXII cycle - Computer and Control Engineering

CommiteeProf. Bill Kapralos, Referee, University of Ontario Institute of TechnologyProf. Adalberto Simeone, Referee, Katholieke Universiteit LeuvenProf. Giuseppe Serra, Università di UdineProf. Riccardo Sisto, Politecnico di TorinoProf. Filippo Stanco, Università di Catania

SupervisorProf. Fabrizio Lamberti, Politecnico di Torino

INTRODUCTIONComputer Graphics

Core enabling technologies, e.g., for information visualization, Virtual Reality (VR) and Augmented Reality (AR)

INTRODUCTIONChallenges

Production & Utilization

Different user categories Expert users

First-time users

To improve the effectiveness of existing methods, tools and paradigms for the

production and use of computer

graphics content

By leveraging recent advances in the

fields of multi-modal interactionand intelligent computing

INTERFACES FOR HUMAN-CENTERED PRODUCTION AND USE OF COMPUTER GRAPHICS ASSETS

GOALResearch activities

AGENDAResearch activities

• Computer animation: How to bring a virtual character to life

• Is immersive virtual reality the ultimate interface for 3D animators?

• Interfaces and methods supporting the generation of graphics assets

• Leveraging graphics assets: Interactive applications development

• Augmented, mixed and virtual reality: A new perspective for human-machine interaction

COMPUTER ANIMATION: HOW TO BRING A VIRTUAL CHARACTER TO LIFE

• Labor-intensive task• Involvement of users

with significant expertise

• Sophisticated interfaces

• Motion capture• Reconfigurable tangible

user interface (TUI)• Natural language

processing (NLP)

[1] I., Satoru, et al. "Performance-based control interface for character animation", ACM Transactions on Graphics, 2009.

[2] A., Dennis & A. Whitehead "Wearable sensor networks for motion capture", Proc. Intelligent Technologies for Interactive Entertainment, 2015.

[3] N. Numaguchi, et al. "A puppet interface for retrieval of motion capture data", Proc. SIGGRAPH/Eurographics Symposium on Computer Animation, 2011.

[4] Y. Wataru, et al. "An actuated physical puppet as an input device for controlling a digital manikin", Proc. Conference on Human Factors in Computing Systems, 2011.

[5] J., Alec, et al. "Tangible and modular input device for character articulation", ACM Transactions on Graphics, 2015.

[6] O. Glauser, et al. "Rig animation with a tangible and modular input device", ACM Transactions on Graphics, 2016.

[7] M. Oshita. "Generating animation from natural language texts and semantic analysis for motion search and scheduling ", The Visual Computer, 2010.

CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE

Generically and Specially Shaped Tangible Devices

[4][3]

Reconfigurable Tangible Interfaces

[5] [6]

Related work

[1] [2]

Body and Hand Motion Tracking

Natural Language Processing

[7]

“Neo waves to Jack. At the same time,

Jack takes the red bottle”

Reconfigurable TUIs & Body tracking

Keyframing & Performance driven animation

Architecture designed for extensibility

F. Lamberti, G. Paravati, V. Gatteschi, A. Cannavò, P. Montuschi "Virtual character animation based on affordable motion capture and reconfigurable

tangible interfaces", IEEE Transactions on Visualization and Computer Graphic, 2017

CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE

Motion capture and reconfigurable tangible interfaces



CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE

Motion capture and reconfigurable tangible interfacesAutomatic mapping configurator:- Assignment problem

- Assembly to control all the DOFs

- Building step-by-step instructions

- Supporting forward and inverse kinematics



CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE

52 participants:- 26 Unskilled users (UNUs)- 26 Skilled users (SKUS)

Four animation tasks:- Lamp ref task- Crocodile ref task- Dyno ref task- Dyno free task


Animations systems evaluated:- Proposed System (Prop.)- Mouse & Keyboard (M&K)

Objective evaluation:- Completion time- Animation accuracy- Amount of work

Subjective evaluation:- Post-test questionnaire

Lamp ref task

Crocodile ref task

Dyno ref task



CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE

UNU SKU


Subjective evaluation

Objective evaluation

Required inputs:- screenplay- actions library

Modules:- Human-Computer Interaction (HCI)- Semantic Engine (SE)- Rendering Engine (RE)

F. Lamberti, V. Gatteschi, A. Sanna, A. Cannavò "A Multimodal Interface for Virtual Character Animation based on Live Performance and Natural Language Processing ",

International Journal of Human–Computer Interaction, 2019

Multimodal interfaces based on live performance and natural language processing

CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE

CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE




50 participants:- 25 UNUs (Prop. & Blender)- 25 SKUS (Prop.)

wag: 2.6

walk: 1.1

P: The dog is in the garden (CAMERASIDE) and is wandering wagging its

tail

C: The dog exits from the garden (CAMERAFRONT) and walks looking

around (CAMERASIDE)

N: The dog looks at the balls on the walkway and smells the foods

P: The dog exits from the garden (CAMERAFRONT) and walks looking

around (CAMERASIDE)

C: The dog looks at the balls on the walkway and smells the foods

N: Once the dog arrives close to the river it lies down for a moment

on the shore

look to Actor_Ball_2

jump to Actor_Log_2

P: The dog gets up (CAMERABACK) and hops on the log that is floating

on the water

C: Finally the dog jumps on the bridge

N:

GardenWalkway

River

Objective evaluation:- Completion time

Subjective evaluation:- Usability (Nielsen) - Effectiveness (Dimension Star Model)

Animation 1 Animation 2


Post-test questionnaire

Post-test questionnaire


Skilled

Unskilled

CO

MPU

TER A

NIM

ATION

:

HO

W TO

BR

ING

A V

IRTU

AL C

HA

RA

CTER

TO LIFE




NIELSEN DIMENSION STAR MODEL

0

1

2

3

4

5

−Concr

eteness

*

User c

ontributio

n

Cohesion*

Continuity

*

Stru

cture

−Cogniti

ve effo

rt*

Virtualit

y

Spat

iality*

Control

Inte

ract

ivity

*

Collabora

tion*

Imm

ersion*

Blender Proposed

Blender

Proposed

0

1

2

3

4

5

Lear

nabily

Effice

ncy

Mem

orabilit

y

Errors

Satis

fact

ion

Skilled Unskilled

0

1

2

3

4

5

−Concr

eteness

User c

ontributio

n

Cohesion

Continuity

Stru

cture

−Cogniti

ve effo

rt

Virtualit

y

Spat

iality

Control

Inte

ract

ivity

Collabora

tion

Imm

ersion

Skilled Unskilled

NIELSEN DIMENSION STAR MODEL

UNUs

SKUs vs UNUs

IS IMMERSIVE VIRTUAL REALITY THE ULTIMATE INTERFACE FOR 3D ANIMATORS?

Sense of presence3D interaction paradigm

VIRTUAL REALITY

SYSTEM OUTPUT

USER INPUT

[1] D. Vogel, et al. "AnimationVR – Interactive controller-based animating in virtual reality", Proc. IEEE Conference on Virtual Reality and 3D User Interfaces, 2018

[2] B. Berford, et al. "Building an animation pipeline for VR stories", Proc. ACM SIGGRAPH 2017 Talks, 2017.

[1] [2]

Research prototypes Commercial tools

Animations functionalities

Integration with animation suites

Extensive validations

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR

S?Related work

F. Lamberti, A. Cannavò, P. Montuschi "Is immersive virtual reality the ultimate interface for 3D animators?", COMPUTER, 2020.

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR

S?Immersive virtual reality-based interfaces for character animation

Buttons statusController

position/orientation

3D images of Blender’s viewport

VIRTUAL REALITY PLUGIN

BLENDER

HTC VIVE CONTROLLERS

HTC VIVE HEADSET

Transformations/properties changes

User’s commands

User & elements position/orientation

User’s head position/

orientation

VIRTUALREALITY

VIEWPORT

Animation functionalities accessible in VR

Tool for improving some of the stages of the animation pipeline


IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR

S?Immersive virtual reality-based interfaces for character animation27 participants:

- 15 Non-professional users (NPUs)- 12 Professional users (PRUs)

Five animation tasks:- Posing- Keyframing- Path animation- Performance anim.- Interpolation curve editing

Animations systems evaluated:

- VR-based interface (VRI)- Mouse & Keyboard (BNI)

Objective evaluation:- Completion time- Animation accuracy

Subjective evaluation:- System usability (SUS)- User’s satisfaction

Posingtask

Keyframing task

Performance animation task

Interpolation curve editing task

Path animation task


IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR


A. Cannavò, C.G. Demartini, L. Morra, F. Lamberti "Immersive virtual reality-based interfaces for character animation", IEEE ACCESS, 2019.

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR


New animation functionalities supporting Rigging, Skinning and Posing of virtual characters

BLENDER

Controller position/orientation

Buttons status

3D images of Blender’s viewport

VIRTUAL REALITY PLUGIN

HTC VIVE CONTROLLERS

HTC VIVE HEADSET

Transformations/properties changes

User’s commands

User & elements position/orientation

User’s head position/orientation

VIRTUALREALITY

VIEWPORT

INTERFACECONFIGURATOR

Configurations

Mapping

Customize VR functionalities

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR

S?Immersive virtual reality-based interfaces for character animation23 participants:

- 13 Non-professional users (NPUs)- 10 Professional users (PRUs)

Four animation tasks:- Rigging- Skinning- Posing- Complete anim. pipeline

Animations systems evaluated:

- VR-based interface (VRI)- Mouse & Keyboard (BNI)


Riggingtask

Skinning task

Posing task

Complete animation

pipeline task

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR



PRUs

Subjective evaluation

Objective evaluation

A. Cannavò, F. Lamberti "A Virtual Character Posing System based on Reconfigurable Tangible User Interfaces and Immersive Virtual Reality",

Proc. Smart Tools and Applications in Graphics (STAG), 2018.

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR

S?A virtual character animation system based on a reconfigurable TUI and immersive VR

Navigation of the scene

Selection Modalities:• Gaze selection• Proximity selection

VR Visual Feedbacks

20 participants:- 10 Unskilled users (UNUs)- 10 Skilled users (SKUs)

Animations systems evaluated:- Virtual Reality system (VRS)- Projected system (PS)- Mouse & Keyboard (M&K)

SKUs

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR

S?

A. Cannavò, F. Lamberti "A Virtual Character Posing System based on Reconfigurable Tangible User Interfaces and Immersive Virtual Reality",


A virtual character animation system based on a reconfigurable TUI and immersive VR

Animation task:- Posing task

Posing character through 3D sketching

Optimization problem

IS IMM

ERSIV

E VIR

TUA

L REA

LITY THE U

LTIMATE

INTER

FAC

E FOR

3D A

NIM

ATOR

S?

Rigged character

3D sketches

Characterpose

corr. matrix

input stroke

deformed vertices regularization term

constraints

ω: correspondence param.p: pose param.

y1

y2

yM

v1

v2

vK

3D sketch

Character skeleton

A. Cannavò, C. Zhang, W. Wang, F. Lamberti "Posing 3D Characters in Virtual Reality through In-the-Air Sketches",

Proc. 33rd International Conference on Computer Animation and Social Agents (CASA), 2020.

INTERFACES AND METHODS SUPPORTING THE GENERATION OF GRAPHICS ASSETS

3D computer graphics software suites

Steep learning curve

High flexibility

A. Cannavò, A. D'Alessandro, D. Maglione, G. Marullo, C. Zhang, F. Lamberti "Automatic Generation of Affective 3D Virtual Environments from 2D Images",

Proc. 15th International Conference on Computer Graphics Theory and Applications (GRAPP), 2020.

INTER

FAC

ES AN

D M

ETHO

DS SU

PPOR

TING

THE

GEN

ERATIO

N O

F GR

APH

ICS A

SSETSAutomatic generation of affective 3D virtualenvironments from 2D images

Source image Generated scene

3.08

2.75

3.58 3.17

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

4.25 4.08

4.25 4.42

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

Indoor Set. #1 Indoor Set. #2 Outdoor Set.#1 Outdoor Set. #2

Scene similarity Mood perception

A. Cannavò, F. Cermelli, V. Chiaramida, G. Ciccone, F. Lamberti, P. Montuschi, G. Paravati, "T4T: Tangible interface for tuning 3D object manipulation tools",

Proc. IEEE Symposium on 3D User Interfaces (3DUI), 2017.

G. Attanasio, A. Cannavò, F. Cibrario, F. Lamberti, P. Montuschi, G. Paravati, "HOT: Hold your own tools for AR-based constructive art",

Proc. IEEE Symposium on 3D User Interfaces (3DUI), 2017.

Interaction paradigm: - Cursor mode: virtual cursor- Tuning mode: tangible interface

INTER

FAC

ES AN

D M

ETHO

DS SU

PPOR

TING

THE

GEN

ERATIO

N O

F GR

APH

ICS A

SSETSAugmented reality for constructive art

Interaction paradigm:- Multi-touch capabilities- Printed markers

T4T: Tangible interface for tuning 3D object manipulation tools

HOT: Hold your own tools for AR-based constructive art

A participative system for tactics analysis in sport training based on immersive VR

A. Cannavò, M. Musto, F.G. Pratticò, F. Raho, F. Lamberti "A Participative System for Tactics Analysis in Sport Training based on Immersive Virtual

Reality", Proc. 4th Workshop on Everyday Virtual Reality (WEVR), 2018.

Pla

yer

app

licat

ion

Co

ach

ap

plic

atio

n

INTER

FAC

ES AN

D M

ETHO

DS SU

PPOR

TING

THE

GEN

ERATIO

N O

F GR

APH

ICS A

SSETSVirtual reality for sport training

A. Cannavò, D. Calandra, G. Basilicò, F. Lamberti "Automatic Recognition of Sport Events from Spatio-Temporal Data: an Application for Virtual

Reality-based Training in Basketball", Proc. 14th International Conference on Computer Graphics Theory and Applications (GRAPP), 2019

Automatic recognition of sport events from spatio-temporal data: An application for virtual reality-based training in basketball

VR Playbook

LEVERAGING GRAPHICS ASSETS: INTERACTIVE APPLICATIONS DEVELOPMENT

VR EXPERIENCE ALL NIPPON AIRWAYS (ANA)NATIONAL MUSEUM SINGAPORE,

SINGAPORE

MUSEO ARCHEOLOGICO

VIRTUALE (MAV), ITALY

SWAROVSKI VRSHOPPING EXPERIENCE

End-user development:- Visual language programming

Difficult for unskilled users

A. Cannavò, F. De Pace, F. Salaroglio, F. Lamberti "A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions",

International Journal of Human-Computer Studies, 2020

LEVER

AG

ING

GR

APH

ICS A

SSETS: INTER

AC

TIVE

APPLIC

ATION

S DEV

ELOPM

ENT.

A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions

GUI of Visual Scene Editor



LEVER

AG

ING

GR

APH

ICS A

SSETS: INTER

AC

TIVE

APPLIC

ATION

S DEV

ELOPM

ENT.

A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions20 participants:

- Three Expert users - 14 Unskilled users

Objective evaluation:- Completion time- Number of blocks- Number of errors

Tools evaluated:- Visual Scene Editor (VSE)- Leap Embedder (LE)- Blender Game Engine (BGE)

Subjective evaluation:

- System usability (SUS)- Task load (NASA-TLX)- Users’ preferences

Test application includes three scenes



LEVER

AG

ING

GR

APH

ICS A

SSETS: INTER

AC

TIVE

APPLIC

ATION

S DEV

ELOPM

ENT.

A Visual Editing Tool Supporting the Production of 3D Interactive Graphics Assets for Public Exhibitions

154

579

1,227

0

200

400

600

800

1000

1200

1400

VSE LE BGE

Co

mp

leti

on

tim

e [

s]

16

34

85

0

20

40

60

80

100

VSE LE BGE

Nu

mb

er

of

blo

cks

379

859

0

200

400

600

800

1000

1200

VSE LE

Co

mp

leti

on

tim

e [

s]

0.36

3.14

00.5

11.5

22.5

33.5

44.5

5

VSE LE

Nu

mb

er

of

err

ors

System SUS score

VSE 79.11

LE 53.57

System NASA-TLX Ind.

VSE 33.48

LE 59.79

Expert User

Unskilled users

AUGMENTED, MIXED AND VIRTUAL REALITY: A NEW PERSPECTIVE FOR HUMAN-MACHINE INTERACTION

High mental workloadSystems difficult to useEffectiveness of the system

CG assets as tools for providing effective feedback

D. Calandra, F.G. Pratticò, A. Cannavò, L. Micelli, F. Lamberti "Building reconfigurable passive haptic interfaces on demand using off-the-shelf

construction bricks", Proc. 6th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2019), 2019.

AU

GM

ENTED

, MIXED

AN

D V

IRTU

AL R

EALITY: A

NEW

PER

SPECTIV

E FOR

HU

MA

N-M

AC

HIN

E INTER

AC

TION

.Building reconfigurable passive haptic interfaces on demand using off-the-shelf construction bricks

Passive haptics built with

reconfigurable TUIs

Use case: Tasks of an escape room game

A. Cannavò, F.G. Pratticò, G. Ministeri, F. Lamberti ''A Movement Analysis System based on Immersive Virtual Reality and Wearable Technology for Sport

Training‘’, Proc. International Conference on Virtual Reality (ICVR 2018), 2018.

Real-time visual feedback

AU

GM

ENTED

, MIXED

AN

D V

IRTU

AL R

EALITY: A

NEW

PER

SPECTIV

E FOR

HU

MA

N-M

AC

HIN

E INTER

AC

TION

.A movement analysis system based on immersiveVR for sport training

17.3415.01

02468

101214161820

Eucl

idea

n d

ista

nce

[D

eg]

Projected Virtual

2255.09

1595.36

0300600900

1200150018002100240027003000

DTW

co

st

Projected Virtual

Ghost metaphor

Feedback on body

articulation

Overall evaluation

A. Cannavò, F. Lamberti, ''User interaction feedback in a hand controlled interface for robot team tele-operation using wearable augmented reality‘’,


Hand tracking input method (Leap Motion)Wearable video-see-trough deviceAR contents include:

- interaction area of the leap motion controller- video stream - command issue- status of the robot

AU

GM

ENTED

, MIXED

AN

D V

IRTU

AL R

EALITY: A

NEW

PER

SPECTIV

E FOR

HU

MA

N-M

AC

HIN

E INTER

AC

TION

.Interaction feedback for robot tele-operation with wearable augmented reality

2.53

3.33

2.53

3.133.40 3.67

2.40

3.13

2.13

3.00

0

1

2

3

4

Sco

re

Learnability* Efficiency* Memorability Error recov.* Satisfaction*

WearableDesktop

Completion time [sec]

WearableDesktop

ARM

CONCLUSIONS

• Design and develop of interfaces to support users in the creation and use of computer graphics assets

• Improvements in both objective and subjective terms

• Allowing unskilled users to approach the Computer Graphics field

• Improvements are observed in different stages of the pipeline and for numerous application domains

THANKS FOR YOUR ATTENTIONQuestions & Answers

Documents

Interfaces for human-centered production and use of computer … Cannavò_presentation.pdf · Prof. Fabrizio Lamberti, Politecnico di Torino. INTRODUCTION Computer Graphics Core enabling