Syntherella: A Feedback Synthesizer for

Efficient Exploration of Virtual Worlds using a Screen Reader

Bugra Oktay and Eelke FolmerPlayer-Game Interaction Lab

Outline

Discussion

Syntherella

Intro

Problem

Future Work

Introduction

Virtual WorldScreen Reader

IntroductionVirtual World

TextSL

Problem

Densely Populated Environments

Lots of Information to consume

Easy with visual perception;

overwhelming when transformed into text

overwhelming

Problem (cnt’d)

Command-line interface

Iterative Queries

Slow interaction(even when there are

very few objects)

“describe”

“describe objects”

“describe table”“zzz… zzz…”

Consumable

Syntherella

There are 5 animals, 4 tools, a house and a bike.

There are 2 books, a cat, a dog and a house to your west; a dog, a bike … to your east; ... 2 cats are described as “Angora cat with blue eyes”. One of them is interactive and for sale for $20 ...

Syntherella: Scanning

Scanning objects inside a virtual circle around the user’s avatar where the radius is defined by the user.

Syntherella: Filtering

Filtering objects that do not contribute to the textual description of the environment (i.e. objects with non-descriptive or meaningless names).

Syntherella: Thresholding

cat

stoneboxundescriptiv

e

too far

catlaptop

table

book

Good enough

16

74

too small

30 Size / Distance /Name / Description /

Interaction Capabilities

Syntherella: Grouping

cat

book

cat

cat

chair

chair

chair

chair

table

book

cat

book

cat

cat chair chair

chair

chair

table

book

cat, book, chair, chair, cat, table, chair, book, chair and cat. 3 cats, 2 books, 4 chairs and table.

Syntherella: Aggregation

object

cat

animalvehicle

mammal

…

dog

reptile…

snake 2 bikes, 3 cats, 5 dogs and a snake.

2 bikes, 8 mammals and a snake.

2 bikes and 9 animals.

There are

ground vehicle

bike

11 objects.

more specific

Syntherella: Detailing

Level 3: 3 cats are described as “Angora cat with blue eyes”. 2 of them are interactive. 1 of them is not interactive. 2 chairs are described as “black leather office chair”. They are not interactive.

Level 4: 3 cats are described as “Angora cat with blue eyes”. 1 of them is interactive but not for sale. 1 of them is interactive and for sale for L300. 1 of them is not interactive and not for sale. 2 chairs are described as “black leather office chair”. They are not interactive but for sale for L20.

Level 2: 3 cats are described as “Angora cat with blue eyes”. 2 chairs are described as “black leather office chair”.

There are 2 cats to your north; 1 cat to your south and 2 chairs to your east.

Syntherella: Example

There are 936 objects: window, object, prim, box_by_vi$boyz, chair, tree, flowerz, book v2.0, stone, dog, cat, book, window, window, object, object, cat,

no_name, obj23, owl, table, window, object, notecard dispenser, SL educational places kiosk, landmark_3.4, buy_me_L34, object, window, table,

chair, chair, car, chair, bicycle, rabbit, rose, chair, chalk, object, skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote, voice activated robot helper, phone booth, object,

skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote, voice activated robot helper, phone booth,

window, object, prim, box_by_vi$boyz, chair, tree, flowerz, book v2.0, stone, dog, cat, book, window, window, object, object, cat, no_name, obj23, owl,

table, window, object, notecard dispenser, SL educational places kiosk, landmark_3.4, buy_me_L34, object, window, table, chair, chair, car, chair,

bicycle, rabbit, rose, chair, chalk, object, skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote, voice

activated robot helper, phone booth, object, skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote,

“describe objects”

Pretty

long

Syntherella: Example

There are 87 objects: frame, object, dog, step, object, object, object, cat, prim, box_by_vi$boyz, object, stone, tree, rose, chair, object, object, frame, nozd_jkie&&jack, stone, step, dog, object, object, object, object, chair, table, cat, object, prim, prim, tree, prim, box, Julia’s Gull, cat, chair, object, object, object, object, object, prim, prim, object, stone, step, object, chair, table, cat, object, prim, prim, tree, prim, box, cat, chair, object, object, object, object, object, prim, prim, object, stone and step.

After scanning

There are 26 objects: frame, dog, step, cat, stone, frame, tree, rose, chair, frame, stone, dog, chair, table, cat, cat, chair, stone, step, stone, tree, chair, stone, step, Julia’s Gull and frame.

After filtering

Syntherella: Example

There are 14 objects: dog, cat, tree, rose, chair, dog, chair, table, cat, cat, chair, tree, chair and Julia’s Gull.

After thresholding

There are 14 objects: 2 dogs, 3 cats, 2 trees, rose, 4 chairs, table and Julia’s Gull.

After grouping

There are 3 plants, 5 animals, 5 furniture and Julia’s Gull.

After aggregation (word limit: 14)

There are 13 known and 1 unknown objects.

After aggregation (word limit: 10)

Syntherella: Example

There are 2 dogs, 1 cat, 2 trees to your north; 4 chairs, 1 table, 1 rose to your west and 2 cats, Julia’s Gull to your south. 2 dogs are described as “black and brown Saint Bernard”. 3 cats are described as “Angora cat with blue eyes”. 1 tree is described as “large oak tree”. 1 tree is described as “green and tall palmae tree”. 4 chairs are described as “brown stylish dining chair”. Rose is described as “Red rose”. Table is described as “diner table for 4”. Julia’s Gull is described as “Laughing gull that can fly; designed by Julia Doe”.

After detailing

Discussion & Future Work

• User Studies• Performance Measurements• Widening the Scope (augmented reality, navigation, web accessibility etc.)• Goal/Intention Oriented Feedback Generation• Exploiting Command History

•Information Source•Perceptional Challenges

http://ear.textsl.org

Discussion & Future Work

• User Studies• Performance Measurements• Widening the Scope (augmented reality, navigation, web accessibility etc.)• Goal/Intention Oriented Feedback Generation• Exploiting Command History

•Information Source•Perceptional Challenges

http://ear.textsl.org

bugra@cse.unr.edu