CUbiC Confidential

CUbiC Confidential

• Computers enter everyday life; improvements in GUI and interface devices are accelerated

History of Computer Interfaces

19511943 1954 1960-68 1970-80 1980-90

Macintosh(1984)

Altari ST(1985)

Amiga(1985)

Windows(1985)

CUbiC Confidential

History of Computer Interfaces

19511943 1954 1960-68 1970-80 1980-90 1990-Present

WIMP Model

CUbiC Confidential

Recent Advances in Computer Interfaces

Motion Sensors.

Interactive feedback to users.

Multi-touch

Intuitive menus for ease of use

Multi-touch

Realistic object interaction

CUbiC Confidential

Beyond HCI

Culture, Age, Social, Situational and Contextual influences on HCI.

Safety and Reliability as inclusive components of HCI.

Emotional research into HCI.

CUbiC Confidential

Analysis of a new paradigm

CUbiC Confidential

Origins of HCC

Overcome problems with HCI

Technology has put too many computing devices (hence different interfaces) around the humans

Focus on the human and not the machine

CUbiC Confidential

En

HCC Ideology

Human

Machine

Natural interaction

Automation

Individualized information

Human-Machine task collaboration

Environment

CUbiC Confidential

• Establish Human Machine collaboration at a fundamental level

Human Machine Collaboration

CUbiC Confidential

Center for Cognitive Ubiquitous

Computing (CUbiC)

Team of multidisciplinary researchers, scientists and engineers.

CUbiC Confidential

CAReS

CAReS

Diagnostics (D)

Rehabilitation (R)

Assistive Technology(A)

CUbiC Confidential

Evolution of sensory signals in

humans

Concepts

Percepts

Signal

Cognition

Perception

Sensation

Example

Visual input

Color Percept

Concept

CUbiC Confidential

HCMC for CAReS

• Study sensory/cognitive capabilities/limitations through deficits

– Parkinson's

– Alzheimer's

– Visual Impairment

– Autism

– Elderly Care

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

13

CUbiC Confidential

iCARE

information technology Centric Assistive and Rehabilitative Environments

Flagship project at CUbiC

Initial focus on People with visual impairments

CUbiC Confidential

Inspirations from Focus Groups

“I’d like to be able to greet my friends as they walk toward me and sense the environment …”

- Jenny

“Don’t make me look like a Martian.”

- Robert

“My biggest dream is to walk into a library, pick up a book, and read it.”

- Alice

CUbiC Confidential

iCARE – Foundation of CAReS

iCARE

6. Presentation

Interface

1. Reading Interface

2. Social Interaction

3. Embodied Interface

4. Shopping Interface

5. Distal Object

Perception

CUbiC Confidential

1. iCARE Reading Interface

• Assist people who are visually impaired with reading text

• Converts visual text input into audio output

• Consists of:– Camera

– Ear phones

– PDA

– OCR

– Screen Reader

CUbiC Confidential

Development phases

Camera

Phase 1

• 2005

• 3 prototypes developed

• Deployed in ASU and

AzSDB

Phase 2

• 2006

• Personal size

• Customization

capabilities

Phase 3

• Current development

• Incorporating high

resolution digital cameras

on the glasses

CUbiC Confidential

2. iCARE Social Interaction

• Assist users to engage in social interaction

• Interprets social actions by recognizing people, emotions, expressions etc

• Consists of:

– Camera

– Ear phones

– PDA

– Face detection/recognition

CUbiC Confidential

Important Non-Verbal Cues

• Web survey on 8 needs found through focus group– 8 questions for the 8 needs– 5 answer levels: Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree– 27 participants: 16 blind, 9 low vision, and 2 sighted specialists

Question Number

Need Mean Score

8 Feedback on personal mannerism 4.5

6 Facial expression 4.4

3 Identity of people 4.3

7 Body gestures 4.2

1 Number of people in a group and where they are standing

4.1

2 Attention 4.0

5 Physical appearance change 3.5

4 Physical appearance 3.4

CUbiC Confidential

Some design concepts explored

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

21

CUbiC Confidential

Internals of iCARE Social

Interaction

Camera Face Detection

Face Processing *

Text-to-Speech Conversion

Audio Output

* Face Processing includes Face Recognition, Facial Expression and Emotion Recognition, Human Subject Counting, Human Subject Tracking, etc.

CUbiC Confidential

Operational detailsFace Detection Face Recognition with

Confidence

Capture faces

Add to Database

Retrain

CUbiC Confidential

Video of Social Interaction

Camera

( Typical Use Scenario )

Software

CUbiC Confidential

FacePix Database

Capture Setup

CUbiC Confidential

Face Pose Estimation

Face Images

Dimensionality Reduction through Supervised Manifold Learning

Linear Multi-Variate Regression

Pose Angle

CUbiC Confidential

Block Diagram: Face Pose Estimator

Wearable camera on user

Face detection (Viola – Jones:

Adaboost)

Pose Estimation (Supervised

Manifold Learning)

“Face detected to my near

right,”

“Person, standing to my near right, is looking to my

right”

CUbiC Confidential

Application as Assistive Device

• There are 4 people in the room

• Two are looking at the user

• One person is standing while three are sitting

CUbiC Confidential

3. iCARE-Embodied Interface

Traditional approach to Human Activity Analysis

An alternative approach… Wearable Sensors.

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

29

Illumination Variations

Occlusion

Low resolution, Low power wireless sensors like accelerometers, gyroscopes etc.

CUbiC Confidential

Human Motion Analysis through

wearable sensors - Applications

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

30

Wearable

Motion Capture

and Analysis

System

Assistive

Technology

Rehabilitative

Technology

Computer Games

Sports Medicine

Military Applications

Diagnostics•Abnormal Gait

Detection

•Tremor

Detection and

quantification

• Monitoring

movements for

Physiotherapy

•Fall Detection

and monitoring

•Long term

activity

monitoring for

elderly care

•Simulating real

movements in

the virtual game

•Automated

action report

generation

•Identifying

incorrect postures

CUbiC Confidential

Our focus…

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

31

Pattern Recognition

Feature Space Analysis

A

P

P

L

I

C

A

T

I

O

N

S

Domain

Information

CUbiC Confidential

Objective

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

32

Feature Extraction Classification

CUbiC Confidential

Framework

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

33

Fixed Length Segments

Statistical Feature

Extraction

Multi Class AdaBoost

Sitting

Standing

Walking

Running

Lying DownAccelerometers placed on the thigh and ankle

CUbiC Confidential

Data Visualization using PCA

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

34

CUbiC Confidential

Component 2: Store

server with a relational

database

Wearable

RFID

Reader

PDA with speech

Bluetooth

Component 1: User

wearable devices

Store

Database

Wi-Fi

Communication Model

Data FlowRFID Reader

© CUbiC Arizona State University

4. iCARE-Shopping Interface

CUbiC Confidential

5. iCARE Distal Object Perception

• Provides description of distant objects to the user

• Interprets scene in front of the user

• Consists of:

– Camera

– RFID reader

– Earphones

– Vibrators on gloves

– PDA

CUbiC Confidential

Related Work

Assistive devices for remote object perception

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

37

Shape: bowlTexture: smooth, etc.

CUbiC Confidential

Proposed Framework

CUbiC Confidential

6. iCARE –Presentation Interface

Current modalities

High information bandwidth

New modality

Active research tracks

Haptics

No exploited modes of presentation yet

Could be beneficial for multimodal interaction

MEMS has promise

CUbiC Confidential

Multimodal Perception

Task Describe an

apple

Descriptors

1. Color

2. Weight

3. Texture

4. Odor

Vision Touch Smell

© CUbiC Arizona State University

CUbiC Confidential

iCAREPresentation

Audio Data

EarphonesBone

Conduction Headphone

Haptic Data

Haptic glove Haptic Belt

Advantages

Easy to use

Easy Learning

Disadvantages

Hinders Hearing

Long term effects

Advantages

Does not affect normal

senses

New Modality for data delivery

Disadvantages

Learning Curve

Hindrance to movement

Modes of Presentation

CUbiC Confidential

Haptics – Sense of Touch

• The Hand is a contextual recognition engine.

• Hand movement used to perceive an object haptically are characteristic and show a one-to-one relation with feature being perceived.

• Haptic perception is most efficient in “egocentric” reference frames

• Haptic modality is most sensitive to surface property

CUbiC Confidential

Tactics

• Research towards Tactics has four components1) Development of visual-to-tactile image translation tools for specific

programs including MapQuest, PowerPoint, etc.

2) Development of novel image processing algorithms designed to improve visual-to-tactile image translation including algorithms for edge detection and content-dependent algorithms for face enhancement

3) Conduct experiments to explore the psychophysics and psychology of haptics related to tactile image exploration

4) Comparison of methods for tactile rendering including tactile printers/embossers and refreshable Braille displays

Center for Cognitive and Ubiquitous Computing (CUbiC) http://cubic.asu.edu

43

CUbiC Confidential

What and Where ?

CUbiC Confidential

Funding Agencies, Collabortors and Partners

NSF