Employing Multimodal Interactions to Ensure Transparent …€¦ · Ubiquitous / Pervasive Computing: ¾Weiser’s vision for the computing future in the 21st century: ... order to

IRIT - University of Toulouse

World Class Standards

Employing Multimodal Interactions to Ensure Transparent Authentication in Pervasive

Environments

Dana Al KukhunFlorence Sèdes

Multimodal Interaction with Mobile Devices

IRIT - University of Toulouse 2


PlanIntroductionPervasive EnvironmentsSecurity Challenges in Pervasive EnvironmentsCurrent Authentication MechanismsThe need for transparent interactions for mobilitySome directions towards transparencyConclusion



IntroductionUbiquitous / Pervasive Computing:

Weiser’s vision for the computing future in the 21st century:• Computing elements will disappear from user’s consciousness while

functioning homogeneously with his environment.

Pervasive Computing:Mobile UserInteracting with a systemUsing a mobile device

A Quality of Service challenge:How can we enable the user to interact seamlessly with a system at :

• Anytime • Anyhow different devices, connectivity channels, etc.• Anywhere



Pervasive computing

For hardware and computing elements to disappearSoftware needs to disappear.The spatial and temporal relationships between people and objects (HCI)has to be well defined in the early design phase

Pervasive information systemsAre interactive user-Centered systems That aim to facilitate user interaction within unfamiliar environments.

Objective:Transparent servicesProactivity (trying to anticipate the user needs)Efficiency, usability & user satisfaction



Challenges in Pervasive Environments

Data

Hardware SoftwareUsability Vs Automation

Accessibility Vs SecurityHeterogeneity Vs Connectivity

Adaptation



Importance of Multimodal Interaction

Would allow a greater portion of users to access a system:Old aged peopleDisabled people

• Handicapped, blind, …• Sick patients (robotics, medicine taking, calling for an emergency, etc)

Would allow easier interactionMobile devices have small screens, small keybords, etc

Would allow users to be more functional & productiveMulti functioning Faster interaction (voice commands, etc.)



Mobile Devices: some Limitations & some future directions for solutions

Low battery CapacitiesMultimodal interaction can consume the battery power .

• Waiting for increased battery power for higher efficiency.• Research works are looking for means to employ the surrounding

environment to generate power (solar power, physical power, etc.)Not all machines are fully equipped to apply successful multimodal interactions

cameras for Iris recognition need high resolution.Challenges in voice recognition for commands in different languages & in noisy environments (mobile users).

The cost of equipped devices is highHigh technology tends to be for the previleged society

• Looking for cheap & good quality machines for all society portions



Accessibility Vs SecurityHow to provide users with high accessibility while guaranteeing the system’s security?Security requires frequent re-authentication, but this would limit usability.

SecurityAuthenticationAccess ControlCryptography



Current authentication Mechanisms

Password (textual input)Bank kiosks, credit card shopping (french system), email

Digital SignatureCredit card shopping (American system) text recognition

VoiceVocal input voice recognition

Digital finger print, IrisBiometric & Image recognition

Machine TokensRFID chips, smart cards, …



Some Works on Multimodal Authentication

1. MUST: Multimodal User Authentication for Secured TransactionObjective: to investigate effective methods for the recognition of people over the internet based on voice and facial characteristics in order to facilitate, protect and promote various financial and other services over this growing telecommunication medium.

• Speaker verification• Face localization and verification

• taking into consideration lighting effects and several conditions.• Fusion of modalities for multimodal authentication.

European Project SER C01.0120 (2002 – 2005)



2. A Multimodal Authentication System for Authorizing the Access to NGN Services

Context : New generation networks• Many new services , Offered by different players (service providers). • Each service might have it’s own authentication method and use different

credentials.

Need for more reliable, flexible and easy-to-use methods.Proposition: Separating application-specific credentials (e.g. smart cards) hosted in the network, from personal identification (e.g. Smart cards). After the biometric identification is carried out, the contactless smart card allows to sense the user presence, in order to unlock his/her workstation and to recover from a server, using a secure protocolThe mechanism strength is guaranteed by limiting in time and space the validity of the logical association between biometric identification and card ownership

Melen et al., “A Multimodal Authentication System for Authorizing the Access to NGN Services”, International conference on Networking and Services, ICNS’06. Page: 47



Authentication @ the Pervasive age

The ultimate goal for pervasive computing is achieving invisibility.For ensuring security and privacy in those environments,

Both human beings and surrounding devices should be authenticated under the interaction methods that are used for ubiquitous services. But there exist many challenges as:

• The invisibility of devices, • the adaptiveness of interactions, • and the varying performance of devices

That will make it difficult to achieve invisibility.



Camera

GPS

Microphone

Loudspeaker

Stylus-Activated Sketch Pad

WearableComputeServer

Docking Stationfor Car PC

Biosensorfor Authentication

& Emotional Feedback

GSM for Telephone,Fax, Internet Connectivity

Directions to Efficient Solutions: 1) The SmartCom Project - The Handheld Assistant 2001



Smartcard/ Credit Cardfor authenticationand billing

Docking stationfor PDA/Notebook/Camcorderhigh speed and broadbandwidth Internet connectivity

High-resolutionscanner

Loudspeaker

Room microphone

Face-tracking camera

Virtual touchscreenprotected againstvandalism

Multipoint videoconferencing

The SmartCom Project - The Public Kiosk



Some Applications

Augmented Reality: Combining Speech, Gestures andGraphics for Mobile Web Access1. Mobile Dialog with a Virtual Tourist Guide2. Location-adaptive Query Interpretation 3. Multimodal Route Description4. Mobile Speech Translation & Multilingual Information Access5. Speech-based Access to 3D Virtual Views6. Multimodal Output from a Digital Library and Speech-based Access to Internet

Content



2) The Secure Phone ProjectA technical solution that improves security is needed

Solution joining

Biometric authentication• Makes use of VOICE, FACE and SIGNATURE

Multi-modal authentication

Mobile-device based authentication

Electronic signature

to secure information exchange

Privacy??

Protected by storing info on a secure SIM module

The use of multiple modaliaties would enhance security



Secure Phone

Strong authentication anywhere, to access databases, deal transactions, business operations, etc…Necessary security:

Gartner: more than 250.000 PDA are stolen and lost yearly in USAairports

http://www.secure-phone.info

VoiceFace

Written Signature



Fortifying Security by joining modalities

Pre-processing

ModellingModelling

Modelling

Pre-processingPre-processing

Access DeniedAccess Granted

FUSION

Face Voice Written Signature

Modelling



Some ApplicationsCorporate:

Secure transactions between sales force and headquartersSecure transactions with suppliers

Commercial:Mobile electronic banking Mobile private bankingMobile payments, on-line reservations

Governments and Public Administration:Administrative transactions through mobile phonePolice force communicationsMobile medical services



New Technologies for more transparency in authentication and interaction

Phidget Sensors:Touch Sensor RFID TagsFinger print readerAnd other multimodal authentication techniques.

Are suitable for mobile users and to enforce more security and Are important aspects for transparency & interactivity with pervasive environments

Documents

Employing Multimodal Interactions to Ensure Transparent …€¦ · Ubiquitous / Pervasive Computing: ¾Weiser’s vision for the computing future in the 21st century: ... order to