DataTiles: a modular platform for mixed physical and graphical interactions (rekimoto, CHI2001)

Copyright© 2001 Sony Computer Science Laboratories, Inc.

DataTilesDataTiles ：：A Modular Platform for Mixed Physical A Modular Platform for Mixed Physical

and Graphical Interactionsand Graphical Interactions

April 4, 2001April 4, 2001

Jun RekimotoJun Rekimoto** Brygg Ullmer Brygg Ullmer†† Haruo Oba Haruo Oba**

* * Sony CSL Interaction LaboratorySony CSL Interaction Laboratory† † MIT Media LaboratoryMIT Media Laboratory

2Copyright© 2001 Sony Computer Science Laboratories, Inc.

MotivationMotivation

Flexible interface platform for controlling Flexible interface platform for controlling digital informationdigital information

Merging two UI paradigms Merging two UI paradigms Graphicsl UIs & Physical/Tangible UI Graphicsl UIs & Physical/Tangible UI

Visual+Physical “construction kit” for end Visual+Physical “construction kit” for end usersusers

Some historical Some historical ancestors…ancestors…

“Denshi (electronic) Block” (‘70s)Japan Railload “MARS” terminal (‘70s)

“Triangles” (Gorbet/Orth/Ishii, 1998)

“AlgoBlock” (Suzuki/Kato, 1995)

“Intelligent Beermats” (Frazer, 1982)

4Copyright© 2001 Sony Computer Science Laboratories, Inc.

DataTiles

5Copyright© 2001 Sony Computer Science Laboratories, Inc.

Transparent objects (tiles) as information modules → tight coupling of input & output

Integration of Physical manipulation & graphical information Used as Home-Computer interfaces, Collaboration supports, Internet-”Toys”

DataTiles

6Copyright© 2001 Sony Computer Science Laboratories, Inc.

Time Point Information

7Copyright© 2001 Sony Computer Science Laboratories, Inc.

Tile ClassesTile ClassesApplication Tiles

Portal (Remote) Tiles

Parameter Tiles

Container Tiles

Stores information

Specific Services, Applications

Connects to real-world objects(places, people, devices)

Visualizes & Controls other connected tiles

8Copyright© 2001 Sony Computer Science Laboratories, Inc.

video

9Copyright© 2001 Sony Computer Science Laboratories, Inc.

DataTiles DataTiles basic interaction style basic interaction style

Pen Sensitive LCD Display (“Tray”)ID-Reader

tagged transparent objectRFIDRFID

GroovesPrinted Information

Input devices (Pen, etc.)InputGraphical Output

10Copyright© 2001 Sony Computer Science Laboratories, Inc.

Interacting with DataTilesInteracting with DataTiles Interaction TechniquesInteraction Techniques

Tile placement, reconfiguration, removalTile placement, reconfiguration, removal Pen manipulations on each tile (as GUI)Pen manipulations on each tile (as GUI) Physical manipulators (e.g., dial)Physical manipulators (e.g., dial) Inter-tile pen gesturesInter-tile pen gestures

Tile adjacency Tile adjacency communications between tiles communications between tiles Parameter flow: Query Parameters, Time pointsParameter flow: Query Parameters, Time points Data-flow: Image, movie, soundData-flow: Image, movie, sound

11Copyright© 2001 Sony Computer Science Laboratories, Inc.

Printed & Grooved TilesPrinted & Grooved Tiles

=+

+ =

printed information:high resolution,

static

displayed information:low resolution,

dynamic

grooved widget:passive haptic

feedback

combined:

12Copyright© 2001 Sony Computer Science Laboratories, Inc.

Tiles with manipulatorsTiles with manipulators

13Copyright© 2001 Sony Computer Science Laboratories, Inc.

Container PaintPortal

WebCam

print

image

real printer Application Tile

Container Tile

Portal Tile

Container

Movie Movie Movie

Examples of Tile CombinationsExamples of Tile Combinations

ParametersBaseballTimeWheelScribble

time point

parameter tile parameter tile

query parameters

14Copyright© 2001 Sony Computer Science Laboratories, Inc.

Visual Feedback of tile connection Visual Feedback of tile connection typestypes

Inter-tile pen gesturesInter-tile pen gestures

Movie Wall Screen ATimeMachineWeather

Simple connection

PeopleContainer

Menu for choosingConnection types

Mail

Fax

Invokes one-shot transmissionPhotoAlbumContainer Remote Portal

Suspends streamed transmission

Animation indicating streamed media transmission

System ConfigurationSystem Configuration

RFID tags (TIRIS)

RFID reader(TIRIS)

RFID multiplexer+

PC interface(PIC16F873)

RS232C

Switches 12 antenna coils（ 100ms / ID)

RFID reader coils

PC (Sony VAIO LX)Windows 2000

Pentium III (866MHz)Java JDK1.3

LCD Display with built-in pen tablet

16Copyright© 2001 Sony Computer Science Laboratories, Inc.

LCD display

Back lightPen digitizer

Wire to the RF reader

Acrylic tile cover glass

6mm

RF tag

RFID reader coils

Wire to the RFID reader

RFID tag

Reader coil

Sensor ConfigurationSensor Configuration

17Copyright© 2001 Sony Computer Science Laboratories, Inc.

Implementation (software)Implementation (software)

The System dynamically loads and activates a corresponding Java Class for each tile ID.

Each Java program also dynamically checks nearby tile classes and interfaces.

public class WebCamTile extends Tile implements ImageCreateInterface,

TimeMachineInterface, … {…………

}

……Tile t = getLeftTile();if (t instanceof TimeMachineInterface) {

…..}….

18Copyright© 2001 Sony Computer Science Laboratories, Inc.

User ExperiencesUser Experiences Data manipulation by physical layoutData manipulation by physical layout

Many users were willing to explore various types of Many users were willing to explore various types of tile combinations. Some of them were beyond our tile combinations. Some of them were beyond our expectation.expectation.

Visual feedback was essentialVisual feedback was essential

Effects of tiles’ graspable capabilityEffects of tiles’ graspable capability Users could always take control Users could always take control

(e.g., remove a music tile to stop it) (e.g., remove a music tile to stop it) Suitable for multi-user activitiesSuitable for multi-user activities

19Copyright© 2001 Sony Computer Science Laboratories, Inc.

ApplicationsApplications Interfaces to Networked InformationInterfaces to Networked Information

Portal to Information servicesPortal to Information services Information Appliances ControlInformation Appliances Control Remote CollaborationRemote Collaboration

Media Construction EnvironmentsMedia Construction Environments Movie/Music Editing, Video JockeyMovie/Music Editing, Video Jockey (Networked) Game Environment(Networked) Game Environment Education Tools, ToysEducation Tools, Toys

Support of Collaborative Work, Decision Support of Collaborative Work, Decision makingmaking Simulation, database visualizationSimulation, database visualization

20Copyright© 2001 Sony Computer Science Laboratories, Inc.

SummarySummary

DataTiles:DataTiles: 　　 A platform environment with mixed Physical-A platform environment with mixed Physical-

Graphical InteractionsGraphical Interactions Future WorkFuture Work

Study of Physical Language “Grammar”Study of Physical Language “Grammar” Other configuration method (e.g, stacking)Other configuration method (e.g, stacking) Active (CPU embedded) tiles with sensorsActive (CPU embedded) tiles with sensors