Cross-Media Information Spaces and Architectures (CISA)

2 December 2005

Cross-Media Information Spaces and Architectures

Prof. Beat Signer

http://www.beatsigner.com

Web & Information Systems Engineering Lab (WISE)

Department of Computer Science

Vrije Universiteit Brussel

WEB & INFORMATION SYSTEMS ENGINEERING

01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110

Fluid Cross-Media Information Spaces

Prof. Dr. Beat Signer Interactive Paper, Cross-Media

Information Architectures

Dr. Bruno Dumas Multimodal Interaction, Cross-

Media Information Architectures

Lode Hoste Multimodal Interaction, Midas

and Mudra Framework

Brecht De Rooms Implicit Context-aware Human-

Computer Interaction, HCII

CISA

Human-Machine &

Human-Information

Interaction

Information

Systems &

Management

Information

Visualisation

& Navigation


CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)

Sandra Trullemans Personal Cross-Media

Information Management

Reinout Roels MindXpres: Extensible Content-

driven Presentation Tool

Ahmed A.O. Tayeh Open Cross-Media Authoring,

Fluid Document Formats


CISA

Human-Machine &

Human-Information

Interaction

Information

Systems &

Management

Information

Visualisation

& Navigation

CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)

Methodology Conceptual Modelling

General Frameworks

Applications

User Studies

Rapid

Prototyping

Beat Signer - Department of Computer Science - [email protected]

April 24, 2013

PARC User Interface (Mid 70s)

Desktop metaphor

WIMP interaction

"filing cabinets" with

hierarchical folders

Paper paradigm

"What You See Is What

You Get" (WYSIWYG)

what about richer

document formats that

existed at that time?

Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]


April 24, 2013

Digital Documents as a Paper Simulator?

Vannevar Bush

Ted Nelson

Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS.

When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association.

As We May Think, Atlantic Monthly, July 1945

Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009


April 24, 2013

What is Wrong with Digital Documents?

Existing document formats are based on the simulation

of paper affordances on desktop computers

How to manage mixed-media "documents" in open and

fluid cross-media information spaces? on the data level

- context-sensitive adapatation

- cross-media transclusion

on the visualisation and navigation level

- zoomable user interfaces

on the cross-media interaction level

- fluid multimodal cross-media interfaces

Remediation of the "paper simulation" approach WYSIWYG is only one out of many options!

CISA

Human-Machine &

Human-Information

Interaction

Information

Systems &

Management

Information

Visualisation

& Navigation


April 24, 2013

RSL Metamodel and iServer

Global Information Systems Group, ETH Zurich


April 24, 2013

Links

Users

Layers Plug-ins

Structure

Active

Content



April 24, 2013

Bridging the Paper-Digital Divide


April 24, 2013

Paper-Digital Integration

Amazon Kindle Anoto Digital Pen and Paper Technology

Augmented / Interactive Paper Electronic Paper


April 24, 2013

Digital Pen and Paper Technology

Anoto offers

camera technology

pattern license

(virtual paper space

of 60 million km2)

Pen manufacturers

Nokia

Maxell

Adapx

Livescribe


April 24, 2013

Digital Pens

Nokia SU-1B Magicomm G303 Livescribe Echo

Smartpen

FLY Fusion

Pentop Computer


April 24, 2013

Cross-Media Link Server (iServer)

Global Information Systems Group

ETH Zurich


April 24, 2013

Interactive Paper Applications

Enhanced Reading

paper-digital maps

interactive festival brochures

educational materials

auction brochures

scientific publications

…

Enhanced Writing

photo album

researcher's notebook

mammography annotation

query by sketching

…

Paper-based Interfaces

PaperPoint

document proof-editing

digital libraries

video analysis tool

educational games

Art Installations

Lost Cosmonaut

Generosa Enterprise

Interactive Tabletops

photo browser

collaborative sketch tool


April 24, 2013

Interactive Festival Guide (EdFest)


ETH Zurich


April 24, 2013

EdFest



April 24, 2013

Visual and Interaction Design

Visualisation of interactive

areas and functionality

design patterns

Interaction design

online versus offline input

processing

lack of modal dialogues

multimodal interaction

Findings might be appli-

cable to traditional GUIs

e.g. cross-application widget

interaction


ETH Zurich


April 24, 2013

PaperPoint Presentation Tool

Mobile PowerPoint

presentation tool

non-linear presentations

real-time annotations

digital whiteboard

...

Multi-pen support

brainstorming


ETH Zurich


April 24, 2013

Paper-Digital Maps

Szabolcs Becze


April 24, 2013

RSL-based Associative File System

signer

My Pictures

Vancouver ER 2010

Granville Island

Vancouver Harbour

Public Market

Keynote J. Thorp

Olga and Lamia

Dinner ER 2010

multiple classification

structural link

navigational link

My Videos

Vancouver Aquarium

ER 2010 Keynote

cross-media association

My Presentations

ER 2010 WISE 2009

cross-media

transclusion

single slide

transclusion

(via OOXML

RSL selector)

Gregory Cardone


April 24, 2013

RSL-based Associative File System ...

Gregory Cardone


April 24, 2013

Personal Information Management

Keeping, organising and

re-finding information

digital and physical

Investigating user-

information interaction

Conceptual PIM model

(based on RSL)

Cross-media PIM system

document tracking

context model

...

Sandra Trullemans


April 24, 2013

Interactive Tables

Pen-Based Google Earth Browser Pen-Based Drawing Tool

Photo Browser Photo Browser (Slider) Global Information Systems Group, ETH Zurich


April 24, 2013

What is Wrong with Slideware?

Simulation of physical slides limited space due to the slide concept

linear navigation from slide to slide

difficult to reuse content and embed rich media types

MindXpres addresses these issues unlimited canvas with zoomable user interface

non-linear navigation and associative linking

content-based approach with automatic visualisation


April 24, 2013

MindXpres Presentation Tool

HTML5-based extensible

core with three types of

JavaScript plug-ins

components

- source code, graph, video, ...

containers

- slide, image gallery, ...

structures

- grid, graph, clustered, ...

Presentation as RSL-

based overlay structure

enables navigation across

multiple presentations

Reinout Roels


April 24, 2013

MindXpres Presentation Tool ...

Reinout Roels


April 24, 2013

MindXpres Presentation Tool ...

Functional MindXpres prototype

Ongoing work visual authoring tool

- alternative to XML-based authoring

rich media plug-ins

- videos with subtitles, annotations etc.

- statistical data with alternative visualisations

- live sensor data

- ...

audience-driven presentations

collaborative annotations and information sharing

multimodal interaction (e.g. integration of PaperPoint concepts)

import of existing slideware formats


April 24, 2013

ArtVis

Advanced visualisation

techniques in combination

with a TUI

explore Web Gallery of Art

faceted browsing

Phidgets-based TUI

RFID-tagged physical

objects

Three main components

to explore, analyse and

browse the information

Bram Moerman


April 24, 2013

ArtVis ...

Bram Moerman


April 24, 2013

ArtVis ...

Bram Moerman


April 24, 2013

Video: ArtVis

Bram Moerman


April 24, 2013

Multimodal and Multi-Touch Interaction

EPOC, emotiv Echo Smartpen, Livescribe

Kin

ect, M

icro

soft


April 24, 2013

Midas/Mudra Framework declarative definition of multimodal

and multi-touch interactions

rule-based language approach

rapid protoyping and application development

iGesture Workbench create and test gesture sets and algorithms

different modalities: digital pen, Wii remote, …

open source (www.igesture.org)

HephaisTK Toolkit domain-specific human-computer dialogue

language (SMUIML) for multimodal I/O

speech recognisers, RFID readers, …

Multimodal and Multi-Touch Interaction


April 24, 2013

Mudra

Declarative rule-based language

Fusion across different levels of abstraction fact base

Rapid prototyping simple integration of new input devices

integration of external gesture recognisers


April 24, 2013

Home Automation / Office of the Future


April 24, 2013

SpeeG2

User

Speech recognition

(Microsoft SAPI 5.4)

Skeletal tracking

(Microsoft Kinect)

5

4

2

3

SpeeG2 GUI

6

1

Sven De Kock


April 24, 2013

Video: SpeeG2 Performance (21 WPM)

Sven De Kock


April 24, 2013

Video: Kinect Presenter


April 24, 2013

Video: Augmented Reality (Mental Nomad)

Toon Duwee


April 24, 2013

Fluid Multimodal Cross-Media Interfaces

Investigation of mobile and distributed document formats Mobile Cross-Media Applications

Passive or active information units (crosslets) processing and storage of sensor data

binding of specific multimodal interactions to resources

Implicit context-aware human-computer interaction (HCII)


April 24, 2013

Student Projects

Interactive paper and augmented reality various apprenticeships and thesis proposals in the

area of interactive paper and augmented reality

Multimodal and Multi-touch Interaction next generation text input devices (e.g. combine voice and Kinect)

multimodal mobile interaction

Cross-Media Information Spaces and Architectures fluid cross-media document formats

MindXpres presentation tool, office of the future, PIM, ...

If you should be interested in one of these topics

(or if you have your own ideas) please contact us! http://wise.vub.ac.be/thesis_proposals


April 24, 2013

WISE Courses

Advanced Topics in Information Systems seminar about recent developments in information systems

information systems and information management

information visualisation and navigation

human-machine and human-information interaction

[http://wise.vub.ac.be/content/advanced-topics-information-systems]

Next Generation User Interfaces general HCI introduction

multimodal interaction (theoretical concepts, fusion and fission, ...)

adaptation

pen-based interaction

tangible interaction and augmented reality

[http://wise.vub.ac.be/content/next-generation-user-interfaces]



April 24, 2013

WISE Courses ...

User Interface Design general HCI introduction

- theory

- modelling

- evolution

new forms of human-machine interaction

[http://wise.vub.ac.be/node/15]

Web Information Systems hypermedia and the World Wide Web

Internet architecture and protocols

Web 2.0, Rich Internet Applications and Web 3.0

web search

[http://wise.vub.ac.be/content/web-information-systems]



April 24, 2013

Conclusions

Innovative cross-media document formats which go

beyond simulating paper on desktop computers

RSL-based representation of open and fluid cross-media

information spaces interactive paper solutions

personal information management and associative file systems

MindXpres presentation tool

Multimodal and multi-touch interaction Midas/Mudra, HephaisTK and iGesture

SpeeG2, Kinect Presenter

Augmented reality ArtVis and Mental Nomad


April 24, 2013

References

B. Signer and M.C. Norrie, As We May Link: A General

Metamodel for Hypermedia Systems, Proceedings of the

26th International Conference on Conceptual Modeling

(ER 2007), Auckland, New Zealand, November 2007 http://wise.vub.ac.be/sites/default/files/publications/ER2007.pdf

Bachelor and Master Thesis Proposals http://wise.vub.ac.be/thesis_proposals

B. Signer, Fundamental Concepts for

Interactive Paper and Cross-Media Information

Spaces, ISBN 978-3-8370-2713-6, May 2008

http://wise.vub.ac.be/sites/default/files/publications/ER2007.pdf

http://wise.vub.ac.be/thesis_proposals


April 24, 2013

References ...

B. Signer and M.C. Norrie, Interactive Paper: Past,

Present and Future, Proceedings of the 1st International

Workshop on Paper Computing (PaperComp 2010),

Copenhagen Denmark, September 2010 http://wise.vub.ac.be/sites/default/files/publications/PaperComp2010.pdf

B. Signer and M.C. Norrie, PaperPoint: A Paper-

based Presentation and Interactive Paper Prototyping

Tool, Proceedings of the First International Conference

on Tangible and Embedded Interaction (TEI 2007),

Baton Rouge, USA, February 2007 http://wise.vub.ac.be/sites/default/files/publications/Signer2007.pdf

http://wise.vub.ac.be/sites/default/files/publications/PaperComp2010.pdf

http://wise.vub.ac.be/sites/default/files/publications/Signer2007.pdf


April 24, 2013

References ...

B. Signer, M. Grossniklaus and M.C.Norrie,

Interactive Paper as a Mobile Client for a Multi-Channel

Web Information System, World Wide Web Journal

(WWW), Vol. 10, No. 4, Springer, December 2007 http://wise.vub.ac.be/sites/default/files/publications/WWWJ2007.pdf

L. Hoste, B. Dumas and B. Signer, Mudra: A Unified

Multimodal Interaction Framework, Proceedings of the

13th International Conference on Multimodal Interaction

(ICMI 2011), Alicante, Spain, November 2011 http://wise.vub.ac.be/sites/default/files/publications/ICMI2011.pdf

http://wise.vub.ac.be/sites/default/files/publications/WWWJ2007.pdf

http://wise.vub.ac.be/sites/default/files/publications/ICMI2011.pdf


April 24, 2013

References ...

B. Signer, U. Kurmann and M.C. Norrie. M.C.,

iGesture: A General Gesture Recognition Framework,

Proceedings of the 9th International Conference on

Document Analysis and Recognition (ICDAR 2007),

Curitiba, Brazil, September 2007 http://wise.vub.ac.be/sites/default/files/publications/ICDAR2007.pdf

L. Hoste, B. Dumas and B. Signer, SpeeG: A Multimodal

Speech- and Gesture-based Text Input Solution,

Proceedings of the International Working Conference on

Advanced Visual Interfaces (AVI 2012), Capri Island,

Italy, May 2012 http://wise.vub.ac.be/sites/default/files/publications/AVI2012.pdf

http://wise.vub.ac.be/sites/default/files/publications/ICDAR2007.pdf

http://wise.vub.ac.be/sites/default/files/publications/AVI2012.pdf


April 24, 2013

References ...

L. Hoste and B. Signer, Expressive Control of

Indirect Augmented Reality During Live Music

Performances, Proceedings of the 13th International

Conference on New Interfaces for Musical Expression,

Daejeon (NIME 2013), Korea Republic, May 2013

ArtVis Project http://wise.vub.ac.be/content/artvis-exploring-information-through-

advanced-visualisation-techniques

http://wise.vub.ac.be/content/artvis-exploring-information-through-advanced-visualisation-techniques











