View
2.487
Download
1
Category
Tags:
Preview:
DESCRIPTION
Citation preview
2 December 2005
Cross-Media Information Spaces and Architectures
Prof. Beat Signer
http://www.beatsigner.com
Web & Information Systems Engineering Lab (WISE)
Department of Computer Science
Vrije Universiteit Brussel
WEB & INFORMATION SYSTEMS ENGINEERING
01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110
Fluid Cross-Media Information Spaces
Prof. Dr. Beat Signer Interactive Paper, Cross-Media
Information Architectures
Dr. Bruno Dumas Multimodal Interaction, Cross-
Media Information Architectures
Lode Hoste Multimodal Interaction, Midas
and Mudra Framework
Brecht De Rooms Implicit Context-aware Human-
Computer Interaction, HCII
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
WEB & INFORMATION SYSTEMS ENGINEERING
CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
Sandra Trullemans Personal Cross-Media
Information Management
Reinout Roels MindXpres: Extensible Content-
driven Presentation Tool
Ahmed A.O. Tayeh Open Cross-Media Authoring,
Fluid Document Formats
WEB & INFORMATION SYSTEMS ENGINEERING
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
Methodology Conceptual Modelling
General Frameworks
Applications
User Studies
Rapid
Prototyping
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
PARC User Interface (Mid 70s)
Desktop metaphor
WIMP interaction
"filing cabinets" with
hierarchical folders
Paper paradigm
"What You See Is What
You Get" (WYSIWYG)
what about richer
document formats that
existed at that time?
Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Digital Documents as a Paper Simulator?
Vannevar Bush
Ted Nelson
Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS.
When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association.
As We May Think, Atlantic Monthly, July 1945
Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
What is Wrong with Digital Documents?
Existing document formats are based on the simulation
of paper affordances on desktop computers
How to manage mixed-media "documents" in open and
fluid cross-media information spaces? on the data level
- context-sensitive adapatation
- cross-media transclusion
on the visualisation and navigation level
- zoomable user interfaces
on the cross-media interaction level
- fluid multimodal cross-media interfaces
Remediation of the "paper simulation" approach WYSIWYG is only one out of many options!
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
RSL Metamodel and iServer
Global Information Systems Group, ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Links
Users
Layers Plug-ins
Structure
Active
Content
Global Information Systems Group, ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Bridging the Paper-Digital Divide
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Paper-Digital Integration
Amazon Kindle Anoto Digital Pen and Paper Technology
Augmented / Interactive Paper Electronic Paper
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Digital Pen and Paper Technology
Anoto offers
camera technology
pattern license
(virtual paper space
of 60 million km2)
Pen manufacturers
Nokia
Maxell
Adapx
Livescribe
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Digital Pens
Nokia SU-1B Magicomm G303 Livescribe Echo
Smartpen
FLY Fusion
Pentop Computer
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Cross-Media Link Server (iServer)
Global Information Systems Group
ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Interactive Paper Applications
Enhanced Reading
paper-digital maps
interactive festival brochures
educational materials
auction brochures
scientific publications
…
Enhanced Writing
photo album
researcher's notebook
mammography annotation
query by sketching
…
Paper-based Interfaces
PaperPoint
document proof-editing
digital libraries
video analysis tool
educational games
Art Installations
Lost Cosmonaut
Generosa Enterprise
Interactive Tabletops
photo browser
collaborative sketch tool
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Interactive Festival Guide (EdFest)
Global Information Systems Group
ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
EdFest
Global Information Systems Group, ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Visual and Interaction Design
Visualisation of interactive
areas and functionality
design patterns
Interaction design
online versus offline input
processing
lack of modal dialogues
multimodal interaction
Findings might be appli-
cable to traditional GUIs
e.g. cross-application widget
interaction
Global Information Systems Group
ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
PaperPoint Presentation Tool
Mobile PowerPoint
presentation tool
non-linear presentations
real-time annotations
digital whiteboard
...
Multi-pen support
brainstorming
Global Information Systems Group
ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Paper-Digital Maps
Szabolcs Becze
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
RSL-based Associative File System
signer
My Pictures
Vancouver ER 2010
Granville Island
Vancouver Harbour
Public Market
Keynote J. Thorp
Olga and Lamia
Dinner ER 2010
multiple classification
structural link
navigational link
My Videos
Vancouver Aquarium
ER 2010 Keynote
cross-media association
My Presentations
ER 2010 WISE 2009
cross-media
transclusion
single slide
transclusion
(via OOXML
RSL selector)
Gregory Cardone
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
RSL-based Associative File System ...
Gregory Cardone
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Personal Information Management
Keeping, organising and
re-finding information
digital and physical
Investigating user-
information interaction
Conceptual PIM model
(based on RSL)
Cross-media PIM system
document tracking
context model
...
Sandra Trullemans
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Interactive Tables
Pen-Based Google Earth Browser Pen-Based Drawing Tool
Photo Browser Photo Browser (Slider) Global Information Systems Group, ETH Zurich
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
What is Wrong with Slideware?
Simulation of physical slides limited space due to the slide concept
linear navigation from slide to slide
difficult to reuse content and embed rich media types
MindXpres addresses these issues unlimited canvas with zoomable user interface
non-linear navigation and associative linking
content-based approach with automatic visualisation
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
MindXpres Presentation Tool
HTML5-based extensible
core with three types of
JavaScript plug-ins
components
- source code, graph, video, ...
containers
- slide, image gallery, ...
structures
- grid, graph, clustered, ...
Presentation as RSL-
based overlay structure
enables navigation across
multiple presentations
Reinout Roels
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
MindXpres Presentation Tool ...
Reinout Roels
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
MindXpres Presentation Tool ...
Functional MindXpres prototype
Ongoing work visual authoring tool
- alternative to XML-based authoring
rich media plug-ins
- videos with subtitles, annotations etc.
- statistical data with alternative visualisations
- live sensor data
- ...
audience-driven presentations
collaborative annotations and information sharing
multimodal interaction (e.g. integration of PaperPoint concepts)
import of existing slideware formats
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
ArtVis
Advanced visualisation
techniques in combination
with a TUI
explore Web Gallery of Art
faceted browsing
Phidgets-based TUI
RFID-tagged physical
objects
Three main components
to explore, analyse and
browse the information
Bram Moerman
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
ArtVis ...
Bram Moerman
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
ArtVis ...
Bram Moerman
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Video: ArtVis
Bram Moerman
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Multimodal and Multi-Touch Interaction
EPOC, emotiv Echo Smartpen, Livescribe
Kin
ect, M
icro
soft
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Midas/Mudra Framework declarative definition of multimodal
and multi-touch interactions
rule-based language approach
rapid protoyping and application development
iGesture Workbench create and test gesture sets and algorithms
different modalities: digital pen, Wii remote, …
open source (www.igesture.org)
HephaisTK Toolkit domain-specific human-computer dialogue
language (SMUIML) for multimodal I/O
speech recognisers, RFID readers, …
Multimodal and Multi-Touch Interaction
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Mudra
Declarative rule-based language
Fusion across different levels of abstraction fact base
Rapid prototyping simple integration of new input devices
integration of external gesture recognisers
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Home Automation / Office of the Future
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
SpeeG2
User
Speech recognition
(Microsoft SAPI 5.4)
Skeletal tracking
(Microsoft Kinect)
5
4
2
3
SpeeG2 GUI
6
1
Sven De Kock
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Video: SpeeG2 Performance (21 WPM)
Sven De Kock
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Video: Kinect Presenter
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Video: Augmented Reality (Mental Nomad)
Toon Duwee
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Fluid Multimodal Cross-Media Interfaces
Investigation of mobile and distributed document formats Mobile Cross-Media Applications
Passive or active information units (crosslets) processing and storage of sensor data
binding of specific multimodal interactions to resources
Implicit context-aware human-computer interaction (HCII)
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Student Projects
Interactive paper and augmented reality various apprenticeships and thesis proposals in the
area of interactive paper and augmented reality
Multimodal and Multi-touch Interaction next generation text input devices (e.g. combine voice and Kinect)
multimodal mobile interaction
Cross-Media Information Spaces and Architectures fluid cross-media document formats
MindXpres presentation tool, office of the future, PIM, ...
If you should be interested in one of these topics
(or if you have your own ideas) please contact us! http://wise.vub.ac.be/thesis_proposals
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
WISE Courses
Advanced Topics in Information Systems seminar about recent developments in information systems
information systems and information management
information visualisation and navigation
human-machine and human-information interaction
[http://wise.vub.ac.be/content/advanced-topics-information-systems]
Next Generation User Interfaces general HCI introduction
multimodal interaction (theoretical concepts, fusion and fission, ...)
adaptation
pen-based interaction
tangible interaction and augmented reality
[http://wise.vub.ac.be/content/next-generation-user-interfaces]
WEB & INFORMATION SYSTEMS ENGINEERING
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
WISE Courses ...
User Interface Design general HCI introduction
- theory
- modelling
- evolution
new forms of human-machine interaction
[http://wise.vub.ac.be/node/15]
Web Information Systems hypermedia and the World Wide Web
Internet architecture and protocols
Web 2.0, Rich Internet Applications and Web 3.0
web search
[http://wise.vub.ac.be/content/web-information-systems]
WEB & INFORMATION SYSTEMS ENGINEERING
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
Conclusions
Innovative cross-media document formats which go
beyond simulating paper on desktop computers
RSL-based representation of open and fluid cross-media
information spaces interactive paper solutions
personal information management and associative file systems
MindXpres presentation tool
Multimodal and multi-touch interaction Midas/Mudra, HephaisTK and iGesture
SpeeG2, Kinect Presenter
Augmented reality ArtVis and Mental Nomad
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
References
B. Signer and M.C. Norrie, As We May Link: A General
Metamodel for Hypermedia Systems, Proceedings of the
26th International Conference on Conceptual Modeling
(ER 2007), Auckland, New Zealand, November 2007 http://wise.vub.ac.be/sites/default/files/publications/ER2007.pdf
Bachelor and Master Thesis Proposals http://wise.vub.ac.be/thesis_proposals
B. Signer, Fundamental Concepts for
Interactive Paper and Cross-Media Information
Spaces, ISBN 978-3-8370-2713-6, May 2008
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
References ...
B. Signer and M.C. Norrie, Interactive Paper: Past,
Present and Future, Proceedings of the 1st International
Workshop on Paper Computing (PaperComp 2010),
Copenhagen Denmark, September 2010 http://wise.vub.ac.be/sites/default/files/publications/PaperComp2010.pdf
B. Signer and M.C. Norrie, PaperPoint: A Paper-
based Presentation and Interactive Paper Prototyping
Tool, Proceedings of the First International Conference
on Tangible and Embedded Interaction (TEI 2007),
Baton Rouge, USA, February 2007 http://wise.vub.ac.be/sites/default/files/publications/Signer2007.pdf
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
References ...
B. Signer, M. Grossniklaus and M.C.Norrie,
Interactive Paper as a Mobile Client for a Multi-Channel
Web Information System, World Wide Web Journal
(WWW), Vol. 10, No. 4, Springer, December 2007 http://wise.vub.ac.be/sites/default/files/publications/WWWJ2007.pdf
L. Hoste, B. Dumas and B. Signer, Mudra: A Unified
Multimodal Interaction Framework, Proceedings of the
13th International Conference on Multimodal Interaction
(ICMI 2011), Alicante, Spain, November 2011 http://wise.vub.ac.be/sites/default/files/publications/ICMI2011.pdf
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
References ...
B. Signer, U. Kurmann and M.C. Norrie. M.C.,
iGesture: A General Gesture Recognition Framework,
Proceedings of the 9th International Conference on
Document Analysis and Recognition (ICDAR 2007),
Curitiba, Brazil, September 2007 http://wise.vub.ac.be/sites/default/files/publications/ICDAR2007.pdf
L. Hoste, B. Dumas and B. Signer, SpeeG: A Multimodal
Speech- and Gesture-based Text Input Solution,
Proceedings of the International Working Conference on
Advanced Visual Interfaces (AVI 2012), Capri Island,
Italy, May 2012 http://wise.vub.ac.be/sites/default/files/publications/AVI2012.pdf
Beat Signer - Department of Computer Science - bsigner@vub.ac.be
April 24, 2013
References ...
L. Hoste and B. Signer, Expressive Control of
Indirect Augmented Reality During Live Music
Performances, Proceedings of the 13th International
Conference on New Interfaces for Musical Expression,
Daejeon (NIME 2013), Korea Republic, May 2013
ArtVis Project http://wise.vub.ac.be/content/artvis-exploring-information-through-
advanced-visualisation-techniques
Recommended