Dimitrios Kotsalisa
George Vellisa
Demosthenes Akoumianakisa
Jean Vanderdoncktb
aDepartment of Informatics Engineering, Technological Education Institution of Crete, GreecebUniversité catholique de Louvain, Louvain School of Management, Belgium
‘Implementation-agnostic’ instantiation schemes for ubiquitous, synchronous
multi-user interfaces
4th Mobile Device and Web software Development - Small-systems development (MDSD) 2014"
Athens, GreeceOct. 2-3, 2014
• Research on advanced, creativity-based User Interface engineering, meaning– non-trivial case scenarios– multi-user interfaces – synchronous collaboration – distributed users– ubiquitous settings
• Research protocol– Joint supervision of two researchers of the istlab
(Department of Informatics Engineering of TEI Crete) in PhD program at the Université catholique de Louvain, Louvain School of Management, Belgium
Context of the research
• Case study problem descriptions and overview of solutions (i.e., current version of the implementation)
• Brief and non-technical discussion of the engineering challenges and the technicalities of the work– Design-oriented inspiration– Development platforms and tools– On-going work
Plan of the presentation
• Two human collaborators / players– A single shared digital representation
• The soccer field
– Two types of agents• Human (collaborators) and non-human (soccer players)• Human collaborators virtualized and represented by soccer players
– Affordances of soccer players (inspired from semiotic engineering)• Each human collaborator operates with the currently selected soccer player• Each soccer player operates on a shared space inhabited by other players• Each human collaborator operates through different devices (Desktop/Swing and
Mobile/Android)• The combined experience (i.e., game co-play) entails operations within different
representations
• The system in its current operational setting
Scenario in MDSD’14 - The virtual setting
• How can we design and generate non-trivial multi-user interfaces that support synchronous collaboration between distributed users in ubiquitous settings?
• Existing approaches– Toolkit-based programming– Model-based UI engineering– Both exhibit limitations which do not allow effective
solution to the kind of problem described
Research question & related works
• Theory-based insights– Interaction devices versus affordances– Semiotic engineering for qualifying virtualities by type of agents
and kind of operation
• Engineering approach– Development of an abstraction-based model anchoring
interaction in to human intentions and capabilities– Specify capabilities in a model-based fashion– Extend UsiXML as needed to support the new specification
• Provisions for extensible interaction vocabularies
– Build dedicated • design tools • run-time environment components
Proposed approach
• Current implementation of platform-agnostic UI instantiation schemes
An illustration of the approach
• A scheme that relies on implementation agnostic (i.e., abstract) specifications of UIs
• At run-time and once user and usage context parameters are discovered, the implementation agnostic spec is translated to context-specific interaction vocabularies using dedicated tools
Platform-agnostic instantiation
Widget gallery
Polymorphic classification scheme for the ‘abstract button’ widget
Download
Download
• Introduce new widgets as first-class design objects
Widget Specification – Basic concept
WSL LibrariesResources
Widget Archive
Polymorphism & extensible interaction vocabularies
Widget Archive
• Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages.
An example of polymorphic specification of the ‘abstract SoccerField’ widget
• Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages.
An example of polymorphic specification of the ‘abstract SoccerField’ widget
Run-time scenario (MDSD’14 paper)
Some technical challenges resolvedInput/output techniques and event models
Model
View View
Window Window
selected tapped activated
View
Window
Breakdowns Different semantics Implementation
language/toolkitdependent
Some technical challenges resolvedStates and state transitions
onMouseDown
onMouseRelease
onSelectCommand
onChangePlayerCommand
onTap
onUntap
moving still movingstill movingstill
Sharing schemes for Synchronous collaboration: “Common States” sharing
Not abstract enough since there may be no common states or states at all
More recent developments(THALES-MusiNet, 2012-2015)
Score’ interactive music metaphor: PC, expert users
Tablature’ interactive music metaphor: PC, novice users
The ‘circular’ interactive music metaphor: Web, novice users
• Distributed Collaborative music learning• User roles
– Teacher (Expert)– Student (Novice)
• User stereotypes– Sighted– Blind
• Platforms– PC: Java/swing– PC: JNVT2– Web: HTML5 – Prototype
• Interactive metaphors– Score– Tablature– Circular
ScenariosMulti-user Music Notation Lessons
JNVT2
JNVT2
HTML5/Prototype
• Advanced collaborative features– Interim-feedback, Group awareness support (Multi-
user selection, Multi-user highlighting, Radar view, Film-view), Social awareness
Novel features
• YouTube channel (“MusicNet istLab - Tei Crete”)
Polymorphic specification
• Current and on-going work– Support for web2 UIs– Generic support for group awareness– Run-time adaptivity and UI plasticity in distributed and
ubiquitous settings– Distributed music learning
• Acknowledgements – The work is supported by ARCHIMEDES III, THALES– KUL for PhD dissertations (first two authors)
Future work & acknowledgements
Final remark
• All screenshots presented were taken from running prototypes.
• For those interested some of them are already available in our channels on YouTube (“MusicNet istLab - Tei Crete”, iSTLab TeiCrete).