1. The GhostHands UX: telementoring with hands-on augmented
reality instruction Giuseppe Scavo Knowledge Media Institute
2. 2 AR in manufacturing 2
3. 3 Advantages of AR & Telementoring Decreased error rate
Decreased task completion time Enhanced awareness and safety
Faster/cheaper training 3 What happens during unforeseen
situations? Is AR telementoring an effective replacement for the
experts physical presence? The help of the expert is often crucial
for the successful completion of work/training sessions that
require specific knowledge.
4. 4 AR telementoring vs. physical presence 4 Communication
channel AR tele- mentoring Physical presence Verbal Communication
Shared visual space Overlaid information Visual embodied learning
Manufacture: from latin manu (hand) + factum (made)
5. 5 GhostHands 5 GhostHands allows an instructor to remotely
stream his/her hands in an over-the-shoulder way onto a workers
field of view. Instructor and trainee share visual space, audio
channel and they also see each others hands. Imagine if you could
see the hands of your instructor in front you, interacting with the
environment around you.
6. 6 GhostHands 6
7. 7 GhostHands: A new interaction modality Natural interface
Communication fastening Conversation timing Visual learning
Learn-by-doing experience 7
8. 8 Use cases 1-on-1 training sessions for new workers Changes
in the production process Brand new machinery New maintenance
procedures Machine faults Safety critical situations Any unforeseen
scenario
9. 9 Implementation 9 Instructor-side application Node.js
server Trainee-side application Video stream JSON tracking
data
10. 10 UX evaluation framework Human needs/expectations Product
qualities Relatedness/Conne ctedness Utility Stimulation
Captivation Intuitiveness Playfulness & Entertainment Usability
Learnability Efficiency & Effectiveness Safety/Trust
Competencies Empowerment Increased awareness and knowledge
Efficiency & Accomplishment Visual attractiveness Security
Hedonic qualities Autonomy UX is often neglected in AR. However, it
is extremely important to deliver successful technologies.
11. 11 Experiment 6 couples of participants, all researchers 3
tasks of increasing difficulty to perform, designed after real-life
use cases: Rotor disc servicing. (19 steps) Motor/pump alignment.
(24 steps) Thread feeder unit replacement. (33 steps) Worker
equipped with head-worn camera. A display in background showed the
video feed with GhostHands superimposed Expert equipped with laptop
and Leap Motion sensor. The expert was briefed before the task.
Post-task open answer questionnaires
12. 12 Findings Naturalness & Intuitiveness Frustration due
to imprecision There were slight moments of frustration due to
losing the GhostHands tracking/motion Playfulness &
Entertainment Empowerment When struggling with or forgetting some
name of the component, you just point at it. The same applies when
you cannot find the proper verb or description
13. 13 Findings (2) Connectedness & Joint accomplishment
Efficiency Autonomy GhostHands was as if I had a pair of expert
hands taking mine and guiding me through the entire process I think
it can help in improving team-work because it produces a sense of
shared accomplishment I felt that sometimes the worker may rely
completely on the instructor being like a robot executing the
instruction without really learning
14. 14 Whats next? Deployment on smart glasses Precision issue
addressing Quantitative analysis on error rate and task completion
time Evaluation in real shop floor environment
