Workshop @ JTELSS17

12th October 2017 – Aveiro, Portugal

Multimodal Machines

Daniele DI MITRI & Jan SCHNEIDER

What is multimodality?

Pagina 2

Group assignment: divide into groups and find a picture which best

describe the concepts of Multimodality.

What is multimodality?

Pagina 3

https://www.youtube.com/watch?v=rn9V0cN4NWs https://www.youtube.com/watch?v=3kEHg7cYlP0

Pagina 4

How does multimodality work in learning?

Multimodality for robots

Pagina 5

Group assignment:

equip your robot with

multimodal capabilities

They encode

messages

Pagina 6

Multimodality for computers

e.g. 3D models in AR

They decode

sensor inputs

e.g modern AR

headsets like

Hololenes

Observability

Line

INPUT SPACE

OUTPUT SPACE

Observable dimensions,

can be tracked

with sensors

Unobservable

dimensions,

require human interpretation

assessment

Di Mitri, D., Drachsler, H., Specht, M. (2017) From signals to knowledge. A conceptual model for

multimodal learning analytics. In press.

Multimodal data tree

Pagina 8

Famous multimodal

experiments

Put-that-there

Pagina 10

Bolt, R. A. (1980). “Put-that-there”: Voice and gesture at the graphics interface (Vol. 14, No. 3,

pp. 262-270). ACM.ISO 690

https://www.youtube.com/watch?v=RyBEUyEtxQo

Emotion sensors go to school

Pagina 11

Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009,

July). Emotion Sensors Go To School. In AIED (Vol. 200, pp. 17-24)

Multimodal action based assessment

Pagina 12

Worsley, M., & Blikstein, P. (2013, April). Towards the development of multimodal action based

assessment. In Proceedings of the third international conference on learning analytics and

knowledge (pp. 94-101). ACM.ISO 690

Analytics meet Patient Manikins

Pagina 13

Martinez-Maldonado, R., Power, T., Hayes, C., Abdipranoto, A., Vo, T., Axisa, C., and

Buckingham-Shum, S. (2017) Analytics Meet Patient Manikins: Challenges in an Authentic

Small-Group Healthcare Simulation Classroom. International Conference on Learning Analytics

and Knowledge, LAK 2017, 90-94.

Multimodal experiments

@OUNL

Learning Pulse – are you in the Flow?

Pagina 15

(Csikszentmihalyi, 1972)

Di Mitri, D., Scheffel, M., Drachsler, H., Börner, D., Ternier, S., & Specht, M. (2017). Learning

Pulse: a machine learning approach for predicting performance in self-regulated learning using

multimodal data.ISO 690

Presentation Trainer

Pagina 16

Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015, November). Presentation

trainer, your public speaking multimodal coach. In Proceedings of the 2015 ACM on

International Conference on Multimodal Interaction (pp. 539-546). acm.ISO 690

https://www.youtube.com/watch?v=ElB6OvbL8fA

WEKIT.eu project

Pagina 17

https://www.youtube.com/watch?v=RkGXdpSB_AM

Calligraphy Learning

Pagina 18

Q&A

Thanks for listening!

Daniele Di Mitri & Jan Schneider