LaVonda Brown Georgia Institute of Technology LearnLab Workshop 2012 Carnegie Melon University

Model of Engagement for Educational Agents based on Mouse and Keyboard Events

LaVonda BrownGeorgia Institute of Technology

LearnLab Workshop 2012Carnegie Melon University

Int’l Conference on Robotics and Automation 2010Anchorage, AlaskaHumAnS Lab. Presentation2

Purpose Motivation Background Design Hypotheses Results Conclusion Future Work

Outline


Purpose

To develop a reliable, non-invasive method of monitoring academic engagement within the domain of computer-based education (CBE)


Motivation Socially Interactive

Robot Tutor (SIRT)◦ Real-time monitoring of

actions to determine level of engagement and understanding

◦ Adapt to educational needs through instructional scaffolding

◦ Engage children by forming a personal relationship Essential for longevity

Platform


Background Teachers/tutors are able to observe the

student’s engagement in real-time and employ strategies to reengage the student

They are able to determine engagement by following behavioral cues from students

This improves attention, involvement and motivation to learn

Behavioral engagement is often related to the academic achievement of a student


Related Work Scales/Surveys

◦ Used to evaluate motivation once the student has completed a system

Electroencephalography (EEG) signal measurements◦ Able to identify subtle shifts in alertness,

attention, and workload in real time Eye Gaze and Head Pose

◦ Able to determine six user states in an e-learning environment: attentive, full of interest, frustrated/struggling to read, distracted, tired/sleepy, and not paying attention


Design Eye gaze and head pose will be the baseline Will use this to develop a novel model of

student engagement based on mouse and keyboard events.

Two tests of high and low difficulty Three event processes will be monitored

◦ Total Time slow, average, or fast

◦ Response Validity correct or incorrect

◦ Proper Function Execution on-task or off-task


Hypothesis 1 Hypothesis 1. The student is engaged if his

or her series of events (or combination of events) are classified as:◦ On-task and correct (regardless of speed)◦ On-task, slow or average, and incorrect


Hypothesis 2 Eye gaze and head pose will not be an

accurate measure of user state/engagement for the high difficulty test.

The use of pencil and paper will create false negatives since eye gaze will be directed towards the paper instead of the computer screen.


Hypothesis 3 Various combinations of the event processes can

determine the following about the engaged student: ◦ If the student is on-task and has a series of fast

responses with a series of correct answers, the student needs questions of higher difficulty.

◦ If the student is on-task and has a series of slow and/or average responses with a series of correct answers, the student does a great deal of thinking and understands the material.

◦ If the student is on-task and has a series of slow and/or average responses with a series of incorrect answers, the student lacks understanding and needs questions of lesser difficulty


Results: Question Time


Results: Question Time


Results: Test Time


Results: Test Responses

Int’l Conference on Robotics and Automation 2010Anchorage, AlaskaHumAnS Lab. Presentation

Results: Event Combinations

15

This chart shows the how often we received each combination of events throughout both tests (S=slow, A=average, F=fast, C=correct, I=incorrect, O=on-task, O’=off-task). The combinations OIF, O’CS, O’CA, O’CF, O’IS, O’IA, O’IF do not occur during this study.


Results: Eye Gaze


Conclusion In order for a student to be engaged, he or

she must be on-task or choose events that successfully execute functions needed to navigate through the assessment.

However, if a student is on-task, but answers incorrectly, and at a fast pace (OIF), he or she is classified as being disengaged.

If a student is classified as being off-task, we can automatically classify this student as being disengaged (regardless of speed and/or response).


Future Work Add a survey to the end of the experiment to better

determine understanding, engagement, and difficulty.◦ If the student is on-task and has a series of fast responses

with a series of correct answers (OCF), the student may need questions of higher difficulty.

◦ If the student is on-task and has a series of slow responses with a series of correct answers (OCS), the student may understand the material and require more time to think.

◦ If the student is on-task and has a series of slow responses with a series of incorrect answers (OIS), the student may lack understanding and need questions of lesser difficulty.

This additional information will be used in the future to better integrate instructional scaffolding and adaptation with the device.


Questions???

Documents

LaVonda Brown Georgia Institute of Technology LearnLab Workshop 2012 Carnegie Melon University