Learning Motivation.pdf

7/27/2019 Learning Motivation.pdf

1/8

Self-Assessment of Motivation: Explicit and Implicit

Indicators in L2 Vocabulary Learning

Kevin Dela Rosa, Maxine Eskenazi

Language Technologies Institute, Carnegie Mellon University,

5000 Forbes Avenue, Pittsburgh, Pennsylvannia, USA

{kdelaros,max}@cs.cmu.edu

Abstract. Self-assessment motivation questionnaires have been used in

classrooms yet many researchers find only a weak correlation between answers

to these questions and learning. In this paper we postulate that more direct

questions may measure motivation better, and they may also be better

correlated with learning. In an eight week study with ESL students learning

vocabulary in the REAP reading tutor, we administered two types of self-

assessment questions and recorded indirect measures of motivation to see which

factors correlated well with learning. Our results showed that some user actions,

such as dictionary look up frequency and number of times a word is listened to,

correlate well with self-assesment motivation questions as well as with how

well a student performs on the task. We also found that using more direct self-

assesment questions, as opposed to general ones, was more effective in

predicting how well a student is learning.

Key words: Motivation Modelling, Intelligent Tutoring Systems, Computer

Assisted Language Learning, Motivation Diagnosis, English as a Second

Language

1 IntroductionMotivation modelling and its relation to user behavior has receieved attention by the

educational computing community in recent years. William and Burden define

motivation as a state of cognitive and emotional arousal which leads to a conscious

decision to act, and which gives rise to a period of sustained intellectual and/or

physical effort in order to attain a previously set goal (or goals)

[1]. The use of self-assessment questionaires is one common approach to measuring motivation. One such

construct is Motivated Strategies for Learning Questionnaire (MSLQ), an 81-item

survey designed to measure college students' motivational orientations and their use

of various learning strategies [2]. While questionnaires are useful to detect enduring

motivational traits, some are criticized, particularly those administered prior to

interaction. Since a students motivation is likely to change during an interaction, it is

important to use them with other methods to adapt instruction and to gather more

transient information about a students motivation [3]. Other methods of assessing


2/8

2 Kevin Dela Rosa, Maxine Eskenazi

motivation include direct communication with students, emotion detection, and

recorded interactions with an intelligent tutor. For modelling and understanding user

behavior automatically, Baker [4] showed that machine learning models trained on

log data of student activity can be used to automatically detect when a student is off-

task. A study by Cetintas et al. [5] reached a similar conclusion, using a regression

model personalized to each student. And, Baker et al. [6] showed that a latent

response model can be used to determine ifa student is gaming the system in a way

that leads to poor learning.

An important issue has been how to automatically detect a students current

motivational state. As mentioned above, one method of measuring motivation is

questionnaires that cover a variety of motivation aspects. One important consideration

is how detailed and/or direct these survey questions should be with respect to the task

or, in other words, is it better to have questions that are tightly focused on the tasksbeing performed by the student or is it better to construct questions that are more

general and can cover many difference aspects of motivation, such as the MLSQ.

Also, a students usage of a tutoring system, as indicated by the amount of activity

and types of actions taken, may furnish good implicit indicators of a students

motivation.

We propose that in a computer-assisted L2 language learning environment certain

recorded student interactions during learning activities can act as implicit indicators of

that students motivation. We also propose that these implicit indicators, as well as

explicit ones like self-assessment surveys, can be used to predict the amout of

learning that is taking place. Lastly we postulate that more direct questions may

measure motivation better and may also be better correlated with learning.

For this study we used a web-based language tutor called REAP [7]. REAP, whichstands for REAder-specific Practice, is a reading and vocabulary tutor targeted at

ESL students, developed at Carnegie Mellon University, which uses documents

harvested from the internet for vocabulary learning. REAPs interface has several

features that help to enhance student learning. One key feature in REAP is that it

provides users with the ability to listen to the spoken version of any word that appears

in a reading, making use of Cepstral Text-to-Speech1 to synthesize words on demand

when they are clicked on. Additionally, students look up the definition of any of the

words, during readings, using a built-in electronic dictionary. REAP also

automatically highlights focus words, the words targeted for vocabulary acquisition in

a particular reading. REAP is a language tutor and a testing platform for cognitive

science studies [8, 9], as is the case of this study.

In this paper we describe a classroom study that compares the effectiveness of

different motivational indicators in a vocabulary learning environment. We define the

different types of survey questions we used as explicit measures of motivation and the

various user actions we recorded as indirect indicators of motivation. Next we

describe the results of a classroom study that integrated our various motivation

indicators and how well they correlated with our learning measures. Finally we

discuss the implications of our results and suggest future directions.

1 Cepstral Text-to-Speech. http://www.cepstral.com


3/8

Self-Assessment of Motivation: Explicit and Implicit Indicators 3

2 Classroom StudyIn order to determine which of our hypothesized indicators of motivation were most

related to learning we conducted a classroom study with a web-based tutor, focused

on L2 English vocabulary learning, and recorded responses to motivation

questionnaires and user actions that we log, which may indirectly indicate a students

motivation level. The classroom study consisted of a pre-test and post-test with

multiple choice fill-in-the-blank vocabulary questions, and six weekly readings, each

followed by practice vocabulary questions similar to, but not the same as those in the

pre-test and post-test. During the pre-test and post-test, a set of seventeen self-

assessment motivation questions were administered, and after each weekly session

there were a set of five motivation questions.

21 intermediate-level ESL college students at the University of PittsburghsEnglish Language Institute participated in the study and completed all of the

activities. For this study the readings and vocabulary questions had 18 focus words,

taken from either the General Service List 2 or the Academic Word List3, and not part

of the class core vocabulary list.

In the following subsections we describe the types of questionnaires administered,

the recorded user actions, and the metrics we used to measure student learning.

2.1 Motivation QuestionnaireWe administered motivation survey questions as explicit measures of motivation after

each reading. 17 survey questions were administered in the pre-test and post-test, as

shown in Table 1, using a five-point Likertscale, with a response of 5 indicating thegreatest agreement with the statement and 1 indicating the least agreement. The 17

questions were divided into two groups: GeneralandDirect.

We call General survey questions high-level survey questions which have been

used in past REAP studies because of their generality; they are used in many studies

in the Pittsburgh Science of Learning Center4. For example, one of the General

questions we used was, When work was hard I either gave up or studied only the

easy parts, which can be used for many different subject matters. The design of these

questions was guided by the MLSQ [2], and aimed to use the fewest number of

questions possible that cover the most motivational constructs.

We call Directquestions the more explicit items that focused on aspects directly

related to the reading activities accomplished over the course of the study. An

example of a Direct question: Learning vocabulary in real documents is aworthwhile activity. This is focused on the specific REAP tasks.

A total of twelve General and five Direct self-assessment motivation questions

were administered during the pre-test and post-tests. Additionally, the five Direct

motivation survey questions in Table 2 were asked after each weekly reading activity,

2 The General Service List. http://jbauman.com/gsl.html3 The Academic Word List. http://www.victoria.ac.nz/lals/resources/academicwordlist4 Pittsburgh Science of Learning Center (PSLC). http://www.learnlab.org


4/8


at regular intervals in between the pre-test and post-test, to see how the responses

correlated with student behavior and learning at each reading. We wanted to

determine if there was a difference in how well each of these two question groups

correlates to the learning measures we recorded (multiple choice questions). We

hypothesize that questions more directly related to the tasks/activities performed will

be better at predicting motivation and learning. This is guided by unpublished results

of past REAP studies which have shown that higher-level questions generally failed

to correlate well with learning measures, and previous success with direct questions

by Heilman et al. [10].

Table 1. Pre-test/Post-test Motivation Survey Questions

ID Survey Question Prompt Group Type

S1 I am sure I understood the ideas in the computer lab sessions. General E

S2 I am sure I did an excellent job on the tasks assigned for the computer labsessions.

General E

S3 I prefer work that is challenging so I can learn new things. General A

S4 I think I will be able to use what I learned in the computer lab sessions in my

other classes.

General V

S5 I think that what I learned in the computer lab sessions is useful for me to know. General V

S6 I asked myself questions to make sure I knew the material I had been studying. General O

S7 When work was hard I either gave up or studied only the easy parts. General A

S8

I find that when the teacher was talking I thought of other things and didn't reallylisten to what was being said. General A

S9

When I was reading a passage, I stopped once in a while and went over what I

had read so far. General O

S10 I checked that my answers made sense before I said I was done. General O

S11 I did the computer lab activities carefully. General E

S12 I found the computer lab activities difficult. General A

S13 I continued working on the computer lab activities outside the sessions. Direct A

S14 I did put a lot of effort into computer lab activities. Direct A

S15 I did well on the computer lab activities. Direct E

S16 I preferred readings where I could listen to the words in the document. Direct V

S17 Learning vocabulary in real documents is a worthwhile activity. Direct V

Table 2. Post-reading Survey Motivation Questions

ID Survey Question Prompt Type

Q1 Did you find the spoken versions of the word helpful while reading this document? V

Q2 Do you find it easy to learn words when you read them in documents? E

Q3 Did you find this document interesting? V

Q4 Did you learn something from this document? V

Q5 Does reading this document make you want to read more documents? A

Furthermore, for this study we grouped the questions into three types:

Affective (A): Deal with emotional reactions to a task Expectancy (E): Deal with beliefs about a students ability to perform a task Value (V): Deal with goals and beliefs about the importance and interestof a

task

The 3 groups are based on the Pintrich and De Groot components of motivation

(self-efficacy, intrinsic value, test anxiety) [11]. We used this grouping to simplify the

analysis of the results. Note that in the tables and figures, Other (O) signifies a


5/8


question that failed to group into one of the three types, typically a question on

learning strategies.

2.2 Recorded User InteractionsIn addition to survey questions, we recorded actions taken by the students which we

hypothesize would indirectly correspond to motivation and might also correlate with

learning. The following were recorded during each activity:

Word lookup activity, using our built-in electronic dictionary

A1: Total number of dictionary lookups

A2: Number of focus words looked up in the dictionary

A3: Number of dictionary lookups involving focus wordsWords listening activity, using our built-in speech synthesis

A4: Mean number of listens per word

A5: Total number of listens

A6: Number of words listened to

Average time spent on activity tasks

A7: Time spent reading the documents

A8: Time spent on practice questions

2.3 Learning MeasuresIn order to assess how well students learned the target vocabulary words, we recorded

the following measures:

L1: Average post-reading practice question accuracy (for all questions appearing

directly after reading the documents)

L2: Pre-test to post-test normalized gain

L3: Post-test accuracy

L4: Average difference between pre-test and post-test scores

Note that that L2 and L4 are two different ways of looking at the improvements made

by students over the course of the study, where L2 is tuned to the relative difference

between the test scores, and L4 is sensitive to the absolute difference in scores.

3 ResultsThe results of our study show that the use of the REAP system significantly helpedstudents improve their performance on the vocabulary tests, as evident in the average

overall gains between the pre-test and post-test {L3} (p < 0.004), whose average

scores were 0.3439 ( 0.0365) and 0.5000 ( 0.0426) respectively. The average post-

reading practice question accuracy was 0.8417 {L1} ( 0.0466). The overall average

normalized gain {L2} between pre-test and post-test was 0.2564 ( 0.0466), and

average difference in score {L4} between the pre-test and post-test was 0.1561 (

0.0232).


6/8


Fig. 1. Significant correlations values between motivational & learning factors, and betweenimplicit & explicit motivation indicators. Color signifies level of significance, with green

representing strong statistical significance (p < 0.05), and yellow representing moderate

signifance (p < 0.1). Note that self-correlations and correlations with low significance values

were omitted. Also note that Q1-Q5 correspond to the students' average survey response values

for all reading activities, and with respect to the implict indicators, A1-A8 correspond to the

students' average values of those indicator values over all reading activties.

In order to find the motivational factors that best correlated with learning, we

computed a Pearson correlation coefficient matrix for the values for the different

factors and grouped question responses, and determined the significance of each pair

of correlations using a two-tailed test. Figure 1 summarizes correlation and

signifcance values found between the motivational and learning factors, and shows

how the indirect indicators correlate with the expicit indicators of motivation.Additionally, when we looked at the post-reading accuracies of each individual

reading activity (as oppose to the overall averages, which were shown in Figure 1),

the following motivational factors tended to significantly correlate with the post-

reading practice question accuracies at significance levels varying levels over the six

readings betweenp < 0.01 andp < 0.05:

Q2 response: Do you find it easy to learn words when you read them indocuments?


7/8


Q3 response: Did you find this document interesting? Total number of dictionary lookupsNumber of focus words looked up in the dictionary Time spent on practice questions

4 DiscussionWe see in Figure 1 that most of the Generalquestions did not significantly correlate

with the various learning measures, while the Directquestions did. In fact, the only

sub-group ofGeneralquestions that correlated with the learning measures was Other,

those questions that did not fit well into our three types (Affective, Expectancy, and

Value), which is not surprising since the Otherquestions mainly focused on learningstrategies as opposed to motivation. Furthermore, all of the direct questions asked

after each reading, except forQ5 (Does reading this document make you want to read

more documents?), had significant correlations with the learning measures. Perhaps

the reason Q5 failed to have significant correlations is due to the fact students were

not given the option to actually act on the desire to read more documents in our tutor

during the semester, due to class constraints. Therefore, our results imply that General

questions are less effective at predicting the learning outcome of a student than are the

Directmotivation questions which are more closely focused on the tasks performed.

Moreover, our results hew closely to past results by Bandura [12] in the domain of

self-efficacy.

Additionally, the implicit motivation indicators, based on recorded student actions,

seemed to correlate well with our learning measures, particularly our word listening

and dictionary lookup-related interactions, which implies that these kinds of actions

can also help in predicting a students motivational state. Interestingly, the amount of

time spent on reading and answering questions did not correlate well with the learning

factors, which implies that simply using the absolute amount of time spent on task

may not be a good factor to use in predicting a students learning outcomes, and

perhaps taking into account how the time was used by students would be a better

factor to consider. Furthermore, most of the implicit indicators we recorded had

significant correlations with one or more of the direct motivation survey questions in

Figure 1 that we asked after each reading, which implies that implicit indicators may

be effective in predicting the motivational state of the student using an intelligent

tutor on an activity-by-activity basis.

5 ConclusionUnderstanding and modeling motivation and student behavior is an important issue

for intelligent tutoring systems. We proposed that some student interactions recorded

by tutors can act as implicit indicators of motivation, and that these implicit

indicators, as well as explicit ones like self-assessment motivation questions, can be

used to predict student learning. We tested our hypothesis with a classroom study

using a vocabular tutor that integrates implicit and explicit indicators. The results


8/8


show that some user actions, such as dictionary look ups and listening to words,

correlate well with motivation questions and student performance. We also found that

the use ofDirectquestions, specifically tailored to the tasks, rather than Generaland

all-encompassing questions, was more effective in predicting student performance.

Acknowledgements. This project is supported by the Cognitive Factors Thrust of the

Pittsburgh Science of Learning Center which is funded by the US National Science

Foundation under grant number SBE-0836012. Any opinions, findings, and

conclusions or recommendations expressed in this material are those of the authors

and do not necessarily reflect the views of the NSF. The authors would like to thankBetsy Davis, Tamar Bernfeld, and Chris Ortiz at the University Pittsburghs ELI for

their participation and input in the studies.

References

1. Williams, M., Burden, R. L.: Psychology for language teachers: A social constructivistapproach. Cambridge University Press, Cambridge,(1997)

2. Pintrich, P.R., Smith, D.A.R., Garcia, T., McKeachie, W.: A manual for the use of themotivated strategies for learning questionnaire (MSLQ). Report, Ann Arbor (1991)

3. de Vincente, A., Pain, H.: Motivation Diagnosis in Intelligent Tutoring Systems. In: 4thInternational Conference on Intelligent Tutoring Systems (1998)

4. Baker, R.S.: Modeling and understanding students' off-task behavior in intelligent tutoringsystems. In: 25th ACM SIGCHI Conference on Human Factors in Computing Systems,

pp. 10591068. ACM, New York (2007)

5. Cetintas, S., Si, L., Xin, Y.P., Hord, C.: Automatic Detection of Off-Task Behaviors inIntelligent Tutoring Systems with Machine Learning Techniques. IEE Trans. Learn. Tech.3, 228-236 (2010)

6. Baker, R.S., Corbett, A.T., Koedinger, K.R.: Detecting Student Misuse of IntelligentTutoring Systems. In: Lester, J.C., Vicari, R.M., Paraguau, F. (eds.) ITS 2004. LNCS,

vol. 3220, pp. 5476. Springer, Heidelberg (2004)

7. Heilman, M., Collins-Thompson, K., Callan, J., Eskenazi M.: Classroom success of anIntelligent Tutoring System for lexical practice and reading comprehension. In: 9thInternational Conference on Spoken Language (2006)

8. Dela Rosa, K., Parent, G., Eskenazi, M.: Multimodal learning of words: A study on the useof speech synthesis to reinforce written text in L2 language learning. In: 2010 ISCA

Workshop on Speech and Language Technology in Education (2010)

9. Kulkarni, A., Heilman, M., Eskenazi, M., and Callan, J.: Word Sense Disambiguation forVocabulary Learning. In: 9th International Conference on Intelligent Tutoring Systems

(2008)

10. Heilman, M., Juffs, A., Eskenazi, M.: Choosing Reading Passages for VocabularyLearning by Topic to Increase Intrinsic Motivation. In: 13th International Conference on

Artificial Intelligence in Education (2007)

11. Pintrich, P.R., De Groot, E.V.: Motivational and Self-Regulated Learning Components ofClassroom Academic Performance.J. of Ed. Psych.82, 3340(1990)

12. Bandura, A.: Self-Efficacy. In: Ramachaudran, V.S. (ed.) Encyclopedia of HumanBehavior, vol 4, pp. 7181. Academic Press, New York (1994)

Documents

Learning Motivation.pdf