Evaluating the Effect of Social Evaluating the Effect of Social Conversation on Learning, Conversation on Learning, Interaction, and Perceived Interaction, and Perceived

Interdependence in a Collaborative Interdependence in a Collaborative Math Problem Solving EnvironmentMath Problem Solving Environment

Rohit Kumar, Gahgene Gweon, Mhesh Joshi, Rohit Kumar, Gahgene Gweon, Mhesh Joshi, Yue Cui, Adaeze Nwaigwe, Carolyn P. RosYue Cui, Adaeze Nwaigwe, Carolyn P. Roséé

Language Technologies InstituteLanguage Technologies Institute and and Human-Computer Interaction InstituteHuman-Computer Interaction Institute

Carnegie Mellon UniversityCarnegie Mellon University

Why study help?Why study help?

Offering of deep help and elaborated Offering of deep help and elaborated explanations predicts post test explanations predicts post test performance (e.g., Webb et al., 2002)performance (e.g., Webb et al., 2002)

Help behavior mediates learning (e.g., Help behavior mediates learning (e.g., Gweon et al., 2006; Gweon et al., Gweon et al., 2006; Gweon et al., 2007)2007)

Exchanging help in the context of Exchanging help in the context of collaborative learning increases collaborative learning increases identification with a learning identification with a learning community and motivation, and community and motivation, and improves race relations (Sharan, 1980)improves race relations (Sharan, 1980)

Providing help leads to feelings Providing help leads to feelings empowerment for “low status” students empowerment for “low status” students (Elbers & Hann, 2004)(Elbers & Hann, 2004)

How can we prompt helping How can we prompt helping behavior?behavior?

Explicit prompts (Gweon et al., 2006)Explicit prompts (Gweon et al., 2006) Manipulating availability of help from problem Manipulating availability of help from problem

solving environment (Gweon et al., 2007)solving environment (Gweon et al., 2007) Girls offered more help with delayed feedback from Girls offered more help with delayed feedback from

environmentenvironment Boys offered more help with immediate help from the Boys offered more help with immediate help from the

environmentenvironment Current study:Current study:

Will students follow the example of conversational Will students follow the example of conversational agents offering conceptual help?agents offering conceptual help?

Will social prompts affect how students feel and Will social prompts affect how students feel and behave?behave?

Chat AgentChat AgentTutor:Tutor: Another way to think of dividing by fractions is this: If I have Another way to think of dividing by fractions is this: If I have

1 divided by 1/2, can I turn that into something divided by 1?. If 1 divided by 1/2, can I turn that into something divided by 1?. If I want to turn the denominator, which is 1/2, into 1, what do I I want to turn the denominator, which is 1/2, into 1, what do I need to do to it? need to do to it?

Student 1:Student 1: yep yep Tutor:Tutor: Actually, you need to multiply by two. And If I multiply the Actually, you need to multiply by two. And If I multiply the

denominator by 2, what do I have to do to the numerator?denominator by 2, what do I have to do to the numerator?Student 1:Student 1: 2 2 Tutor:Tutor: Right. So 1 divided by 1/2 is the same as 2 divided by 1. Right. So 1 divided by 1/2 is the same as 2 divided by 1.

So that is one reason why the flip and multiply strategy for So that is one reason why the flip and multiply strategy for dividing by fractions works. If you multiply the denominator by dividing by fractions works. If you multiply the denominator by something to turn it into 1, you have to multiply the top by the something to turn it into 1, you have to multiply the top by the same value. So let's try another one. If I want to divide same value. So let's try another one. If I want to divide something by 2/3, what do I need to multiply the numerator and something by 2/3, what do I need to multiply the numerator and denominator by? denominator by?

Student 1:Student 1: 3 3 Tutor:Tutor: Actually, it is 3/2. You should think about why it's 3/2. Actually, it is 3/2. You should think about why it's 3/2.

Collaborative Problem Solving EnvironmentCollaborative Problem Solving Environment

Jan packed several books to amuse herself on along car ride to visit her grandma. After 1/5 ofthe trip she had already finished 6/8 of the booksshe brought. How many times more books shouldshe have brought than what she packed?

Extraneous Entertainment?Extraneous Entertainment?

Connection with VMT WorkConnection with VMT Work

Chat analysis from VMT sessionChat analysis from VMT session ““Class clown” at first appeared to be Class clown” at first appeared to be

“goofing off”“goofing off” Later reflection revealed he was a tone Later reflection revealed he was a tone

setter, putting his fellow students at easesetter, putting his fellow students at ease His light hearted behavior supported his His light hearted behavior supported his

fellow students in foraging ahead to solve fellow students in foraging ahead to solve difficult problemsdifficult problems

Experimental DesignExperimental Design

Participants: 30 6Participants: 30 6thth graders from a suburban graders from a suburban elementary schoolelementary school

Learning in pairs with or without social promptsLearning in pairs with or without social prompts 2 Condition between subjects design2 Condition between subjects design

Dependent MeasuresDependent Measures Tests and QuizzesTests and Quizzes QuestionnaireQuestionnaire Process analysis of conversational behaviorProcess analysis of conversational behavior

Experimental ProcedureExperimental Procedure

Day 1Day 1 Pre-testPre-test

Day 2Day 2 45 minute lab session45 minute lab session Quiz 1Quiz 1

Day 3Day 3 45 minute lab session45 minute lab session Quiz 2Quiz 2 QuestionnaireQuestionnaire

Day 4: [separated from Day 3 by a weekend]Day 4: [separated from Day 3 by a weekend] Post-testPost-test

QuestionnaireQuestionnaire

Perceived problem solving competence of self Perceived problem solving competence of self and partnerand partner

Perceived benefit of collaborationPerceived benefit of collaboration Perceived help receivedPerceived help received Perceived help providedPerceived help provided

All questions were answered with a 6 point likert All questions were answered with a 6 point likert scale ranging from 0 (strongly dissagree) to 6 scale ranging from 0 (strongly dissagree) to 6 (strongly agree)(strongly agree)

Questionnaire ResultsQuestionnaire Results

Control Experimental

Perceived Self Competence 4.2 (.56) 4.1 (.23)

Perceived Partner Competence

4.3 (.62) 3.9 (.49)

Perceived Benefit of Collaboration

4.5 (.74) 4.4 (.70)

Perceived Help Received* 1.8 (1.3) 3.3 (.69)

Perceived Help Provided* 1.8 (1.1) 3.1 (1.1)

Test ResultsTest Results

Very weak evidence in favor of Very weak evidence in favor of Experimental conditionExperimental condition

Consistent (but non-significant!) trend for Consistent (but non-significant!) trend for students in the Experimental condition to students in the Experimental condition to learn morelearn more

Marginal advantage for Experimental Marginal advantage for Experimental condition on Lab day 2 on Interpretation condition on Lab day 2 on Interpretation problems (p=.06, effect size .55 s.d.)problems (p=.06, effect size .55 s.d.)

Coding Scheme (Gweon et al., Coding Scheme (Gweon et al., 2007)2007)

Help provision mediates learning (Gweon et al., Help provision mediates learning (Gweon et al., 2006; Gweon et al., 2007)2006; Gweon et al., 2007)

(R)(R) Help Requests: “Help me”, “I’m stuck”, “I Help Requests: “Help me”, “I’m stuck”, “I don’t get it.”don’t get it.”

(P)(P) Help Provisions: “Find the common Help Provisions: “Find the common denominator”, “Try the flip and multiply strategy”denominator”, “Try the flip and multiply strategy”

(C)(C) Can’t help: “I don’t know”, “I’m stuck too” Can’t help: “I don’t know”, “I’m stuck too” (D)(D) Deny help: “ask the teacher”, “you’re an Deny help: “ask the teacher”, “you’re an

idiot”, “press the help button”idiot”, “press the help button” (N)(N) Nothing Nothing

Help Request met with Provision of Help Request met with Provision of HelpHelp

Student 1Student 1: What operation do we do?: What operation do we do? <student 2 tries multiplication and gets negative feedback <student 2 tries multiplication and gets negative feedback

from the problem solving environment>from the problem solving environment> <student 2 tries divide and gets positive feedback from <student 2 tries divide and gets positive feedback from

the problem solving environment>the problem solving environment> Student 2Student 2: We divide. Now look at the problem, what is : We divide. Now look at the problem, what is

the other fraction we must divide by?the other fraction we must divide by?

Student 1:Student 1: What do we put on top of the fraction? What do we put on top of the fraction? Student 2:Student 2: Did you find a common denominator? Did you find a common denominator? <student 1 correctly finds the common denominator><student 1 correctly finds the common denominator>

Help request met with Can’t HelpHelp request met with Can’t Help

Student 1:Student 1: Why 16? Why 16? Student 2: Student 2: I don’t know.I don’t know.

Student 1: Student 1: I need help.I need help. Student 2: Student 2: SameSame Student 1:Student 1: 23/2 23/2 Student 2:Student 2: What’s 23/2? What’s 23/2? Student 1:Student 1: 11.5 11.5

Help request met with Deny HelpHelp request met with Deny Help

Student 1: Student 1: I don’t get itI don’t get it Student 2: Student 2: hold onhold on <then student 1 tried something and got negative <then student 1 tried something and got negative

feedback from the problem solving environment.>feedback from the problem solving environment.> <finally student 1 tried something else, which was <finally student 1 tried something else, which was

correct, and got positive feedback from the correct, and got positive feedback from the problem solving environment>problem solving environment>

Student 1:Student 1: I don’t know what to do I don’t know what to do Student 2:Student 2: click on the help button click on the help button

Help request met with No Help request met with No ResponseResponse

Student 1: Student 1: I don’t get itI don’t get it <student 2 tries something and gets negative feedback from <student 2 tries something and gets negative feedback from

the problem solving environment>the problem solving environment> <student 2 tries something else and gets negative feedback <student 2 tries something else and gets negative feedback

from the problem solving environment>from the problem solving environment> <student 2 clicks on the help button><student 2 clicks on the help button> <student 1 tries something that is correct and gets positive <student 1 tries something that is correct and gets positive

feedback from the problem solving environment>feedback from the problem solving environment>

Student 1: Student 1: ?? <student 2 tries something and gets negative feedback from <student 2 tries something and gets negative feedback from

the environment>the environment> <student 1 tries something, which is correct, and gets <student 1 tries something, which is correct, and gets

positive feedback from the environment>positive feedback from the environment>

More Help Related Episodes More Help Related Episodes per Problem in Experimental per Problem in Experimental

ConditionCondition

F(1,15) = 16.8,p < .001F(1,15) = 16.8,p < .001

Mean Experimental = .69, Mean Control = .30Mean Experimental = .69, Mean Control = .30

Effect size = 1 standard DeviationEffect size = 1 standard Deviation

Results from Corpus AnalysisResults from Corpus Analysis

Experimental (Day 1)

Experimental (Day 2)

Control (Day 1)

Control (Day 2)

Total Episodes* 47.1 (8.2) 61.3 (12.3) 33.8 (17.9) 49.1 (26.9)

Social Prompt Episodes*

24.1 (9.9) 33.7 (16.2) 0 (0) 0 (0)

Help Episodes (P)Unsolicited

.79 (1.6)1.7 (2.1)

.36 (1.1)3.2 (6.0)

1 (1.3)2.1 (3.2)

1.4 (2.9)1.9 (3.2)

Unanswered Help Requests (C+R+N)

2.4 (2.7) 1.4 (1.9) 2.2 (1.9) 1.4 (1.4)

Non-Help Episodes* 19.9 (5.6) 35.8(9.3) 30.6 (16.3) 46.3 (25.1)

Proportion on Non-Help Related Proportion on Non-Help Related EpisodesEpisodes

Solicited Help

Unsolicited Help

UnansweredRequests

Non-help episodes

Experimental (Day 1)

Solicited Help

Unsolicited Help

UnansweredRequests

Non-help episodes

Experimental (Day 2)

Solicited Help

Unsolicited Help

UnansweredRequests

Non-help episodes

Control (Day 1)

Solicited Help

Unsolicited Help

UnansweredRequests

Non-help episodes

Control (Day 2)

80.0%

87.8%

85.2%

90.8%

What was happening in non-help What was happening in non-help episodes?episodes?

Coordination Coordination Who should go nextWho should go next Which parts of the problem each person should be Which parts of the problem each person should be

responsible forresponsible for Regulation of speed (Go! Slow down! Hurry up!)Regulation of speed (Go! Slow down! Hurry up!)

Complaining about being hungry or boredComplaining about being hungry or bored Joking about the content of the problemsJoking about the content of the problems Cheering about finishing problemsCheering about finishing problems

More talk about winning or losing in the control conditionMore talk about winning or losing in the control condition More references to “we”, “us”, and “our” in the More references to “we”, “us”, and “our” in the

experimental conditionexperimental condition Insulting each otherInsulting each other

Insults like “loser”, “you stink”, and “stupid” only occurred Insults like “loser”, “you stink”, and “stupid” only occurred in the Control condition!in the Control condition!

Negative Affect in Control ConditionNegative Affect in Control Condition

Student 1:Student 1: finally finally Student 2:Student 2: Shut up Shut up Student 1:Student 1: oooooooooo burn oooooooooo burn Student 2:Student 2: I don't like you I don't like you Student 1:Student 1: fine be that way fine be that way Student 2:Student 2: how did you get that how did you get that Student 1:Student 1: Guessing Guessing Student 2: Student 2: good, do you got it? good, do you got it? Student 1:Student 1: no no Student 2: Student 2: well too bad well too bad

What did we learn?What did we learn?

Social prompts affected student attitudes towards Social prompts affected student attitudes towards each othereach other More insults and competitive attitude in the Control More insults and competitive attitude in the Control

conditioncondition

Social prompts fostered more perceived inter-Social prompts fostered more perceived inter-dependence (based on questionnaire)dependence (based on questionnaire)

Social prompts increased the proportion of help Social prompts increased the proportion of help related episodes per problemrelated episodes per problem

Possible learning effectPossible learning effect Social factors affect learningSocial factors affect learning

TagHelper toolsTagHelper toolshttp://www.cs.cmu.edu/~cprose/TagHelper.htmlhttp://www.cs.cmu.edu/~cprose/TagHelper.html

Supporting data analysis involving Supporting data analysis involving conversational dataconversational data

Supporting on-line assessmentSupporting on-line assessment Triggering interventionsTriggering interventions

TagHelper

Labeled Texts

Unlabeled Texts

Labeled Texts

A Model that can Label More Texts

Time

Hel

p

Student1:Student1: I don’t I don’t understand what to understand what to do next.do next.Student2:Student2: You’re an idiot. You’re an idiot.Support Agent:Support Agent: Student2, Student2, it looks like your partner it looks like your partner could use some help.could use some help.

Supporting Data Analysis for Supporting Data Analysis for helping behaviorhelping behavior

Peer tutoring in Algebra Peer tutoring in Algebra LearnLabLearnLab

Data coded for high-level-Data coded for high-level-help, low-level-help, and no-help, low-level-help, and no-helphelp Important predictor of learning Important predictor of learning

(e.g., Webb et al., 2003)(e.g., Webb et al., 2003)

TagHelper achieves TagHelper achieves agreement of .82 Kappaagreement of .82 Kappa

Can be used for follow-up Can be used for follow-up studies in same domainstudies in same domain

* Contributed by Erin Walker

Example of Triggering InterventionsExample of Triggering Interventions

Collaborative idea Collaborative idea generation in the Earth generation in the Earth Sciences domainSciences domain

Chinese TagHelper Chinese TagHelper learns hand-coded topic learns hand-coded topic analysisanalysis Human agreement .84 Human agreement .84

KappaKappa TagHelper TagHelper

performance .7 Kappaperformance .7 Kappa Trained models used in Trained models used in

follow-up study to trigger follow-up study to trigger interventions and interventions and facilitate data analysisfacilitate data analysis

Example DialogueExample Dialogue

SpeakerSpeaker TextText

Student 1Student 1 People stole sand and stones to use for construction.People stole sand and stones to use for construction.

VIBRANTVIBRANT

Yes, steeling sand and stones may destroy the balance and Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development of mountain areas, can you think of a kind of development that may cause a problem? development that may cause a problem?

Student 2Student 2 Development of mountain areas often causes problems.Development of mountain areas often causes problems.

Student 1Student 1 It is okay to develop, but there must be some constraints.It is okay to develop, but there must be some constraints.

* Feedback during idea generation increases learning and idea generation productivity (except during the first 5 minutes) (Wang et al., 2007)

Pairs+Feedback

Individuals+NoFeedback

Pairs+NoFeedback

Individuals+Feedback

Process AnalysisProcess Analysis

Process loss Pairs vs Individuals:Process loss Pairs vs Individuals:F(1,24)=12.22, p<.005, 1 sigmaF(1,24)=12.22, p<.005, 1 sigma Process loss Pairs vs Individuals: Process loss Pairs vs Individuals:

F(1,24)=4.61, p<.05, .61 sigmaF(1,24)=4.61, p<.05, .61 sigmaNegativeNegative effect of Feedback: effect of Feedback:F(1,24)= 7.23, p<.05, -1.03 sigmaF(1,24)= 7.23, p<.05, -1.03 sigma PositivePositive effect of feedback: effect of feedback:

F(1,24)=16.43, p<.0005, 1.37 sigmaF(1,24)=16.43, p<.0005, 1.37 sigma

0 5 10 15 20 25 30

02

46

810

12

Time Stamp

#Uni

que

Idea

s

Unique Ideas

Nom+NNom+FReal+NReal+F Pairs+Feedback

Individuals+NoFeedback

Pairs+NoFeedback

Individuals+Feedback

Pairs+Feedback

Individuals+NoFeedback

Pairs+NoFeedback

Individuals+Feedback

Thank you for listening!Thank you for listening!