VMT CSCL WorkshopJune 2004

VMT CSCL workshop

Evaluation & analysis

VMT Workshop June 2004 3

Math Forum research theories

Interest and motivation research Social networks and the social

construction of community Imagination of community Activity theory and cultural psychology Socio-cognitive approaches to culture Information society/ globalization theory

VMT Workshop June 2004 4

Evaluation in the VMT project

VMT Project overall (macro level) VMT Working groups (meso level)

Math collaboration (micro level)

• Face-to-face: what is happening?

• Transfer to virtual: what is different?

• Virtual explore: what is happing?

• Virtual support: computer support?

VMT Workshop June 2004 5

Analysis methods

1. Quantitative content analysis

2. Ethnographic analysis

3. Conversation analysis

VMT Workshop June 2004 6

1. Quantitative content analysis

In quantitative content analysis the communication is segmented (if this is applicable), coded, summarized and frequencies/ percentages are used for comparisons and/ or statistical testing.

VMT Workshop June 2004 7

Exploring chat

Quantitative content analysis

• Segmentation – define the units of utterances

• Threading – define the references between

• Coding – categorize along multiple dimensions

• Conversation (based on Beers et al., 2004)

• Social (based on Renninger & Shumar, 2002)

• Problem-solving (based on Polya, 1957)

• Mathematics (plus math accuracy and progress)

• Support (i.e. automated entries and facilitation)

VMT Workshop June 2004 8

Threading 1: Conversational

Conversational dimension

Social dimension

Support dimension

Threading 2: Problem solving

Problem solving

Mathematical moves

Coding scheme

VMT Workshop June 2004 9

Reliability: Beyond dyadic chat

Threading affects reliability• Conversational threading (C-thread)

• Problem solving threading (P-thread)

Preliminary reliability• C-thread

• Conversation

• Support

• Social

VMT Workshop June 2004 10

Preliminary C-thread reliability

Independent To agreed threading

A × B A × C B × C A × G B × G C × G

Thread or no thread assigned

77.8% 83.2% 75% 86.2% 87% 91.8%

Same thread or no thread assigned

63.4% 67.8% 65% 75% 73% 83.2%

Different threads assigned

27.2% 26.8% 31.3% 19% 24% 13.3%

VMT Workshop June 2004 11

Preliminary reliabilities

Percentage

A × B A × C B × C

Conversation 50.6% 75% 69.2%

Social 63.4% 55% 52.4%

Support 56% 78.5% 60%

VMT Workshop June 2004 12

Coded chat excerpt (1)

3 AME has entered the chat room. 7:00:10 nc en 4 FIR has entered the chat room. 7:00:11 0:00:01 nc en 5 MCP has entered the chat room. 7:00:12 0:00:01 nc en 6

MFP

Let's go around and have everyone share a greeting with the group. I'll start by saying that I'm really looking forward to seeing you talk about math tonight! 7:00:14 0:00:02 nc sf

7 MCP Hi! 7:00:49 0:00:35 s gr 8 AME hello 7:00:55 0:00:06 s gr 9 FIR Hey! 7:00:58 0:00:03 s gr

VMT Workshop June 2004 13

Coded chat excerpt (2)

32

AME I have an idea that might help us find whats wrong with the pic.

7:06:19

0:00:49 s is 33

MCP We could use good ol' Pythag thm to see what BV is

7:06:30

0:00:11 o cg s m 34 AME Lets not 7:06:40 0:00:10 33 d cg 33 rf 35 MCP What's your idea? 7:06:46 0:00:06 32 rq ci 32 36

AME It states that something is wrong with the pic.

7:07:01

0:00:15 35 e 35 o 37 AME so we can't find what BV is 7:07:08 0:00:07 36 el cg 36 t 38

MCP Yeah, and I think if we 'found' BV, it would be something not possible.

7:07:31

0:00:23 37 o cg 37 t 39 MCP 16 + BV^2 = 21.16 7:08:10 0:00:39 o 33 p m 40 MCP BV^2 = 5.16 7:08:20 0:00:10 39 el 39 p m 41 AME I got it 7:08:23 0:00:03 se 42 AME I know whats wrong with the pic 7:08:29 0:00:06 41 s is 43 MCP BV = 2.27 7:08:31 0:00:02 39 el 39 r m 44 FIR ok. now i'm following! 7:08:44 0:00:13 39 f ci 39

VMT Workshop June 2004 14

Coded chat excerpt (3) 59 AME What did you say BV was? 7:10:51 0:00:08 58 rq ci 43 60 FIR 2.27 7:11:05 0:00:14 59 rp 59,43 re 61

MCP With the numbers given, BV would be 7:11:10 0:00:05 59 rp 59

62 MCP yeah 7:11:11 0:00:01 60 f 60 63 AME I think thats wrong 7:11:14 0:00:03 60 d ci 60 ch 64 FIR how so? 7:11:19 0:00:05 63 rq 63 65

AME I know whats wrong with the pic 7:11:28 0:00:09 s is

66 MCP base would be twice that 7:11:31 0:00:03 61 o 61 r m 67 FIR what 7:11:33 0:00:02 65 rq 68 MCP 4.54 ish 7:11:41 0:00:08 66 et 66 69 AME The diagnol is not 4.6 7:11:45 0:00:04 67 e 67 t m 70 MCP Right 7:11:51 0:00:06 69 f 69 71 FIR exactly 7:12:02 0:00:11 69 a 69 ch 72

MCP Otherwise, the red lines and the base are almost an equilateral triangle

7:12:14 0:00:12 70 o 69 rf

m

VMT Workshop June 2004 15

Ethnographic analysis

An interpretative method where data is produced through an encounter between the ethnographer and the ‘Other’.

The analysis provides ‘thick descriptions’ of social situations.

Uses theoretical frameworks combined with interpretative methods in order to understand the ‘natives’ point of view with a social context.

VMT Workshop June 2004 16

Ethnography in VMT

Using a modified grounded approach Traditional grounded theory is inductive Modified is more theory driven This work will be developed more this

summer

VMT Workshop June 2004 17

Grounded analysis of chat

Looking for connections across lines of text Grounded analysis is relational where coding is

discrete Coding gives an important quantitative picture

of what people are doing Grounded analysis allows us to think about

what the actors are thinking at particular moments

VMT Workshop June 2004 18

Analysis of video clips

Looking at how people communicate with their bodies in space and time

Contrast with online world where extra-linguistic cues are different

VMT Workshop June 2004 19

3. Conversation analysis

A qualitative, explicitly interpretive analysis of the social practices or interaction methods used by the participants in a conversation to accomplish what they are doing. Typically stresses the sequentiality of conversational moves and how shared meaning is created and negotiated through tacit practices.

Ex.: yesterday’s data session interpreting the video clips of taxicab geometry problem solving.

VMT Workshop June 2004 20

Conversation analysis in VMT

Bridges the content analysis and ethnographic approaches

Looks closely at the micro-cultural interactions

Allows us to link the specific discursive processes of meaning production with larger social issues, i.e. mathematical thinking or resistance to mathematics

VMT Workshop June 2004 21

Future research in VMT

Design research: consecutive iterations to improve overall design

Understand the practices of online math collaboration

Specify the requirements for software design and pedagogy

Design problems, software and service Evaluation in PoW-wow! context