DMW 2007 The Open University's Institute for Educational Technology Open Mentor: Supporting tutors with their feedback to students Denise Whitelock Open

DMW 2007 The Open University's Institute for Educational Technology

Open Mentor: Supporting tutors with their feedback to students

Denise WhitelockOpen University, Walton Hall, Milton Keynes

[email protected] Watt

The School of Computing, The Robert Gordon University, Aberdeen

[email protected]


Formative feedback for tutors: OpenMentor – a training and monitoring e-Tool?

• What implicit /explicit models do tutors have about marking assignments?

• What training is given to tutors about marking assignments?

• Do we hold the company line when marking?

• Tutor questionnaire


What is Open Mentor?

“An open source mentoring tool for tutors”

“Open source” = free and easy to use, and to embed in an institutions infrastructure and working practices

“mentoring” = designed to help people learn how to give feedback effectively, through reflection and social networks

“tutors” = primarily intended for teaching staff, but with clear applications for those involved in quality


Coding into Categories

• Bales analysis

• Psychology 1950s

• Analyses talk

• Includes socio-emotive categories

• Flander’s (1970)categories inappropriate as also includes classroom control


Bales Categories

• Four main groupings

• A. Positive reactions; agreeing and boosting the other person

• B. Directing/teaching

• C. Questions: requesting information, clarification etc

• D. Negative reactions: disagreement


Coding the comments

Categories Specific Examples

Positive Reactions

A1

A2

A3

1. Shows solidarity

2. Shows tension release

3. Shows agreement

Jokes, gives help, rewards others

Laughs, shows satisfaction

Understands, concurs, complies, passively accepts

Attempted Answers

B1

B2

B3

4. Gives suggestion

5. Gives opinion

6. Gives information

Directs, proposes, controls

Evaluates, analyses, expresses feelings or wishes

Orients, repeats, clarifies, confirms

Questions

C1

C2

C3

7. Asks for information

8. Asks for opinion

9. Asks for suggestion

Requests orientation, repetition, confirmation, clarification

Requests evaluation, analysis, expression of feeling orwishes

Requests directions, proposals

Negative Reactions

D1

D2

D3

10. Shows disagreement

11. Shows tension

12. Shows antagonism

Passively rejects, resorts to formality, withholds help

Asks for help, withdraws

Deflates others, defends or asserts self

Bales’ Interaction Process


Identifying trends: H801

0 5 10 15 20 25

A Pass 1

A Pass 2

A Pass 3

A Pass 4

B Pass 1

B Pass 2

B Pass 3

B Pass 4

C Pass 1

C Pass 2

C Pass 3

C Pass 4

D Pass 1

D Pass 2

D Pass 3

D Pass 4

Ba

les'

In

tera

cti

on

al

Ca

teg

ori

es

at

ea

ch

Pa

ssL

leve

l

Number ofIincidences

Graph to show conflated Bale’s categories against mean number of incidences in H801 scripts


Identifying trends: H801

5.96

17.13

5.73

1.61

A

B

C

D

Pie Chart to show the mean number of incidences per pass per conflated Bales' Interactional Category for all four levels of pass in H801 scripts

Key:

A = Positive reactions

B = Responses

C = Questions

D = Negative reactions


How general is the model?

• S103 – Foundation Science Course

• B820 – Masters Business School Course


Identifying trends

A

B

C

D

A

B

C

D

A

B

C

D

Pie Charts to show the mean number of incidences per conflated Bales Interactional Category for ‘Pass 1’ and ‘Pass 4’ in the following courses:

Key:

A = Positive reactions C = Questions B = Responses D = Negative reactions

Pass 4

Pass 1

B820 S103

A

B

C

D

A

B

C

D

A

B

C

D

H801


Explaining Open Mentor’s Rules

• Four categories

• A – Positive Reactions

• B – Attempted Answers

• C – Questions

• D – Negative Reactions


‘A’ - Positive Reactions

Category Examples of Rules Examples of commentsA - Positive Reactions1. Shows solidarity A1 ...excellent... Excellent Conclusions.

A1 ...(good|comprehensive)... Good, you are drawing on hard facts here.A1 ...nicely... Very nicely stated. Your analysis is thorough and your

conclusions consistent regarding the attractiveness of the budget airline sector. This is a good example of critical thinking.

A1 ...well presented... Very well presented diagram with interesting information.A1 ...effective use... Effective use of the case material here.A1 …well (structured|stated)… Report very well structured.A1 ...(well|clear)(ly)*

(structured|structure|summary| summarised|presented|presentation)...

The corporate vs. business unit strategy is well presented and nicely tied to strategies.

A1 ...reasonable.... A reasonable structure as listed in your table of contents.A1 ...useful point(s)... Generally useful points in this section.

2. Shows tension release A2 ... a helpful...

A2 …(thanks|thank you)…3. Shows agreement A3 ...yes... Yes, the intellectual reactions are both real.

A3 ...indeed... Indeed – if it has one basic strategy it is surely differentiation, though it still has to control costs.

4. Praise then direction A4 good...but... Good model, good quote, but be careful about what industry you analyse ??


‘B’ Attempted Answers

Category Rules Examples of commentsB - Attempted Answers

B4 …perhaps… Perhaps even better here to explain the link in your mind between "analysis of stratgeies" and "strategic issues".

B4 …requires…B4 …take care… Take care with your STEP analysis not to make it too industry

focussed.B4 …useful to… Innovation is closely linked to structure and culture- it would be

useful to see some book 6-8 concepts here too.

B4 …you (might|ought)… You ought to have explicitly stated these.

B4 Don’t|never … Don’t introduce new frameworks just for the sake of it in the conclusion. The conclusion should be pulling together what went before.

B4 Please (see|refer to|look at)... Please make sure to read and understand the question correctly

5. Gives opinion B5 I (am|think)... I think I can see where you are going, though a numbered report format might have demonstrated the approach better

B5 This is.... this is an introduction rather than a “summary”B5 ...sounds...like... This sounds as if it could be very popular!!B5 ...not sure... I am not sure about the balance between the environmental

analysis and the review of the resources, capabilities (power, culture, structures and systems) as raised in the question.

B5 I (thought|agree|suggest)… I thought it was because they did not need any external input and saw a significant market sector they could address themselves.

B5 I (do|don’t) think… I don’t think this exercise has helped to develop your analysis. I also think that the development of the perspectives is superficial

6. Gives information B6 …(demonstrates|shows) this….B6 Also… Also, cross link to Leadership issues, Pettigrew on Strategic

Thinking tooB6 ...Q1... etc Q1 = 59/100


‘C’ Questions

Category Rules Examples of commentsC - Questions7. Asks for information C7 ...?... Is a reference and date needed here for the fit quote?

C7 Why... why is this a useful model to use?C7 (What|where|how|can)… How might you define the boundaries of the airline industry –

passenger carriers only ? charter, scheduled, no frills ? European or global ?

C7 In what…. What has caused this break? Has something changed in the environment?

C7 And where... Where does the radar data come from?8. Asks for opinion C8 what conclusions... What conclusions do you draw from using the model? Is the

industry attractive?


‘D’ Negative reactions

Category Rules Examples of commentsD - Negative Reactions

10. Shows disagreement D10 ...no... I've read no justification for this output!!!

D10 ...not the (point|question)... A bit of an esoteric argument about what strategy means - that is not the question posed in Part B

D10 …says nothing… This sentence says nothing.D10 …not clear… It is not clear how success factors equate to attractiveness. You

need to show the link more clearlyD10 …missed the point… You have somewhat missed the point of STEP analysis here –

STEP is about the macro issues, ie those that affect all businesses in all industries not just those relevant to this industry.

D10 …very shallow… Very shallow.D10 I disagree… I disagree. Analysis of the copier market as it then was showed a

monopoly player earning excess economic rent.


Example Rules

• ...you could... (B4)

• ...good...but... (B4)

• ...excellent analysis... (A1)

• ...this...does not.... (D10)

• ...what are... (C7)

• ...in my opinion... (B5)


Building the rule set

Comments Classified from Training Data

0102030405060708090

100

1 2 3 4 5 6 7 8

Training Iteration

% o

f co

mm

ents

cla

ssif

ied


Is the rule set generic?

Comments Classified from Test Data

0102030405060708090

100

B820 TestSet 1

B820 TestSet 2

B820 TestSet 3

A850 M878 S809

Test Data Set

% o

f co

mm

ents

cla

ssif

ied


Is this a model that works in the way tutors and students expect?

47 students and 44 tutors responded to a questionnaire to test OM’s underlying tutorial model

Findings suggest:

Lower grades should attract more detailed comments and explanation by the tutor (students)

Higher grades should attract more positive comments (students and tutors)

Lower grades attract more questions and suggestions (tutors)

Model supported by pedagogical study


How Open Mentor handles comments

“Good work”

“Yes, well done”

“Yes, but is this useful?”

“Can you explain what you mean”

“This does not follow”

A = positive reactions

A = positive reactions

B = attempted answers, and not a positive reaction

C = questions

D = negative reactions


Open Mentor in action












Inside Open Mentor

Course info

Assess-ments

ClassifierExtractor

AnalyserWeb

interface

Bench-marks


Current and future developments Embedding in institutional practice

Enhancing quality of first year provision

Links to VLEs and information systems

Further development

Support for students

Course evaluation

Open Comment

Ensuring emotional support as well as conceptual guidance for learners

Documents

DMW 2007 The Open University's Institute for Educational Technology Open Mentor: Supporting tutors with their feedback to students Denise Whitelock Open