Automated content analysis of reflective writing

Thomas Ullmann, Institute of Educational Technology

CALRG seminar 05 May 2016 1

Overview

●Theory

●Method

●Evaluation

●Conclusion

2

Importance of reflection and its detection

Reflection: core to educational practice

● UK Quality Assurance Agency (QAA)

● Organisation for Economic Co-operation and Development (OECD)

● Programme for International Student Assessment (PISA)

Grand challenges in TEL

● E-assessment and automated feedback

3

Which side is reflective?

I need to tell her honestly about the

tutorial, the feedback and my

disappointment in myself.

I was immediately embarrassed by my

callous attitude especially when so

many people had died and were

injured.

Finally I believe that throughout these

weeks I have learned some interesting

issues about interactive skills and

cross-cultural communications.

4

I will begin by giving some

background information on the family,

I will then go on to identify the

various stressors and explain how

the framework can be applied.

This week we are performing some

mock appraisal interviews in class,

where I will participate as an

interviewee and an observer.

Hughes states that bed-rails should

be avoided due to the risk of injury

caused when the patient climbs over

them and falls to the floor.

Left or right?

Text-based learning analytics

Automated detection of reflective thinking in texts

5

Ullmann, T. D. (2015). Automated detection of reflection in texts. A

machine learning based approach. The Open University. Available

at http://oro.open.ac.uk/45402/

Try the demo: http://qone.eu/reflectr

Reflection Detection

(Classification)

Text as input

Theory

6

Models to analyse reflective writings

7

Ross (1989), Sparks-Langer and Colto (1991), Gore

and Zeichner (1991), Tsangaridou and O’Sullivan

(1994), Hatton and Smith (1995), Richardson and

Maltby (1995), Pultorak (1996), Hutchinson and Allen

(1997), Scanlan and Chernomas (1997), Taylor (1997),

Valli (1997), Bain et al. (1999), Kim (1999), Duke and

Appleton (2000), Rogers (2001), Bain et al. (2002), Jay

and Johnson (2002), Spalding et al. (2002), MacLellan

(2004), Tillema (2004), Thorpe (2004), Ward and

McCotter (2004), Lee (2005), Korthagen and Vasalos

(2005), Lee (2005), Kansanaho et al. (2005), Kreber

(2005), Wessel and Larin (2006), Mann et al. (2007),

Chretien et al. (2008), Kreber and Castleden (2008),

Minott (2008), Wilson (2008), Gulwadi (2009),

Friedman and Schoen (2009), Le Cornu (2009),

Badger (2010), Granberg (2010), Lambe (2011),

Cohen-Sayag and Fischl (2012), Crawford et al.

(2012), Etscheidt et al. (2012), Leijen et al. (2012),

Corlett (2013), Medwell and Wray (2014), McDonald et

al. (2014), Nguyen et al. (2014), Chaumba (2015), Hill

et al. (2015), and McKay and Dunn (2015)

Sparks- Langer et al. (1990), Wong et al.

(1995), Sumsion and Fleet (1996), McCollum

(1997), Kember et al. (1999), Hawkes and

Romiszowski (2001), Hawkes (2001, 2006),

Fund et al. (2002), Hamann (2002), Pee et al.

(2002), Williams (2000), Boenink et al. (2004),

O'Connell and Dyment (2004), Plack et al.

(2005), Ballard (2006), Mansvelder-Lonaryoux

(2006), Mansvelder-Longayroux et al. (2007),

Abou Baker El-Dib (2007), Chirema (2007),

Plack et al. (2007), Kember et al. (2008),

Wallman et al. (2008), Chamoso and Caceres

(2009), Findlay et al. (2010), Lai and Calandra

(2010), Bell et al. (2011), Clarkeburn and

Kettula (2011), Findlay et al. (2011), Fischer et

al. (2011), Birney (2012), Ip et al. (2012), Wald et

al. (2012), Mena-Marcos et al. (2013), Poom-

Valickis and Mathews (2013), Poldner et al.

(2014), Prilla and Renner (2014)

Models to analyse reflective writings

8

Qualities of reflective writings

● Depth dimension (hierarchy of levels)

● Breadth dimension (describes types of reflection)

9

descriptive reflective

?

Synthesis of common categories

Author(s) Experience Feelings Personal Critical Perspective Outcome

Sparks-Langer et al. (1990) ✔ ✔ ✔ ✔

Wong et al. (1995) ✔ ✔ ✔ ✔ ✔

McCollum (1997) ✔ ✓ ✓ ✔ ✔

Kember et al. (1999) ✔ ✔ ✔ ✔ ✔

Fund et al. (2002) ✔ ✔ ✔ ✔ ✔ ✓

Hamann (2002) ✔ ✔ ✔

Pee et al. (2002) ✔ ✔ ✔ ✔

Williams et al. (2002) ✔ ✔ ✔ ✔ ✔

Boenink et al. (2004) ✔ ✔ ✔ ✔

O’Connell and Dyment (2004) ✔ ✔ ✔

Plack et al. (2005) ✔ ✔ ✔ ✔ ✔ ✔

Ballard (2006) ✔ ✔ ✓ ✔

Mansvelder-Longayroux (2006,2007) ✔ ✓ ✔ ✔ ✔

Plack et al. (2007) ✔ ✔ ✔ ✔ ✔ ✔

Kember et al. (2008) ✔ ✔ ✔ ✔ ✔

Wallman et al. (2008) ✔ ✔ ✔ ✔ ✔ ✔

Chamoso and Cáceres (2009) ✔ ✓ ✔ ✔

Lai and Calandra (2010) ✔ ✔ ✔ ✔ ✔ ✔

Fischer et al. (2011) ✔ ✔ ✔ ✔ ✔

Birney (2012) ✔ ✔ ✔ ✔ ✔ ✔

Wald et al. (2012) ✔ ✔ ✔ ✔ ✔ ✔

Mena-Marcos et al. (2013) ✓ ✔ ✔

Poldner et al. (2014) ✔ ✓ ✔ ✔

Prilla and Renner (2014) ✔ ✔ ✓ ✔ ✔ ✔

10

Model for reflection detection

●Depth of reflection● Descriptive vs. reflective

●Breadth of reflection● Description of an experience: Subject matter of the reflective writing

● Feelings: Doubts, uncertainty, frustration, surprise, excitement, etc.

● Personal: One's assumptions, beliefs, knowledge of self

● Critical stance: Critical mindset; awareness of problems

● Perspective: Awareness of other perspectives

● Outcome: Retrospective: lessons learned; prospective: future intentions

11

Claims

1. Machine learning algorithms can be used to

distinguish between descriptive and reflective

text segments (RQ1)

2. Machine learning algorithms can be used to

detect common categories of reflective writings

(RQ2)

12

Method

Method overview

Dataset

Training data Test data

Models Assessment

Text collection

Annotation Task

Annotated units

EvaluationData generation

14

Dataset generation process

15

Text collection

Identifcation of

suitable text collections

Sampling of

text collection

Unitising text collection

Dataset of unlabelled units

Annotation task

Task design Pilots

Quality standard

Rated units

Dataset

Reliability

Validity

Annotated units

Datasets

16

Dataset reliability estimates

17

Reliability annotation task

Simple majority

18

Model validation

19

Correlation between reflection indicator and common categories

Research design

Dataset for machine learning

Training data Test data

Model selection Model assessment

Dataset of labelled units

Data pre-processing Splitting

Feature construction

Feature selection

Oversampled dataset

Resampling

Model tuning

Original class distribution

Pre-processsing Machine learning

20

Evaluation

21

Instantiation of method for RQ1

Can machine learning be used to distinguish between

descriptive and reflective text segments?

22

Rule-based models

Tree-based models

High performance

Reflection

Experience

Feelings

Personal

Critical stance

Perspective

Outcome

Datasets Research designResearch

question

High performance

models

RQ1 I1

RQ1 I2

RQ1 I3

Three lines of investigation to answer research question 1

Evaluation of the common categories of reflection to answer research question 2

RQ2 Exp.

RQ2 Feel.

RQ2 Pers.

RQ2 Crit.

RQ2 Persp.

RQ2 Out.

RQ1 Results

Comparison of the three lines of investigation

23

Instantiation of method for RQ2

Can machine learning algorithms be used to detect common

categories of reflective writing?

24

Experience

Feelings

Personal

Critical stance

Perspective

Outcome

Datasets Research designResearch

question

High performance

models

RQ2 Exp.

RQ2 Feel.

RQ2 Pers.

RQ2 Crit.

RQ2 Persp.

RQ2 Out.

RQ2 Results

Indicator N Cohen’s k % Landis & Koch BM % CA BM

Experience 654 0.83 0.92 Almost perfect Top

Feelings 521 0.73 0.88 Substantial Middle

Beliefs 449 0.66 0.83 Substantial Middle

Difficulties 526 0.60 0.80 Moderate Middle

Perspective 396 0.55 0.88 Moderate Middle

Intention 727 0.71 0.95 Substantial Top

Learning 364 0.63 0.83 Substantial Middle

Reflection 456 0.70 0.89 Substantial Middle

Automated detection of common categories of reflection

25

Comparison of model and dataset

Per cent agreement

26

Conclusion

Conclusion

Machine learning algorithms can be used to distinguish between

descriptive and reflective text segments

Machine learning algorithms can be used to detect common

categories of reflective writings

28

Limitations

● Investigated language

● Investigated unit of analysis

29

H818 The networked practitioner

Introduction to reflective writing to support TMAs and EMA

30

Text-based learning analytics

Automated detection of reflective thinking in texts

31

Ullmann, T. D. (2015). Automated detection of reflection in texts. A

machine learning based approach. The Open University. Available

at http://oro.open.ac.uk/45402/

Try the demo: http://qone.eu/reflectr

Reflection Detection

(Classification)

Text as input

Thank you

32

See for a different approach

Ullmann, T. D. (2015). Keywords of

written reflection - a comparison between

reflective and descriptive datasets. In

Proceedings of the 5th Workshop on

Awareness and Reflection in Technology

Enhanced Learning (Vol. 1465, pp. 83–

96). Toledo, Spain

Keywords of written reflection

33