Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Evaluating Portfolios: Mirrors Maps and Sonnets Simon Cotterill Sue Gill Jamie Thompson EPICS ePortfolios

Evaluating Evaluating PDP & PDP & ePortfolios:ePortfolios:

MirrorsMirrors

Maps & Maps & SonnetsSonnets

Evaluating Portfolios:Mirrors Maps and Sonnets

Simon Cotterill

Sue Gill

Jamie Thompson

EPICS ePortfolios project& PDP Forum

http://www.epics.ac.uk


MirrorsMirrors

Maps & Maps & SonnetsSonnets • Simon meets Jamie

• ePortfolio meets PDP

• Debate widens

• EPICS and the NE Regional Forum

• Philosophy, Practice and Evaluation

• Symposium

The story so far


MirrorsMirrors


EPICS - North East regional collaboration around e-portfolio progression pathways with illustrative studies

Part of the JISC Distributed e-Learning Programme.

Partners include: 5 HEIs FE colleges (OWL network) JISC Regional Centre

Case studies-demonstrate feasibility / scalability-supporting life-long learning



MirrorsMirrors


to improve(iterative)

What do we mean by What do we mean by “Evaluation” & “Research”?“Evaluation” & “Research”?

• Evaluating which ePortfolio product to use-checklists / criteria eg. Richardson & Ward, 2005

• Formative Evaluation (independent or practitioner) • Action Research (practitioner)

• Summative Evaluation – making a judgment; did the intervention achieve its stated goal(s)

• Research (= rigorous scholarly evaluation?)– tends to be more grounded in theory– narrow, well defined research question– prospective research plan– aims to be reproducible (difficult in complex

educational environments)

}

Before we begin…


MirrorsMirrors


Clarity of PurposeBack to basics:

• Evaluation should address the main aims / purpose(s) of your portfolio / PDP– What is the purpose? – How will you know if it has been achieved? i.e. measure it? (quantitatively and/or qualitatively)

• Use this to inform your evaluation strategy

• Are there existing (validated) research instruments that will address your questions? e.g. ELLI (life-long learning)

e.g. ASSIST (approaches to learning)

e.g. Bloom’s taxonomy (reflective learning)


MirrorsMirrors


Diez, M. (1994) The Portfolio: Sonnet, Mirror and Map in Burke, K. (1996) Professional Portfolios. Arlington Heights: IRI Skylight


MirrorsMirrors


See also

• Story, Journey, Laboratory (P and L Paulson)

• Positivist v Constructivist (Wilkerson and Lang)

• My digital clone, work companion, butler, dashboard, planner, IPR Management assistant (Serge Ravet)

• Toothbrush, Caterpillar, Kaleidoscope, Window


MirrorsMirrors


Paulsons (1994)• Positivist Portfolios

“The purpose of the portfolio is to assess learning outcomes and those outcomes are, generally, defined externally. Positivism assumes that meaning is constant across users, contexts, and purposes… The portfolio is a receptacle for examples of student work used to infer what and how much learning has occurred.” (p.36)

• Constructivist Portfolios“The portfolio is a learning environment in which the learner constructs meaning. It assumes that meaning varies across individuals, over time, and with purpose. The portfolio presents process, a record of the processes associated with learning itself; a summation of individual portfolios would be too complex for normative description.” (p.36)

• Tension between two approaches“The two paradigms produce portfolio activities that are entirely different...“The positivist approach puts a premium on the selection of items that reflect outside standards and interests....“The constructivist approach puts a premium on the selection of items that reflect learning from the student’s perspective.” (p.36)


MirrorsMirrors


Iterative PDP / ePortfolio

Map

Mirror

Sonnet

– Where am I going and how will I get there?

– Reflecting … beyond surface learning … making meaning.

– Presenting my truth. Who I am and how I know.

Employer Learner Teacher HE ?

Mirror

Map

Sonnet

Stakeholder

Mode


MirrorsMirrors


Employer Learner Teacher

Mirror

Map

Example 1. Does planning and evidencing learning outcomes and improve learning in SSMs for 4th year medical students?

Sonnet


MirrorsMirrors


Cotterill SJ, Aiton J, Bradley PM, Hammond GR, McDonald AM, Struthers J, Whiten S. A flexible component-based ePortfolio: adapting and embedding in the curriculum. In: In Jafari A, Kaufman C, ed. Handbook of Research on ePortfolios. Pennsylvania: Idea Group Inc, 2006

Evaluating a ‘MAP’ type portfolioExample: ePortfolio for year 4 Medicine, Newcastle University

Mapping/PlanningRecording/EvidencingReflection


MirrorsMirrors


Q. Building the ePortfolio was a useful learning experience

0 10 20 30 40 50

Strongly Disagree

Weakly Disagree

Agree

% responses

80% thought it wasa useful learning experience

83% felt they had recorded good evidence

93% reflected on their learning after the [placement]

0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

Q. Having clearly defined intended learning outcomes influenced the way in which I approached the [placement]

72% felt that the LOsinfluenced their approach

n = 157Fixed-response questions (eg Likert scale)

n=157 (84% response rate)


MirrorsMirrors

Maps & Maps & SonnetsSonnets “It encouraged me to really give thought to what I

wanted to achieve during the [placement], which was especially useful as this was my first [placement]. As a result of the portfolio I think I got much more out of the [placement] than I would have otherwise.”

“It made me concentrate on creating aims at the start of the [placement] and allowed me to plan the [placement] with my supervisor in a defined way. Overall it made my learning for the [placement] more organised and focused.”

…but simple open-ended questions can be very illuminating:

“Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little value to try and invent a way in which they could.”

eg. what did you most like [dis]like about using the portfolio?


MirrorsMirrors


Measuring Development

ELLI• Achievement v

Personal development

• Characteristics of learning power

• Measurable change

ELLI and HE• 2006 – 2008• Situated research• Range of contexts

• Interested in participating?


MirrorsMirrors


Employer

Learner Teacher HE ?

Mirror

Map

Sonnet

Stakeholder

Mode


MirrorsMirrors


Group Discussion

Critique of the model• In terms of your own experience and use

of PDP/ePortfolio; what do you think of the MMS metaphor? Which other perspectives are important?

Developing the model• Identify a type of portfolio and a

perspective relevant to your practice or experience; share and discuss relevant research or evaluation questions and methodology


MirrorsMirrors


The problem of evaluation

• Evaluation of achievement not appropriate (positivist assumptions)

• Causative connections between PDP/ePortfolio difficult to establish

• Notion of situational evaluation


MirrorsMirrors


Thank you for your participation!

Contacts:[email protected]@newcastle.ac.uk

EPICS ePortfolios project& PDP Forum



Documents

Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Evaluating Portfolios: Mirrors Maps and Sonnets Simon Cotterill Sue Gill Jamie Thompson EPICS ePortfolios