Evaluating designs for learningNERA 2009

Associate Professor, PhD, Jeppe Bundsgaard (presenter) – jebu@dpu.dk School of Education, University of Aarhus

Director, PhD, Thomas Illum Hansen – thih@ucl.dkNational Knowledge Center for Designs for Learning

Aim

• To present a framework for evaluating designs for learning

• To provide a functional evaluation of the framework through an outline of an evaluation of the Practice Scaffolding Interactive Platform The Editorial Office.

What is a design for learning

• A constellation of artifacts arranged by someone with the intention to initiate and support someone's learning.

Artifacts

• Material tools and objects • Ideal phenomena• Primary artifacts

– a) physical objects, tools and technologies, – b) layout of the physical environment, – c) material aspect of texts (marks in media)

• Secondary artifacts– a) texts (conceptual aspect of text, when marks are read), genres– b) rules (laws of the state, moral laws, (computer) algorithms, etc.), – c) processes ((social) algorithms (what is to be done first, next, and then...)), – d) mental and social models (world view, personal relations, authority, ideology);

• Tertiary artifacts, which Wartofsky defines as imagined worlds,

Artifacts in situation

• Social phenomenon in a social context

• The status of artifacts are constantly negotiated in the social context.

• Power relations

• Core contradictions

The situation

Designs for learning are situated

• Always situated and defined by the context in which it is used.• Goal of an evaluation: to express the potentials of prototypical designs for

learning in prototypical situations.

• Potential learning potential– i.e. the affordances and challenges of the design for learning,

• Actualized learning potential– i.e. the incorporation and integration of the design for learning with a situation in

a given context, and • Actual learning

– i.e. the participants' development of competencies.

• Open and generic framework– Can be unfolded in many ways. Our model provides a number of examples

Model: Evaluation framework

Potential learning potential

Potential learning potential

• Explicit and implicit learning goals (competencies)• Explicit and implicit theory of teaching and learning• Explicit and implicit layout of the room and the roles of the

participants – An ordinary text book implicitly expects a teacher to teach from a

teachers desk, and students to sit in rows and listen and solve the assigned tasks.

– An educational computer game expects students to sit in front of their computers and play.

– A storyline framework expects teachers and students to collaborate in many places and through a variety of interactions, etcetera.

Analyzing potential learning potential

• (Textual) analysis of the artifacts• For example:

– Process analysis • (Hansen 2008)• Context; Description; Analysis; Intentionality, Functionality

– LORI (Learning Object Review Instrument)• (Leacock & Nesbit 2007)• Content quality; Learning goal alignment; Feedback and adaptation; Motivation; Presentation

design; Interaction quality; Accessibility; Reusability; Standards compliance– A Framework for Design and Evaluation of Internet-based Deistance Learning Courses

• (Baker 2003)• Bloom Criterion• Tyler Objectives• Tyler Experiences• Tyler Organization• Tyler Evaluations

Process analysis

• Context• Description• Analysis

– Content– Modality– Activity

• Intentionality• Functionality

Example: outline of an analysis ofThe Editorial Office

• Context– Thesis: Authentic activity promotes motivation and learning– Assumption: News paper production involves numerous

learning potentials – Problem: Project based learning in trouble (Barron et al.,

1998): • Problems with collaboration, • lack of knowledge of working procedures, • weak subject learning, • to many students lost.

• Suggestion for solution: Practice Scaffolding Interactive Platform (PracSIP)

Practice Scaffolding Interactive Platform

o Scaffolds by• Organizing collaboration• Structures the process• Provides relevant tools from the shared repertoire

o Makes the community of practice relevant for school by o Focusing on the aspects of the community of practice

which actualizes the relevant competencies• Supporting development of subject related competencies

o Reduces complexity in the practice(Bundsgaard in press)

Description: Students are supported structuring their work

Phases from profile over planning, research, photo, focus and writing to Layout and Deadline

Collaborationo Organization of the work

from beginning to the end

• Students set up articles …

• allocate tasks …• set deadlines …

o Students know what to do

o The teacher survey the situation

o The teacher can help the students in need of help.

Subject learning: Interactive assistants

o An interactive assistanto Takes a group of

students’ specific problem/task as starting point

o Leads the student through the process or realisation

o The computer structures – the student thinkso The computer does not have all the answers – they

are not ”multiple choice”.

Interactive assistants

o The computer asks thoroughly thought out questions, in order for the student to think

o The computer integrates the student’s response in the process

o Presents subject related concepts and methods in the process integrated with the questions

Summary of description ofThe Editorial Office

• Practice Scaffolding Interactive Platform

• Many different modalities (film, icon, writing, images, animation etc.) and very stylish

• Good integration between content and form

• Subject learning through participation in a simulated community of practice

• Very user friendly

Analysis

• Content

• Modality

• Activity

Analysis

• Connection between content and modality– Multimodality– Films explain community of practice– Flash explains tools– Writing and layout in interactive assistants explains subject

knowledge and methods– Structure of workflow mirrored in interface

• Relevant content– Organized in relation to typical tasks in a journalistic

community of practice– 42 interactive assistants

Activity

• Tools from the community of practice – mind map, layout tool, photo editor etc.

• Support – of collaboration, – mutual response, – and the process of research, writing and publication.

• Support roles of the community of practice– Journalists, photographers, graphical designers, editors– Teacher act as adviser, mentor, guide, authority etc.

Intentionality

• Goal: – To support students’ development of a number of

competencies unfolded in a journalistic practice

• Implicit: – Participation in a simulated practice with an

authentic outcome (a real news paper, relevant relations and work) motivates and leads to important learning

• Differentiated teaching and learning (inclusion)

Learning goals: Competencies

• Competencies of a community of practice– Mutuality of engagement (“the ability to engage with other

members and respond in kind to their actions, and thus the ability to establish relationships […]”)

– Accountability to the enterprise (“the ability to understand the enterprise of a community of practice deeply enough to take some responsibility for it […]”)

– Negotiability of the repertoire (“the ability to make use of the repertoire of the practice […]”)

(Wenger, 1998, p. 137)

Generalized competencies

• The internal structure of a demand-oriented competence(Rychen & Salganik 2003)

Competencies & The Editorial Office

• Competencies of the community of practice– Engagement

• Collaboration among journalists, photographers, editors, graphic designers.• Working to a deadline• Giving response

– Enterprise• Working as a watchdog, hear both sides of the story• Understand and enact the role of journalism in a democratic society• Knowing how to decide if a case is relevant to work with (or how to make it relevant)

– Repertoire• Researching• Prepare and organize the presentation of a story (make focus, express and word the

story in an understandable and well-organised way).• Take photos, layout the paper• Think like a journalist: Search for truth, trustworthiness, freedom of speech.

Generalized competencies

• Social competence– The ability to plan and enact collaboration– Establish rapport (with interview subjects etc.)– Value collaboration, experience the joy and importance of having an audience

• Communicative competence– Write to an audience– Organize content and present it– Value written communication as a means to meaningful participation

• Information competence– Getting information from different sources (human interaction, searching the Internet,

reading news papers etc.)– Critically evaluate information and bias

• Subject related competencies– Students develop competencies related to the specific content they are working with. For

example drug abuse, political decision processes etc.

Functionality

• Analyzers expectations on the basis of the analyses and interpretations

• Pedagogic functionality

• Relevancy of learning goals

• Critique of shortcomings (for example constrained or non-essential learning goals)

Functionality of The Editorial Office

• Teachers and students– need to accept that learning can take place in

complex collaboration settings – need to “play” the role play

• Would probably not work in a school used to highly structured individualized work and authoritarian relations.

• Highly relevant and transcending learning goals

Actualized learning potential

Actualized learning potential

• Some parts of the potential learning potential of the design for learning are actualized, co-determined by– Students

– Teachers

– Institutional culture

– Space/time

– The historical constellation of artifacts in the situation (including other designs for learning)

The actualized learning potential is manifested in a situation

• A situation is constituted by– Context– Artifacts– Participants– Time, course of events– Place, space

Teaching situation

Analyzing the situation

• A design for learning unfolds some potentials in the situation, and prevents others from unfolding

• Which potentials of the design for learning that are actualized are co-determined by the ‘Zone of Proximal Development’ of the participants (students and teachers) and the culture

• Thus: What is observed in a concrete situation is not the sole responsibility of the design for learning.

Methods of study

• Anthropological/ethnographic/ phenomenographic studies

• Quantitative and qualitative studies of the participants’ experience of satisfaction and their evaluation of success, motivation, learning.

Focus and goal

• With a focus on “what works” in this context, is this a success, did the students “learn something”, did the participants feel engagement and motivation for the work, etc.

• What are the challenges in this context of integrating this design for learning

• Goal: Generalization: How does this design for learning participate in a situation of this type.

The Editorial Office: Actualized learning potential

• Observed by the present authors and colleagues and master students in more than 10 classes.– Vide variety of contexts, cultures, teachers, students.

• Collaboration– Editorial offices => Smaller groups of students collaborate. Often further split down in groups of one or

two students.– General experience: A lot of engagement and activity. Students work at home, in recess, from their

sickbed, when out of town etc.• Room, place and space

– Students work in the computer lab, classroom, library, common areas, out of school territory etc. • Time

– Organized by the time planner and status tool. Students use these tools and seem to be helped by them. But some students do sit more or less inactive and wait.

• Artifacts– IT-platform regarded stylish. – The tools function as an integrated part of the students process.– In some classes the students use the interactive assistants, in some they don’t.

Cf. Henderson 2008

Participants’ experience and evaluation of the work

• 75% of the students (n=182) think that The Editorial Office is good or very good (8% find it bad or very bad).

• 98% of the teachers (n=97) consider The Editorial Office as good or very good.

• 57% of the students describe them selves as much more or little more active than in ‘normal teaching’. (Pedersen 2009).

Actual learning

Actual learning

• Learning: Change in capacity.– Learning is the processes (which are not maturation or

aging) from chaos/conflict/non-capability arises (or the intuitive understanding/capability matures) to the combination and change takes place so the subject is capable of something the subject couldn’t do before (in situations it couldn’t be done before).

• Capable of: Can do (bodily and mentally), can say, can understand, can combine, can perceive, can feel (emotion), wants to (attitude)

Actual learning – assessment

• Standardized tests– Criteria based, multiple choice, assessment of

practice

• Qualitative studies– Anthropological, ethnographic– Teachers estimation– Students’ self-report, teachers evaluation of

students’ self-report

The Editorial Office: Actual learning

• 95% of the teachers (n=96) evaluate their students’ academic development as satisfying or very satisfying (42%).

• 69% of teachers (n=96) says the academic level is appropriate. 25% find it hard (1% find it to hard).

Upcoming study

• Before and after standardized test of 4 classes working with the Editorial Office, and 4 working with related content (newspapers or the subject of the newspaper).

• Tests based on analysis of the competencies developed according to the potential learning potential of the two different designs for learning.

• Criteria based evaluation of double blinded test results done by two independent teachers.

Representing and choosing designs for learning

A good design for learning is a design for learning being used

• Teachers chose designs for learning based on– Compellingness– Accessibility of sharing and reuse

(Falconer et al. 2007)

Conclusions

Evaluating designs for learning Approaches

• Potential learning potential– What are the intentions and potentials of the design for

learning? What are the forecasted shortcomings?

• Actualized learning potential– What is the “effect”, what happens when the design for

learning is integrated with the situation?

• Actual learning– What is the outcome of the students and teachers work with

the design for learning in specific and generalized contexts?

References• Bang, J. C.; Døør, J.; Steffensen, S.V. & Nash, J. (2007). Language, Ecology and Society. London: Continuum.• Barron, B. J. S.; Schwartz, D. L.; Vye, N. J.; Moore, A.; Petrosino, A.; Zech, L.; Bransford, J. D. (1998). Doing with Understanding: Lessons from Research on Problem- and Project-Based.

The Journal of the Learning Sciences, 7(3/4), pp. 271-311.• Bundsgaard, J. (in press). PracSIP: Practice Scaffolding Interactive Platform. In: Proceedings from CSCL 09. Rhodos 2009.• Bundsgaard, J. (2006): Nøglekompetencer med bud til de humanistiske fagområder. In: Cursiv (1)1: 27-57. • Bundsgaard, J. (2005). Bidrag til danskfagets it-didaktik. Odense: Forlaget Ark.• Cole, Michael (1996). Cultural psychology: A once and future discipline. Cambridge: Harvard University Press. • Engeström, Y (1987). Learning by Expanding: An Activity - Theoretical Approach to Developmental Research. Retrieved February 12, 2009 from

http://communication.ucsd.edu/MCA/Paper/Engestrom/expanding/toc.htm.• Falconer, Isobel; Helen Beetham; Ron Oliver; Lori Lockyer and Allison Littlejohn (2007): Mod4L Final Report: Representing Learning Designs. Retrieved Marts 3, 2009 from

http://www.jisc.ac.uk/publications/publications/mod4lfinalreport.aspx. • Foucault, M. (1975). Surveiller et punir. Paris: Gallimard.• Hansen, T. I. (2008): Læremiddeldidaktik – hvad er det? Skitse til en almen læremiddeldidaktik. I: Tidsskrift for Læremiddeldidaktik 1(1), pp. 4-13. • Hansen, T. I. (2006). Poetik og lingvistik : om forholdet mellem kognitiv og fænomenologisk litteraturteori . PhD dissertation. SDU.• Hendersen, Lisa (2008): Praksisfællesskaber i undervisningen. Elevers deltagelsesformer i undervisning baseret på PracSIP’en: Redaktionen . Masters Thesis. Copenhagen.• Klafki, W. (1996). Neue Studien zur Bildungstheorie und Didaktik: Zeitgemässe Allgemeinbildung und kritisch-konstruktive Didaktik . Weinheim & Basel: Beltz.• Kress, G. R. (2003). Literacy in the new media age. London: Routledge.• Nesbit, J. C., Belfer, K., & Leacock T. L. (2004) LORI 1.5: Learning Object Review Instrument. Retrieved July 26, 2006, from http://www.elera.net.• Nesbit, J. C., Belfer, K., & Vargo, J. (2002). A convergent participation model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3), 105–120.• Marx, K. 1909. Capital. Vol. 1. London: William Glaisher. • Wegerif, R. and Mercer, N. (2000) Language for Thinking. In New Perspectives in Collaborative Learning, edited by Cowie, H., Aalsvoort, D. and Mercer, N. Oxford: Elsevier. • Novak, J. D. & Cañas, A. J. (2006): The Theory Underlying Concept Maps and How to Construct and Use Them. Technical Report IHMC CmapTools. Retrieved 11/10/08:

http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMaps.htm.• OECD (2004). Learning for Tomorrow's World: First Results for PISA 2003. Paris: OECD.• Pedersen, Signe (2009): Redaktionen 2008. En undersøgelse af lærere og elevers oplevelse af skoleprojektet. Copenhagen: Ekstra Bladet (Unpublished).• Rychen, D. S. & Salganik, L. H. (ed.) (2003): Key Competencies for a Successful Life and a Well-Functioning Society . Cambridge, MA: Hogrefe & Huber Publishers.• Selander, S. (2008). Designs for Learning – A Theoretical Perspective. In: Designs for learning 1(1), pp. 10-22.• Wegerif, Rupert (2004) The role of educational software as a support for teaching and learning conversations. Computers and Education, 43, (2), 179-191. • Østerud, Svein & Camilla Wiig (2000): ”Med kulturen i sentrum for IKT-forskningen: En studie i etableringen og gjennomføringen av prosjektet Elektronisk ransel i tre videregående skoler i

Narvik” I: Sten R. Ludvigsen og Svein Østerud: Ny teknologi – nye praksisformer. Teoretiske og empiriske analyser av IKT i bruk . Oslo: Unipub forlag. http://www.itu.no/Dokumenter/Rapporter/t1004526264_03/view (April 15, 2005).