Upload
others
View
11
Download
0
Embed Size (px)
Citation preview
An Introduction to Educational Design Research
SLO • Netherlands institute for curriculum developmentSLO
PO box 20417500 CA EnschedeThe Netherlands
T +31(0)53 484 08 40F +31(0)53 430 76 92E [email protected]
www.slo.nl
SLO is the Netherlands institute for curriculum development. We are bridging the contexts of policy, research, and practice. Our expertise focuses on the development of curricular goals and content for various educational levels, from national policy to classroom practices. We closely collaborate with many different stakeholders from policy circles, schools (boards, principals, teachers), research, civic organizations, and the society at large. This allows us to design and validate relevant curriculum frameworks, to elaborate exemplary materials and to evaluate these in school practices. Our products and services support both policy makers and schools and teachers in making substantive curricular decisions and in elaborating these into relevant, inspiring and effective education.
ISBN: 978 90 329 2329 7
An In
trod
uctIo
n to
Edu
cAtIon
Al dEsIg
n rEsEArch
Editors: tjeerd Plomp &
nienke n
ieveenEditors:Tjeerd Plomp & Nienke Nieveen
An Introduction to Educational Design Research
SLO •Netherlands institute for curriculum development
Tjeerd Plomp & Nienke Nieveen (editors)
Proceedings of the seminar conducted at the East China Normal University, Shanghai (PR China), November 23-26, 2007
an introduction to educational design research2
Authors: JanvandenAkker BrendaBannan AnthonyE.Kelly NienkeNieveen TjeerdPlompEditors: TjeerdPlomp NienkeNieveen
Design: AxisMedia-ontwerpers,Enschede
Production: Netzodruk,Enschede
AN: 7.5115.183
ISBN: 9789032923297
Orderaddress: SLO P.O.Box2041 7500Enschede theNetherlands
www.slo.nl/organisatie/international/publications
Enschede,3rdprintMarch2010
Colophon
an introduction to educational design research 3
Contents
Preface 5
1. EducationalDesignResearch:anIntroduction 9 Tjeerd Plomp
2. CurriculumDesignResearch 37 Jan van den Akker
3. TheIntegrativeLearningDesignFramework:AnIllustratedExamplefrom theDomainofInstructionalTechnology 53 Brenda Bannan
4 WhenisDesignResearchAppropriate? 73 Anthony E. Kelly
5 FormativeEvaluationinEducationalDesignResearch 89 Nienke Nieveen
6 ReferencesandSourcesonEducationalDesignResearch 103 Tjeerd Plomp and Nienke Nieveen
Authorbiographies 125
an introduction to educational design research 5
PrefaceThisbookistheresultofaseminaron‘educationaldesignresearch’organizedfromNovem-ber23-26,2007,byProfZhuZhiting(DepartmentofEducationalTechnology)oftheCollegeofEducationalSciencesattheEastChinaNormalUniversityinShanghai(PRChina).Theprimarygoaloftheseminarwasto introduce a group postgraduate students and lectu-ring staff in China to educational design research as a research approach.Thesecondgoaloftheseminarwastoprepare,basedonthecontributionsofanumberinternationalexperts,proceedings of the seminarwritteninsuchawaythattheycanbeusedinpostgraduateseminarsoneducationaldesignresearchacrossChina.
About75peoplewithbackgroundsmainlyininstructionaltechnology,curriculumandinstructionaldesignparticipatedintheseminar.Mostofthemwereworkinginteachereducation,inschoolsasinstructionaltechnologistand/orindistanceeducation.Althoughparticipantshad(throughtheirstudies)alreadyknowledgeandsomeexperienceinin-structionalorcoursedesignandinresearchmethods,theywereeagertobeintroducedtodesignresearchasarelativelynewresearchapproachforaddressingcomplexproblemsineducationalpractice.
TheseminarstaffconsistedofProfsBrenda BannanandEamonn Kelly(bothGeorgeMasonUniversity,Fairfax,VA,USA)andProfJan van den Akker(UniversityofTwenteandNationalInstituteforCurriculumDevelopment[SLO],Enschede,TheNetherlands),andthetwoedi-torsofthisbookDrNienke Nieveen(NationalInstituteforCurriculumDevelopment[SLO],Enschede)andProfTjeerd Plomp(UniversityofTwente,Enschede,TheNetherlands).Ascanbeseenfromthetableofcontentofthisbook,theyarereflectingthebackgroundofthepar-ticipants,astheyrepresentedexperienceinconductingdesignresearchinthedomainsofcurriculumdevelopment,instructionaltechnologyandmathematicsandscienceeducation.ExpertswereconsciouslyinvitedfrombothEurope(TheNetherlands)aswellastheUSA,soastoascertainthatvariationinbackgroundandperspectiveondesignresearchwasrepre-sentedinconductingtheseminar.
Thechaptersinthisbookarebasedonthepresentationsandthesmallgroupdiscussionsduringthisseminar.Althoughthebookdoesnotprovidea‘howtodoguide’fordesigningandconductingdesignresearch,thechaptershavebeenwritteninsuchawaythattheyreflectboththeconceptualunderpinningandpracticalaspectsofthe‘what’and‘how’ofdoingdesignresearch(chaptersbyPlomp,KellyandNieveen),aswellasprovidethereaderaninsightinthespecificsofdoingdesignresearchinthedomainofcurriculum(chapterbyVandenAkker)andinstructionaltechnology(chapterbyBannan).Toassistthereadersinfindingtheirwayintheabundanceofliteratureondesignresearch,wehaveaddedachapterwithreferencesandsourcesoneducationaldesignresearch.This
an introduction to educational design research6
bibliographyisfarfromcompleteandreflectsverymuchthebackgroundandthebiasesoftheeditorsofthisbook.Yetwetrustthatthischapterwillassisttheinterestedreaderingettingintroducedtothisexcitingandpromisingresearchapproach.
WewanttothankProfZhuZhitingfromtheEastChinaNormalUniversityfortakingtheinitiativeforthisseminar.Similarlywewanttothankourcolleaguesforcontributingtothisbook.
Butaboveall,weliketoexpressourhopethatthisbookwillstimulateandsupportmany(future)researcherstoengagethemselvesineducationaldesignresearch.
JanvandenAkkerDirector General SLO
TjeerdPlompandNienkeNieveenEditors
an introduction to educational design research 7
an introduction to educational design research 9
1. EducationalDesignResearch: anIntroduction Tjeerd Plomp
Introduction
Thepurposeofthischapteristoprovideanintroductiontoeducationaldesignresearchasaresearchapproachsuitabletoaddresscomplexproblemsineducationalpracticeforwhichnoclearguidelinesforsolutionsareavailable.Educationaldesignresearchisperceivedasthesystematicstudyofdesigning,developingandevaluatingeducationalinterventions,-suchasprograms,teaching-learningstrategiesandmaterials,productsandsystems-assolutionstosuchproblems,whichalsoaimsatadvancingourknowledgeaboutthecharacteristicsoftheseinterventionsandtheprocessestodesignanddevelopthem.
Theneedforaresearchapproachthataddressescomplexproblemsineducationalpracticehasbeenarguedbyresearchersinvarious‘corners’ofthedomainofeducationfromthelackofrelevanceofmucheducationalresearchforeducationalpractice.Forexample,theDesign-BasedResearchCollective(2003:5)arguesthateducationalresearchisoftendivorcedfromtheproblemsandissuesofeverydaypractice–asplitthatresultedinacredibilitygapandcreatesaneedfornewresearchapproachesthatspeakdirectlytoproblemsofpracticeandthatleadtothedevelopmentof‘usableknowledge’.Fromhisbackgroundinresearchinthedomainofcurriculum development and implementation,VandenAkker(1999:2)arguesthatmany‘traditional’researchapproachessuchasexperiments,surveys,correlationalanalyses,withtheiremphasisondescriptionhardlyprovideprescriptionsthatareusefulfordesignanddevelopmentproblemsineducation.Heclaimsthatanimportantreasonfordesignresearch1stemsfromthecomplexnatureoftheeducationalreformsworldwide.Ambitiousreformscannotbedevelopedatthedrawingtablesingovernmentoffices,butcallforsystematicresearchsupportingthedevelopmentandimplementationprocessesinavarietyofcontexts.Inhisreviewofthestateofeducationalresearchandmorespecificallyeducationaltechnologyresearch,Reeves(2006:57)concludesthatthereis“alegacyofill-conceivedandpoorlyconductedresearchthatresultsinnosignificantdifferencesor,atbest,inmodesteffectsizes”.Healsoarguesforthedomainofeducational technologythateducationaltechnologists,insteadofdoingmore(media)comparisonstudies,shouldundertaketypesofdesignresearch.Inotherwords,ReevesarguesthatinsteadofdoingmorestudiescomparingwhetherinacertaincontextmethodAisbetterthanmethodB,itisbetterto
1) whichhecalls‘developmentresearch’inhis1999publication
an introduction to educational design research10
undertakedesignresearchaimedatdevelopinganoptimalsolutionforaproblemincontext.Inthefieldoflearning sciences,thebeliefthatcontextmattersleadtotheconclusionthatresearchparadigmsthatsimplyexamineslearningprocessesasisolatedvariableswithinlaboratorysettingswillnecessarilyleadtoanincompleteunderstandingoftheirrelevanceinmorenaturalisticsettings(Barab&Squire,2004;withreferencetoBrown,1992).Inthisfield,design-basedresearchwasintroducedwiththeexpectationthatresearcherswouldsystematicallyadjustvariousaspectsofthedesignedcontextsothateachadjustmentservedasatypeofexperimentationthatallowedtheresearcherstotestandgeneratetheoryinnaturalisticcontexts(Barab&Squire,2004:3).
Thesesourcesillustratetheneedfordesignresearchasanalternativeresearchapproach.Beforeelaboratingondesignresearchthispaperwillfirstdiscussmoregenerallypossiblefunctionsofresearchandhowresearchfunctionsarerelatedtoresearchapproaches.Thendesignresearchwillbedefinedandcharacterizedfromvariousperspectives,suchasthetypeofknowledgethedesignresearchersaimfor,thetypeofresearchquestionsthatcanbeaddressed,andtheoutputsofdesignresearch.Thiswillbefollowedbyasectioninwhichdifferentapproachestodesignresearchareintroducedandsectionsdiscussinghowdesignresearchcanorshouldbeconducted,withamorein-depthdiscussionofformativeevaluationasthemostprominentresearchactivityindesignresearch.Conductingdesignresearchputsresearchersinasituationinwhichtheyhavetofaceanumberofdilemmas.Thesewillbediscussedbeforeendingthechapterwithafewconcludingremarks.
Afinalnoteonterminology,followingVandenAkkeretal.(2006:4)weusedesign researchasacommonlabelfora‘family’ofrelatedresearchapproacheswhomayvarysomewhatingoalsandcharacteristics–examplesaredesignstudies,designexperiments,design-basedresearch,developmentalresearch,formativeresearch,engineeringresearch.
Research functions – research approaches
Beforeelaboratingonthemeaningofdesignresearch,itisimportanttopositiondesignresearchasaresearchapproachnexttootherresearchapproaches,whichisthepurposeofthissection.
Thekeyfocusinallscientificresearchisthesearchfor‘understanding’orfor‘knowing’withtheaimofcontributingtothebodyofknowledgeoratheoryinthedomainofresearch.Otherbroad aimsofdoingeducationalresearcharetoprovideinsightsandcontributionsforimprovingpractice,andtoinformdecisionmakingandpolicydevelopmentinthedomainofeducation.
an introduction to educational design research 11
ResearchfunctionsIngeneral,wecandistinguishvariousresearch functions,eachreflectingcertaintypesofresearchquestions.Examplesofresearchfunctions(withexemplaryresearchquestionsfittingthefunction)are:1. to describe: e.g.whatistheachievementofChinesegrade8pupilsinmathematics;
whatbarriersdostudentsexperienceinthelearningofmathematicalmodelling2. to compare:e.g.whatarethedifferencesandsimilaritiesbetweentheChineseandthe
Netherlandscurriculumforprimaryeducation;whatistheachievementinmathematicsofChinesegrade8pupilsascomparedtothatincertainothercountries
3. to evaluate:e.g.howwelldoesaprogramfunctionintermsofcompetencesofgraduates;whatarethestrengthsandweaknessesofacertainapproach;etc
4. to explain or to predict:e.g.whatarethecausesofpoorperformanceinmathematics(i.e.insearchofa‘theory’predictingaphenomenonwhencertainconditionsorcharacteristicsaremet)
5. to design and develop:e.g.whatarethecharacteristicsofaneffectiveteachingandlearningstrategyaimedatacquiringcertainlearningoutcomes;howcanweimprovethemotivationoflearners.
Inmanyresearchprojectstheresearchquestionsaresuchthatinfactvariousresearchfunctionsdoapply.Forexample,iftheresearchquestionpertainstocomparingthemathematicsachievementofChinesegrade8pupilsascomparedtothatincertainothercountries,thenaspartofcomparingtheresearcherswillevaluatetheachievementofgrade8pupilsineachofthecountriesinvolved.Or,asanotherexample,ifonewantstodesign and developateaching-learningstrategyforacquiringthecompetencyofmathematicalmodelling(ingrade11&12),thenresearchersmayfirstwanttounderstandandcarefullydescribewhatbarriersstudentsexperiencewithmathematicalmodelling,whilstalsotheevaluationfunctionisimportantindeterminingwhethertheteaching-learningstrategythathasbeendevelopediseffective.Bothexamplesillustratethatusuallyaresearchprojecthasaprimaryresearchfunction,butthatotherresearchfunctionsarebeingappliedto‘serve’theprimaryresearchfunction.
Atthelevelofaresearch project,startingfromaresearchproblemorquestion,wearesupposedtohavethefollowingsequence:Research question => (primary) research function =>choice of research approach.Inthischapterwefocusonresearchwhichhasdesign and developastheprimaryresearchfunction.
ResearchapproachesMosttextbooksonresearchmethodologypresentanddiscussanumberofresearch approachesorstrategies(seee.g.Denscombe,2007).Usuallyeachresearchapproachcanbe
an introduction to educational design research12
usedforrealizingmorethanoneresearchfunction.Withoutgoingintodetailhere,examplesofresearchapproachesandtheirpossibleresearchfunctionsare:• survey: todescribe,tocompare,toevaluate• case studies:todescribe,tocompare,toexplain• experiments:toexplain,tocompare• action research:todesign/developasolutiontoapracticalproblem• ethnography:todescribe,toexplain• correlational research:todescribe,tocompare• evaluation research:todeterminetheeffectivenessofaprogramTextbooksonresearchmethodologyusuallydonotpresentanddiscussdesignresearch:• design research: todesign/developanintervention(suchasprograms,teaching-learning
strategiesandmaterials,productsandsystems)withtheaimtosolveacomplexeducationalproblemandtoadvanceourknowledgeaboutthecharacteristicsoftheseinterventionsandtheprocessestodesignanddevelopthem.
Inlinewiththeremarkthatmorethanoneresearchfunctionmayhavetobeappliedtoaddressaresearchquestion,itshouldbenoticedthatinaresearchprojectmorethanoneresearchapproachmayhavetobeapplied.Forexample,ifthereisaneedtocomparehowwellChinesegrade8pupilsperforminmathematicsascomparedtoanumberofothercountries,theprimaryresearchfunctionistocompare,leadinginthiscasetoasurveyasthebestresearchapproach.However,aspartofthedevelopmentofavalidandreliablemathematicstest,theresearchersmaydocorrelationalresearchtodeterminewhetherthetestbeingdevelopedisvalid,i.e.correlateswithothermeasuresofmathematicsachievement.
Asafinalremark,itisimportantthatdesignresearchers,likeallresearchers,keepinmindthatalsofortheirresearchtheguidingprinciplesforscientificresearch(Shavelson&Towne,2002)apply,viz:• Posesignificantquestionsthatcanbeinvestigated• Linkresearchtorelevanttheory• Usemethodsthatpermitdirectinvestigationofthequestion• Provideacoherentandexplicitchainofreasoning• Replicateandgeneralizeacrossstudies• Discloseresearchtoencourageprofessionalscrutinyandcritique
an introduction to educational design research 13
What is design research?
Asstatededucationaldesignresearchisthe systematic study of designing, developing and evaluating educational interventions (such as programs, teaching-learning strategies and materials, products and systems) as solutions for complex problems in educational practice, which also aims at advancing our knowledge about the characteristics of these interventions and the processes of designing and developing them. Thetwofoldyieldofdesignresearch,viz.researchbasedinterventionsaswellasknowledgeaboutthem,canalsofoundindefinitionsofdesignresearchbyotherauthors.Forexample,thebroaddefinitionofBarabandSquire(2004)alsoencompassesmostvariationsofeducationaldesignresearch:“aseriesofapproaches,withtheintentofproducingnewtheories,artefacts,andpracticesthataccountforandpotentiallyimpactlearningandteachinginnaturalisticsetting.
Byitsnature,designresearchisrelevantforeducationalpractice(andthereforealsoforeducationalpolicy)asitaimstodevelopresearch-basedsolutionsforcomplexproblemsineducationalpractice.Startingpointfordesignresearchareeducationalproblemsforwhichnooronlyafewvalidatedprinciples(‘howtodo’guidelinesorheuristics)areavailabletostructureandsupportthedesignanddevelopmentactivities2.Informedbypriorresearchandreviewofrelevantliterature,researchersincollaborationwithpractitionersdesignanddevelopworkableandeffectiveinterventionsbycarefullystudyingsuccessiveversions(orprototypes)ofinterventionsintheirtargetcontexts,andindoingsotheyreflectontheirresearchprocesswiththepurposetoproducedesignprinciples.Manyexamplesoftheneedforinnovativeinterventionscanbegivenatsystemlevelandinstitutionallevel.Atsystemlevel,forexample,onemaywanttodevelopasystemfore-learningtoserveaspecifictargetgroupofstudentsinhighereducation,andatthelevelofschoolorclassroomonemaywant,forexample,toaddressthequestionofwhatareeffectivemethodsforcollaborativelearning.SeealsoGustafson&Branch(2002)whodevelopedataxonomyofinstructionaldevelopmentmodelsbasedonaselectedcharacteristics;theydistinguishbetweenmodelswithaclassroomorientation,productorientationandsystemorientation.
Theresearchprocessindesignresearchencompasseseducationaldesignprocesses.Itis–likeallsystematiceducationalandinstructionaldesignprocesses-thereforecyclicalincharacter:analysis,design,evaluationandrevisionactivitiesareiterateduntilasatisfyingbalancebetweenideals(‘theintended’)andrealizationhasbeenachieved.
2) seealsothechapterofKellyinthisbookwherehediscusseswhendesignresearchisappropriate.
an introduction to educational design research14
Thisprocesscanbeillustratedinvariousways.Justafewexamplesarepresentedheretoshowhowdifferentauthorshavevisualizedtheresearchprocess.
Reeves(2006)depictsthedesignresearchapproachasfollows:
Figure 1: Refinement of Problems, Solutions, Methods, and Design Principles (Reeves, 2000, 2006)
McKenney(2001)illustratesinherstudythiscyclicalprocessasfollows:
Figure 2: Display of the CASCADE-SEA study (McKenney, 2001)
Identify and analyse problems by
researchers & practitioners in
collaboration
Development of prototype solutions: informed by state-
of-art theory, existing design principals & technology innova-
tions
Iterative cycles of testing & refinement
of solutions in practice
Reflection to produce ‘design principles’ &
enhance solution implementation in
practice
160
140
120
100
80
60
40
20
0
num
ber o
f par
ticip
ants
needs & contextanalysis
design, development & formative evaluation semi-summativeevaluation
literaturereview &concept
validation
sitevisits
prototype1
prototype2
cycle width is proportional to timeschale: = circa 6 months
prototype3
prototype4
finalevaluation
query
an introduction to educational design research 15
The‘query’asthelastphaseinMcKenney’sdisplaycanbeinterpretedasthereflectionboxinthemodelofReeves(Figure1).AnotherexampleistheIntegrativeLearningDesignFrameworkthatBannan-Ritlandpresentsinchapter5ofthisbook(seealsoBannan-Ritland,2003).
Authorsmayvaryinthedetailsofhowtheypicturedesignresearch,buttheyallagreethatdesignresearchcomprisesofanumberofstagesorphases:• preliminary research:needsandcontextanalysis,reviewofliterature,developmentofa
conceptualortheoreticalframeworkforthestudy• prototyping phase:iterativedesignphase3consistingofiterations,eachbeingamicro-
cycleofresearch4withformativeevaluationasthemostimportantresearchactivityaimedatimprovingandrefiningtheintervention
• assessment phase:(semi-)summativeevaluationtoconcludewhetherthesolutionorinterventionmeetsthepre-determinedspecifications.Asalsothisphaseoftenresultsinrecommendationsforimprovementoftheintervention,wecallthisphasesemi-summative.
Throughoutalltheseactivitiestheresearcherorresearchgroupwilldosystematic reflection and documentationtoproducethetheoriesordesignprinciples(aconcepttakenfromVandenAkker,1999–seealsochapter2)asthescientificyieldfromtheresearch.Onemaystatethatthissystematicreflectionanddocumentationmakesthatsystematicdesignanddevelopmentofaninterventionbecomesdesignresearch.
Authorsaboutdesignresearchalsoagreeanumberofcharacteristicsofthistypeofresearch.ThesearesummarizedbyVandenAkkeretal.(2006:5):• Interventionist:theresearchaimsatdesigninganinterventioninarealworldsetting;• Iterative:theresearchincorporatescyclesofanalysis,designanddevelopment,
evaluation,andrevision;• Involvement of practitioners:activeparticipationofpractitionersinthevariousstages
andactivitiesoftheresearch• Process oriented:thefocusisonunderstandingandimprovinginterventions(ablack
boxmodelofinput–outputmeasurementisavoided);• Utility oriented:themeritofadesignismeasured,inpartbyitspracticalityforusersin
realcontexts;and• Theory oriented:thedesignis(atleastpartly)basedonaconceptualframeworkand
upontheoreticalpropositions,whilstthesystematicevaluationofconsecutiveprototypesoftheinterventioncontributestotheorybuilding.
3) itispossiblethatthedesign/developmentcomponentinasucharesearchprojectwillnotbeginfromscratchbutwiththeevaluationofanexistinginterventionwiththeaimofidentifyingtheneedforimprovement,whichthenisfollowedbyre-designandanumberofdesigncycles.
4) termtakenfromBannan-Ritland,chapter5
an introduction to educational design research16
ThefeaturesandcharacteristicsofdesignresearcharenicelycapturedbyWademan(2005)inwhathecallstheGenericDesignResearchModel(Figure3).Hismodelclearlyillustratesthatthe‘successiveapproximationofpracticalproducts’(whatwecall‘interventions’)isgoinghandinhandwiththe‘successiveapproximationoftheory’(whichhealsocalls‘designprinciples’).
Figure 3: Generic Design Research Model (Wademan, 2005)
Itisimportanttonotethatdesignresearchfollowsaholisticapproach,anddoesnotemphasizeisolatedvariables.VandenAkkeretal.(2006:5)pointtoitthatyetdesignresearchersdofocusonspecificobjectsandprocesses(interventions)inspecificcontexts,buttheytrytostudythoseasintegralandmeaningfulphenomena.Thiscontextboundnatureofmuchdesignresearchalsoexplainswhyitusuallydoesnotstrivetowardscontext-freegeneralizations.Ifanefforttogeneralizingismade,thenitisananalyticalgeneralization(incontrasttostatisticalgeneralizationwheretheresearchermaygeneralizefromsampletopopulation).(Seealsothesection‘outputsofdesignresearch’)
Practitioner and User Participation
Researchers
Other Sources
Collaboratives
Practitioners
ConsultExperts &
Practitioners
FocusedLiterature
Review
AnalyzePromisingExamples
AnalyzePracticalContext
Tentative Product
Approaches
Tentative Design
Principles
Reflection
FormativeEvaluation
PracticalProduct/Results
Contributionto
Theory
Refinement of Problem, Solution and Method
Refinement of Design Theory
Preliminary investigation of Problem, Context,
& Approaches
Problem in Context
Phases
Prototyping & Assessment of Preliminary Products & Theories
Identification of Tentative Products & Design Principes
Problem Identification Tentative Products& Theories
Problem Resolution & Advancing Theory
Successive Approximation of Theory
Successive Approximation of Product
Redesign & Refinementof Products & Theories
an introduction to educational design research 17
A closer look at design research
Asstatedkeycharacteristicsofdesignresearcharethatitisresearchfocusedondesigninginterventionsintherealcontextofeducationortraining(interventionistcharacteristic)combinedwitheffortstounderstandandimproveinterventions(process orientation),utilizingstateofthearttheorieswhilstthefieldtestingandtheevaluationoftheconsecutiveprototypesshouldcontributetotheorybuilding(theory orientation).Inthissectionwewillhavealookatwhatitmeansthatresearchsupportseducationaldesignprocesses,andreverselythateducationaldesignprocessessupportresearch.Thisisfollowedbyabriefdiscussionofthetypeofresearchquestionindesignresearch.Possibleoutputsofdesignresearchwillbediscussedinthenextsection.
Aswealreadystated,oneoftheaimsofdesignresearchisdesigninganddevelopinganinterventionasan(innovative)solutiontoacomplexproblem,andthereforethestartingpointfordesignresearchareeducationalproblemsforwhichnooronlyafewvalidatedprinciples(‘howtodo’guidelines)areavailabletostructureandsupportthedesignanddevelopmentactivities.Ontheotherhand,designresearchisresearchandthereforetheappropriateyieldfordesignresearch(apartfromausableandeffectiveintervention)isempiricallyfoundedtheory,i.e.thechallengefordesignresearchistocaptureandmakeexplicittheimplicitdecisionsassociatedwithadesignprocess,andtotransformthemintoguidelinesforaddressingeducationalproblems(seeEdelson,2006;101;alsoBarab&Squire(2003),andmanyotherauthors).Thisaspectreferstothetheory orientation,mentionedaboveasoneofthecharacteristicsofdesignresearch.VandenAkker(1999,2006,alsochapter2),Reeves(2006;seefigure1)andWademan(2005;seefigure3)usetheconceptof‘designprinciples’whentheyrefertothetheoreticalyieldsofdesignresearch,whereothersspeakofnewtheories(e.g.Barab&Squire,2003;Edelson,2006).
However,itisnotself-evidenthowthedesignofinterventionsmaycontributetotheorybuilding.WithreferencetothegenericmodelofWademen(Figure3)andtheexemplaryschemesofReeves(2006)inFigure1andMcKenney(2001)inFigure2,onemaystatethattheresearcher(orbetter:thecollectiveofresearchersandpractitioners)-basedonanalysisoftheproblemincontext,andutilizingrelevant,state-of-the-arttheories–designsanddevelops(inaniterativeway)theinterventionwiththeaimthatafteranumberofcyclestheintendedoutcomesarerealized,i.e.asatisfyingsolutiontotheproblemidentified.Eachiterationorcycleisamicro-cycleofresearch,i.e.astepintheprocessofdoingresearchandwillincludesystematic reflectiononthetheoreticalaspectsordesignprinciplesinrelationshiptothestatusoftheintervention,resultingintheendindesignprinciplesortheoreticalstatements.
an introduction to educational design research18
Inotherwords,intheendtheresearcher(orresearchgroup)willconcludeabouthisintervention:
Given my context, if I do <intervention (theory based) > then I expect <intended outcomes>.
Thiscanbedisplayedschematicallyas:
Twopointsareimportantinthisscheme:• theoutcomesoftheinterventionareindicatedasY1,Y2,…,Yn,becauseoftenan
interventionisdesignedtorealizemultipleoutcomes(e.g.betterachievement,improvedstudentattitude,increasedteachersatisfaction,etc).
• theinterventionispresentedas‘inputprocess’,becausedesigningaprocess(e.g.learningenvironment)hastotakeintoaccountalsotheinputsnecessarytomaketheprocessfunction(e.g.certaininstructionallearningmaterials,teacherdevelopment).
Sointheend,theresearchgrouphasnotonlyatitsdisposaltheinterventionresultinginthedesiredoutcomes,butalsobasedonasystematicreflectionandanalysisofthedatacollectedduringthiscyclicalprocessanunderstandingofthe‘howandwhy’ofthefunctioningoftheinterventionintheparticularcontextwithinitwasdeveloped.Thedesignresearcherwillsummarizethisunderstandingofthe‘howandwhy’oftheinterventioninoneormore‘designprinciples’ifwewouldusetheterminologyofVandenAkker(1999,2006)andReeves(2000,2006).Asotherauthors,e.g.Barak&Squire(2004)andEdelson(2006),useof‘theory’astheyieldofdesignresearch,onemayalsospeakof‘interventiontheory’or‘designtheory’(Wademan,2005;Figure3)asasecondgenerictermtorefertotheknowledgegeneratedfromthisresearchendeavour(seebelowforspecificexamples).
Indesignresearch,interventionsaredevelopedinacyclical processofsuccessiveprototypes:
Intervention XInput Process
OutcomesY1, Y2, ...., Yn
Intervention XInput Process
OutcomesY1, Y2, ...., Yn
design principles or intervention theory
Akeyideaisthatwheninacertaincycletheprototypeoftheinterventiondoesnotresultinthedesiredoutcomes,onemayconcludethatthedesignprinciples(orinterventiontheory)appliedarenot(yet)effective(or,inotherwords,thattheinterventiontheory‘fails’).Thishastoresultinare-designorrefinementoftheintervention,whichgoeshand-in-handwiththerefinementoftheinterventiontheoryordesigntheory.Whenafteranumberofiterationstheresearcher(orresearchgroup)concludesthatbasedontheanalysisoftheevaluationdatathe‘realizedoutcomes’arecloseenoughtothe‘intendedoutcomes’thenhecanbesatisfied:thedesignprinciplesappeartobeeffective.Or,inotherwords,theresearcher(orresearchgroup)hasdevelopeda‘local’(intervention)theory(i.e.forthecontextinwhichhe/sheworks):in context Z the intervention X (with certain characteristics) leads to outcomes Y1, Y2, …, Yn.
Twoexamplesaregiventoillustratethis–ratherabstract–phrasingoftheyieldofdesignresearch.TheDesign-BasedLearningResearchCollective(2003:5)statethat“thedesignofinnovationsenablesustocreateleaningconditionsthatlearningtheorysuggestsareproductive,butthatarenotcommonlypracticedorarenotwellunderstood”–inotherwordsincludedintheinnovationsisknowledgeabouthowtocreateconditionsforlearning.Thesecondexampleistakenfromscienceeducation.Lijnse(1995:192)arguesthatdesignresearch(hecallsitdevelopmentalresearch)is“acyclicprocessoftheoreticalreflections,conceptualanalysis,small-scalecurriculumdevelopment,andclassroomresearchoftheinteractionofteaching-learningprocesses.Thefinal,empiricallybaseddescriptionandjustificationoftheseinterrelatedprocessesandactivitiesconstituteswhatwecallapossible“didacticalstructure”forthetopicunderconsideration.”Inotherwords,thelocaltheoryconsistsofadidacticalstructureforteaching-learningprocessesforacertaintopic.
Theresearchquestionindesign/developmentresearchBynowitisclearthatdesigninganddevelopinganinterventionisinitselfnotyetdesignresearch.Butonemayconductadesign/developmentprojectasaresearchprojectbyemployingrigorouslysocialscienceresearchmethodology.Astheresearcherisstrivingtofinddesignprinciples(oranintervention theory)thatarevalidinacertaincontext,theresearchquestioncanbephrasedas:
what are the characteristics of an <intervention X> for the purpose/outcome Y (Y1, Y2, …, Yn) in context Z
Design/develop Implement/try-0ut
Evaluate (formative)
an introduction to educational design research 19
an introduction to educational design research20
Examplesofresearchquestionsare:(i) whatarethecharacteristicsofaneffectivein-serviceprogrammeformathematics
teachersthroughwhichtheydeveloptheabilitytoapplystudent-centredpedagogicalmethods,and
(ii) whatarethecharacteristicsofanin-servicearrangementthatfacilitatestheimplementationofMBL5-supportedlessonactivitiesinphysicseducation(Tecle,2003)?
Obviously,notallresearchersareusingthistypeofphrasing,butthewordingofthemainresearchquestionindesignresearchalwaysimpliesasearchforcharacteristics.Anexampleis:Whatisanadequatelearningandteachingstrategyforgeneticsinuppersecondarybiologyeducationinordertocopewiththemaindifficultiesinlearningandteachinggenetics,andtopromotetheacquisitionofameaningfulandcoherentunderstandingofhereditaryphenomena?(Knippels,2002)
The outputs of design research
Wealreadyconcludedthatdesignresearchresultsininterventions(programs,products,processes)andindesignprinciplesorinterventiontheory.Athirdoutputofdesignresearchisprofessionaldevelopmentoftheparticipantsinvolvedintheresearch.Eachoftheseoutputsisbrieflydiscussed.
OndesignprinciplesorinterventiontheoryDesignresearchaimsatproducingknowledgeaboutwhetherandwhyaninterventionworksinacertaincontext.Intheprevioussectionthistypeofoutputhasbeencalleddesignprinciplesorinterventiontheory.Otherauthorsusetermslikedomainspecifictheories(Gravemeijer&Cobb,2006),designtheory(Wademan,2005;Figure3),heuristicsorjustlessonslearned(seeVandenAkkeretal.2006).Wewillusethetermdesign principlesintheremainingofthispaper.
Designprinciplesareheuristic statementsforwhichVandenAkker(1999)developedthefollowingformat:
“If you want to design intervention X for the purpose/function Y in context Z, then you are best advised to give that intervention the characteristics A, B, and C [substantive emphasis], and to do that via procedures K, L, and M [procedural
emphasis], because of arguments P, Q, and R.” (Van den Akker, 1999)
5) MBL=MicrocomputerBasedLaboratory.
Theheuristicprinciplesaremeanttosupportdesignersintheirtasks,butcannotguaranteesuccess-theyareintendedtoassist(inotherprojects)inselectingandapplyingthemostappropriate(substantiveandprocedural)knowledgeforspecificdesignanddevelopmenttasks.Substantiveknowledgeisknowledgeaboutessentialcharacteristicsofaninterventionandcanbeextracted(partly)fromaresultinginterventionitself.Proceduralknowledgereferstothesetofdesignactivitiesthatareconsideredmostpromisingindevelopinganeffectiveandworkableintervention.Asknowledgeisincorporatedininterventions,itisprofitablefordesignresearchersintheearlystageoftheirresearchtosearchforalreadyavailableinterventionsthatcanbeconsideredusefulexamplesorsourcesofinspirationfortheproblematstake.Carefulanalysisofsuchexamplesincombinationwithreviewingrelevantliterature)willgenerateideasforthenewdesigntask.Thevalueofknowledgeresultingfromadesignresearchprojectwillstronglyincreasewhenitisjustifiedbytheoreticalarguments,well-articulatedinprovidingdirections,andconvincinglybacked-upwithempiricalevidenceabouttheimpactofthoseprinciples.Itisforthisreasonthatauthors(e.g.VandenAkker1999,2006;Reeves,2000,2006)statethatthefinalstageofeachdesignresearchprojectshouldconsistofsystematicreflectionanddocumentationtoproducedesignprinciples.
GeneralizabilityindesignresearchHeuristicdesignprincipleswillbeadditionallypowerfuliftheyhavebeenvalidatedinthesuccessfuldesignofmoresimilarinterventionsinvariouscontexts.Chancesforsuchknowledgegrowthwillincreasewhendesignresearchisconductedintheframeworkofresearchprograms,becausethenprojectscanbuildupononeanother.Herewetouchonthequestiontowhatextentdesignprinciplescanbegeneralizedfromonecontexttoothers.ItisinthiscontextthatEdelson(2006)statesthatdesignresearchshouldresultingeneralizabletheory.Indesignresearch,likeincasestudiesandexperimentalstudies,thefindingscannotbegeneralizedtoalargeruniverse–thereisnostatisticalgeneralizationfromsampletopopulation,likecanbethecaseinsurveyresearch.Yin(2003)pointstoitthatincasestudiesandexperimentalstudies,theinvestigatorisstrivingtogeneralizeaparticularsetofresultstoabroadertheory.Thisisalsothecaseindesignresearch,theresearchershouldstrivetogeneralize‘designprinciples’tosomebroadertheory.Yin(2003:37)pointstoitthatgeneralizationisnotautomatic.Designprinciplesmustbetestedthroughreplicationsofthefindingsinasecond,thirdormorecasesinvariouscontextswiththepurposethatthesameresultsshouldoccur.Oncesuchreplicationshavebeenmade,theresultsmightbeacceptedforamuchlargernumberofsimilarcontexts,eventhoughfurtherreplicationshavenotbeenperformed.Thisreplication logicisthe
an introduction to educational design research 21
an introduction to educational design research22
samethatunderliestheuseofexperimentsandallowsexperimentalscientiststogeneralizefromoneexperiment‘toanother’:Yin(2003)callsthisanalyticalgeneralizability.Butawarningshouldbephrasedhere.Wheredesignprinciplesmayhavebeensupportedbyanumberofreplications,andanewcontextmaybesimilartotheonesfromwhichdesignprincipleshaveemerged,yeteachcontexthasuniquecharacteristicsthatjustifiesthatthedesignprinciplesshouldbeusedas‘heuristic’statements:theyprovideguidanceanddirection,butdonotgive‘certainties’.ItisinthiscontextthatReeves(2006)citesLeeCronbachoneofthemostinfluentialresearchersofthe20thcentury:“When we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion.”(Cronbach,1975:125)
OninterventionsDesignresearchbyitscharacteraimstobepracticallyrelevant.Itisinitiatedtodesignanddevelopinnovativeinterventionstomeetaneedfeltinacomplex,practicalsituationforwhichnoready-madesolutionsorguidelinesareavailable.Thereforedesignresearchersaimatdevelopinginterventions(suchasprograms,teaching-learningstrategiesandmaterials,productsandsystems)thatcanbeusedinpracticeandareempiricallyunderpinnedsolutionstotheproblemsidentified.
OnprofessionaldevelopmentOneofthefeaturesofdesignresearchisthecollaborationofresearchersandpractitioners.Thiscollaborationincreasesthechancethattheinterventionwillindeedbecomepracticalandrelevantfortheeducationalcontextwhichincreasestheprobabilityforasuccessfulimplementation.Buttheparticipationofpractitionersshouldalsobeseenasanimportantformofprofessionaldevelopment.Anextraspin-offmaybethatpractitionerswilldevelopanawarenessofhowresearchmaycontributetoimprovingtheirprofessionalcontext.
Design research differentiation
Designresearchisconductedthroughanumberofcyclesofdesignanddevelopmentresultingintheinitialimplementationoftheinterventioninalimitednumberofcontexts.Asstatedabove,designresearchhasusuallyanumberofstagesorphases(seealsoFigures1,2and3):• needsandcontentanalysis• prototypingphase(iterativecyclesofdesignandformativeevaluation)• assessmentphase(semi-summativeevaluation)
Nieveenetal.(2006)suggestthatdesignresearchthathasresultedinavalidatedandeffectiveintervention(asasolutionfortheproblemunderstudy),andindesignprinciplescanbefollowedbyeffect studies(notnecessarilypartofthesameresearchproject)withan
an introduction to educational design research 23
emphasisonupscalingtheinterventiontoawidercontext,andindoingsoaimingatdesignprinciplestestedinawiderdomain.Effectstudiesmayrangefromsmall-scalelearningexperimentstolarge-scalecomparativetestingofimpact(e.g.viarandomizedcontrolledtrials).
Afurtherdifferentiationindesignstudiesispossibleonthebasisofvariationsingoalsofdesignresearchvizvalidationstudiesversusdevelopmentstudies(seeVandenAkker,Gravemeijeretal.,2006;chapters5and10).
Validation studieshaveafocusondesigninglearningenvironmentsortrajectorieswiththepurposetodevelopandvalidatetheoriesabouttheprocessoflearningandhowlearningenvironmentscanbedesigned.Validationstudiesaimatadvancinglearningandinstructiontheories,suchas(Gravemeijer&Cobb,2006):• micro-theories:atthelevelofinstructionalactivities• localinstructiontheories:atthelevelofinstructionalsequence;• domain-specificinstructiontheories:atthelevelofpedagogicalcontentknowledge.Invalidationstudies,researchersdonotworkincontrolled(laboratoryorsimulated)settings,buttheychoosethenaturalsettingofclassroomas‘testbeds’(althoughtheytendtoworkwithabove-averagenumberofteachingstaff).Usually,thestagesinvalidationstudiesare(Gravemeijer&Cobb,2006):• environment preparation:elaboratingapreliminaryinstructionaldesignbasedonan
interpretativeframework;• classroom experiment:testingandimprovingtheinstructionaldesignorlocal
instructionaltheoryanddevelopinganunderstandingofhowitworks;• retrospective analysis:studyingtheentiredatasettocontributetothedevelopmentofa
localinstructionaltheoryand(improvementof)theinterpretativeframework.DiSessaandCobb(2004:83)warnthat“designresearchwillnotbeparticularlyprogressiveinthelongrunifthemotivationforconductingexperimentsisrestrictedtothatofproducingdomainspecificinstructionaltheories”.Butthepracticalcontributionliesindevelopingandimplementingspecificlearningtrajectoriesthatwereimplementedtotestthetheoreticalbasisofthedesign.(Nieveenetal,2006:153)
Development studiesaimtowardsdesignprinciplesfordevelopinginnovativeinterventionsthatarerelevantforeducationalpractice.“Developmentstudiesintegratestate-of-the-artknowledgefrompriorresearchinthedesignprocessandfine-tuneeducationalinnovationsbasedonpilotinginthefield.…Byunpackingthedesignprocess,designprinciplesthatcaninformfuturedevelopmentandimplementationdecisionsarederived.”(Nieveenetal.,2006:153).Twomaintypesofdesignprinciplescanbedistinguished(VandenAkker,1999):1. proceduraldesignprinciples:characteristicsofthedesignapproach;2. substantivedesignprinciples:characteristicsofthedesign(=intervention)itself.
an introduction to educational design research24
Figurethreesummarizesthecharacteristicsofaresearchcycleconsistingofdesignstudiesandeffectstudies(asdevelopedbyNieveenetal.;2006:155):
Design researchEffectiveness research
Validation studies Development studies
Design aim To elaborate and validate theories
To solve educational problems
-
Quality focus of design
Theoretical quality of design
Practicality of intervention
Effectiveness of intervention
Knowledge claim/ scientific output
Domain-specific instruction theories
Broadly applicable design principles
Evidence of impact of intervention
Methodological emphasis
Iterative design with small scale testing in research setting
Iterative development with formative evaluation in various user settings
Large scale, comparative field experiments
Practical contribution Specific learning trajectories for a specific classroom
Implemented interventions in several contexts/classrooms
Evidence-basedChange at large scale
Figure 4: Educational engineering research cycle (from Nieveen et al., 2006)
Itisimportanttonotethatthisdistinctionbetweenvalidationanddevelopmentstudiesisconceptuallyimportant,butthatinpracticemanyresearchprojecthaveaimsthatareacombinationofsolvingproblemsineducationalpracticeandelaboratingandvalidatingtheories(designprinciples).
Afurtherdifferentiationofdesignresearchisconceivable.Forexample,onecanimaginethatthedisseminationandimplementationofaparticularprogramissupportedbydesignresearch–theresultinginterventionisthesuccessfullydisseminatedandimplementedprogram,whilstthesystematicreflectionanddocumentationoftheprocessleadstoasetofproceduresandconditionsforsuccessfuldisseminationandimplementation(thedesignprinciples).
Asafinalnote,thedifferentiationbetweentypesofdesignresearch,suchasvalidationstudiesversusdevelopmentstudies,servesmainlyconceptualpurposes.Inpractice,designresearchersmaycombinethetwoorientationsintheirresearch.Forexample,startingfromacomplexandpersistentproblemine.g.scienceeducation,theresearchgroupmaydecidetoapplythedesignprinciples(localtheories)resultingfromotherstudiesintheirresearch.Indoingsotheyarenotonlydevelopinganintervention,butatthesametimeexploringthevalidityofdesignprinciples(theory)developedinanothercontextfortheirownproblemcontext.
an introduction to educational design research 25
Howisdesignresearchconducted?Designresearchisconductediterativelyasacollaborationofresearchersandpractitionersinareal-worldsetting.Onlythenthetwoprincipaloutputs(designprinciplesandempiricallyunderpinnedinnovativeinterventions)canberealized.Doingresearchinsuchasettingischallenginganddemandsacarefulresearchdesign.Itisthereforeimportanttoreflectnotonlyonthecyclical,iterativecharacterofthesystematicdesignoftheintervention,butalso–becauseitisresearch-tomakeexplicitthetenetsthatformthefoundationofthistypeofresearch(McKenneyetal.,2006)
McKenneyetal.(2006:77)definethreetenetstoshapedesignresearchforthecurriculumdomain(butthetenetsalsoapplytootherdomains):• Rigor–fordesignresearchtobeabletoresultinvalidandreliabledesignprinciples,the
researchhastomeetrigorousstandardsandapplytheguidingprinciplesforscientificresearchasmentionedbyShavelson&Towne(2002;mentionedabove).Muchliteratureisavailabletoguideresearchinnaturalsettingsthatofferssupporttoissueslikeinternalandexternalvalidity,reliabilityandutililizationoftheresearch.
• Relevance:Designresearchaimstoberelevantforeducationalpractice(andpolicy).Anecessaryconditionforthisisthattheresearchgroupmusthaveagoodworkingknowledgeofthetargetsettingandbeinformedbyresearchanddevelopmentsactivitiestakingplaceinnaturalsettings(ortestbeds).
• Collaboration:fordesignresearchtoberelevantforeducationalpractice,thedesignanddevelopmentactivitiesmustbeconductedincollaborationwithandnotjustforprofessionalsfromeducationalpractice.
Asexplainedinthebeginningofthischapter,designresearchiscyclicalandeachiterationorcyclecontributestosharpeningtheaimsandtobringingtheinterventionsclosertothedesireddesignoutcomesandresearchoutputs.AsisillustratedinFigures1-3,designresearchusuallygoesthroughseveralstageswhichNieveenetal.(2006:154)phraseasfollows(seealsop.15):• preliminary research:thoroughcontextandproblemanalysisalongwiththe
developmentofaconceptualframeworkbasedonliteraturereview;• prototyping stage:settingoutdesignguidelines,optimizingprototypesofthe
interventionthroughcyclesofdesign,formativeevaluation,andrevision–itisimportanttonotethateachcycleinthestudyisapieceofresearchinitself(i.e.havingitsresearchorevaluationquestiontobeaddressedwithaproperresearchdesign);
• assessment stage (summative evaluation):oftenexplorestransferabilityandscaling,alongwith(usuallysmall-scaleevaluationof)effectiveness;and
• systematic reflection and documentation:thisarecontinuousactivities(asillustratedinFigure3)thattakesplaceduringallcyclesintheresearch–however,attheendthe
an introduction to educational design research26
researcherportraystheentirestudytosupportretrospectiveanalysis,followedbyspecificationofdesignprinciplesandarticulationoftheirlinkstotheconceptualframework.
Itisbeyondthescopeofthischaptertodiscussindetailhowtoperformthesestages.Butanexceptionismadeforformativeevaluation,becausethisisthekeyresearchactivityindesignresearchaimedatimprovingthequalityoftheconsecutiveprototypesoftheintervention.
Formativeevaluationindevelopmentresearch6
BasedonpriorworkNieveen(1999;seealsoChapter5)proposesfourgenericcriteriaforhighqualityinterventions(seeTable1).Sheexplainsthesecriteriaasfollows:Thecomponentsoftheinterventionshouldbebasedonstate-of-the-artknowledge(content validity)andallcomponentsshouldbeconsistentlylinkedtoeachother(construct validity).Iftheinterventionmeetstheserequirementsitisconsideredtobevalid.Anothercharacteristicofhigh-qualityinterventionsisthatend-users(forinstancetheteachersandlearners)considertheinterventiontobeusableandthatitiseasyforthemtousethematerialsinawaythatislargelycompatiblewiththedevelopers’intentions.Iftheseconditionsaremet,wecalltheseinterventionspractical.Athirdcharacteristicofhighqualityinterventionsisthattheyresultinthedesiredoutcomes,i.e.thattheinterventioniseffective.
Criterion
Relevance (also referred to as content validity)
There is a need for the intervention and its design is based on state-of-the-art (scientific) knowledge.
Consistency (also referred to as construct validity)
The intervention is ‘logically’ designed.
Practicality The intervention is realistically usable in the settings for which it has been designed and developed.
Effectiveness Using the intervention results in desired outcomes.
Table 1: Criteria for high quality interventions (from Nieveen, 1999; Chapter 5)
Giventhecharacterofdesignresearch,thesefourcriteriamaygetdifferentemphasisindifferentstagesoftheresearchasisillustratedbyFigure5.Forexample,duringthepreliminaryresearchwheretheemphasisisonanalyzingtheproblemandreviewingtheliterature,thecriterionofrelevance(contentvalidity)isthemostdominant,withsomeattentionforconsistency(constructvalidity)andpracticality,whilstinthatstatenoattentionisyetgiventoeffectiveness.Ontheotherhand,intheprototypingstagemuch
6) SeealsoNieveen’schapter5inthisbookinwhichshediscusseshowtodotheformativeevaluationindesignresearch
an introduction to educational design research 27
attentionhastobepaidintheformativeevaluationtothecriterionofpracticality,whilsteffectivenesswillbecomeincreasinglyimportantinlateriterations.Finally,inassessmentstageofsummativeevaluation,thefocuswillbeonpracticalityandeffectiveness(seeFigure5,andFigure2forthestages).
Stage Criteria Short description of activities
1 Preliminary research Emphasis mainly on content validity, not much on consistency and practicality
Review of the literature and of (passed and/or present) projects addressing questions similar to the ones in this study. This results in (guidelines for) a framework and first blueprint for the intervention.
2 Prototyping stage Initially: consistency (construct validity) and practicality. Later on mainly practicality and gradually attention for efficiency.
Development of a sequence of prototypes that will be tried out and revised on the basis of formative evaluations. Early prototypes can be just paper-based for which the formative evaluation takes place via expert judgments.
3 Assessment phase practicality and efficiency
Evaluate whether target users can work with intervention (practicality) and are willing to apply it in their teaching (relevance & sustainability). Also whether the intervention is effective.
Figure 5: Evaluation criteria related to stages in design research
Formativeevaluationtakesplaceinallphasesanditerativecyclesofdesignresearch.AsillustratedbyFigure5,formativeevaluationservesdifferentfunctions,or-inotherwords-isaimedatdifferentcriteria(orcombinationsofthese)inthevariousdevelopmentcycles,eachbeingamicro-cycleofresearchwithitsspecificresearch/evaluationquestionandrelatedresearch/evaluationdesign.OnemaysaythatformativeevaluationhasvariouslayersinadesignresearchprojectasisillustratedinFigure6,takenfromTessmer(1993):frommoreinformalintheearlystagesofaproject(self-evaluation,one-to-oneevaluation,expertreview)tosmallgroupevaluationaimedattestingthepracticalityandeffectiveness,toafullfieldtest(ifapplicable).Theresearch/evaluationdesignforeachcycleshouldreflectthespecificfocusandcharacterofthecycle–seeChapter5byNieveenformoredetails.
an introduction to educational design research28
Figure 6: Layers of formative evaluation (taken from Tessmer, 1993)
Figure6alsoillustratesthatmanypossiblemethodsofformativeevaluationcanbechosen,suchas7
• expertreviewand/orfocusgroups(importanttoconsider‘expertsinwhat’)• self-evaluationorscreening(usingchecklistofimportantcharacteristicsordesign
specifications)• one-to-oneevaluationorwalkthrough(withrepresentativeoftargetaudience)• smallgroupormicro-evaluation• fieldtestortry-outDesignresearchersshouldchooseforeachphaseandforeachprototypeformativeevaluationapproachesthataresuitableforthepurposeofthatparticularstageoftheresearch.Designresearchhastomeetcriteriaforgoodresearch.Itisthereforeimportantthatforeachdevelopmentcycletheresearcher(orresearchgroup)appliesthemethodological‘rules’fordoingresearch,i.e.foridentifyingthetargetaudienceandsampling,forinstrumentdevelopmentandapplytriangulationtoobtaingoodqualityinformation.But
7) seealsoChapter5byNieveen
HighResistanceto Revision
LowResistanceto Revision
Revise
Revise
Revise
Field TestUser Acceptance, Implementability
Organizational Acceptance
Small GroupEffectiveness, Appeal
Implementability
Expert ReviewContent, Design,Techical Quality
One-to-OneCiarity, AppealObvious Errors
Self-Evaluationobvious errors
an introduction to educational design research 29
giventhelayersofformativeevaluationindesignresearch,intheearlycyclesofdevelopmenttheevaluationdesigncanbelessrigorousthaninlaterphases.Figure7adaptedfromNieveen(1999)presentsanexamplethatillustrateshowvariousformativeevaluationmethodsareusedfortherespectiveprototypesinaprojectaimedatdevelopingacomputerassistedsupportsystemforcurriculumdevelopers.
prelimcomp.based
paper-based computer-based versions
final version
Users (n=5)
experts (n=3)
users (n=5)
experts (n=6)
users (n=4)
users (n=4)
users (n=17)
Validity content *) √ ea √ ea
interface √ ea
Practicality content √ wt √ wt √ ea √ me √ to √ ft
interface √ wt √ wt √ ea √ me √ to √ ft
Effectiveness entire system √ to √ ft
*): Content refers to the content of the support system
√ = primary attention of prototype and of formative evaluation
Methods of formative evaluation: me = micro evaluation; wt = walk through; ea = expert appraisal;
ft = field trial; to = try-out
Figure 7: Focus of design and formative evaluation of the prototypes for computer assisted support system for curriculum development (adapted from Nieveen, 1999)
Afinalnoteonthecriteriaofpracticalityandeffectiveness.Itmayoccurincertainstudiesthattheresearcher(orresearchcollaborative)cannotdoafinalfieldtrialoftheinterventionwiththefull(orasampleofthe)targetgroup,buthastorestricthimselftoexpertappraisaland/ormicro-evaluationofthefinalprototypeoftheintervention.Itisobviousthatinsuchasituationtheactual practicalityandtheactual effectivenessoftheinterventioncannotbedemonstrated,butonlyconclusionsabouttheexpected practicalityandtheexpected effectivenesscanbedrawn.Moreevaluationwillthenbeneededtodemonstratetheactual practicalityandtheactual effectiveness.ThiscanbeillustratedwithanexampleadaptedfromMafumiko(2006)whoconducteddesignresearchtoinvestigatewhethermicro–scaleexperimentationcancontributetoimprovingthechemistrycurriculuminTanzania.HisresearchdesignhasbeensummarizedinFigure8.
an introduction to educational design research30
Figure 8: Example of research research design (adapted from Mafumiko, 2006)
SupposearesearcherwouldrestricthimselftothedevelopmentofprototypesoftheinterventionasillustratedinFigure8,anddoesnotplantoinvestigatewhetherVersionIVworksinthetargetcontext.Insuchasituationthemosthecanconcludeiswhetherhisinterventionisexpectedtobepracticalandeffectiveforthetargetcontext.Onlywhenhewouldconductafieldtest,hewillbeinthepositiontodecideuponactualpracticalityandactualeffectiveness(whichiswhatMafumikodid).
Design research dilemmas
Designresearchisconductedinclosecollaborationwitheducationalpractice.Notonlytheproblemaddressedissituatedineducationalpractice,butakeyfeatureofthisresearchisthateducationalpractitionersareactivelyinvolved,oftenasmembersoftheresearchteam.Thisleadstoanumberofchallengesthataretypicalforthistypeofresearch.McKenneyetal.(2006:83,84)havediscussedsomeoftheseandprovidesuggestionsforhowtoaddressthem.Theirpointsarebrieflysummarizedhere.
1. theresearcherisdesignerandoftenalsoevaluatorandimplementer.Severalmeasurescanbetakentocompensateforthispotentialconflictofinterest:• makeresearchopentoprofessionalscrutinyandcritiquebypeopleoutsidetheproject• theresearcherappliesthefollowingruleofthumb:shiftfromadominanceof‘creative
designer’perspectiveintheearlystage,towardsthe‘criticalresearcher’perspectiveinlaterstages(thisisreflectedinTessmer’slayersofformativeevaluation,Figure6)
Summative
evaluation
Appraisel by3 experts
Tryout in3 classrooms
Panel session with experts
Field test infour schools
Tryout with teacher educ students
Version I Version II Version III Version IV
Development of prototypes
Design guidelines &specifications
an introduction to educational design research 31
• haveagoodqualityofresearchdesign,e.g. - strong chain of reasoning(Krathwohl,1998)-themetaphorexpressestheideathat
eachpartoftheresearchdesignisequallyimportant - triangulation–toincreasethequalityofdataandofanalysistriangulationofdata
sourcesanddatacollectionmethodsshouldbeapplied,aswellasinvestigatortriangulationtoavoidtheinfluenceofanyspecificresearcher(seee.g.Denscombe,2007;136)
- empirical testingofboththeusabilityandtheeffectivenessoftheintervention - systematic documentation, analysisandreflectionofthedesign,development,
evaluationandimplementationprocessandtheirresults - haveattentionforvalidityandreliabilityofdataandinstruments - applyavarietyofmethodsandtactics:e.g.usepractitionersandotherresearchersas
‘criticalfriends’;usemultipleobservers/ratersandcalculateinter-observer/raterreliability,etc.
2. real-worldsettingsbringreal-worldcomplicationsDesignresearchisconductedinreal-worldsettingsbecauseitaddressescomplexproblemsineducationalpractice.Oneoftheproblemsisthattheresearchercanbea‘culturalstranger’(Thijs,1999)inthesettingoftheresearchandthatparticipants(e.g.principals,teachersnotinvolvedintheresearch,etc)arehesitanttobecompletelyopentoaresearchercomingfromtheoutside.McKenneyetal.(2006:84)pointstotheimportanceofcollaborationandmutualbeneficialactivitiestogainparticipants’trustandthoroughunderstandingofthecontext(i.e.insiderperspective).Ontheotherhand,theyalsopointtotheadvantagestobeanoutsiderasthismayallowtheresearchertodevelopadegreeofobjectivityand“freedom(orforgiveness)forhonestythatisnotpermittedtothosewithinaparticulargroup”(o.c.85)
3. adaptabilityDesignresearchiscyclicalandtakesplaceinreal-worldsettings.Eachcyclehastotakethefindingsofthepreviousonesintoaccount.Soontheonehandtheresearchdesignhastochange(ordevelop)fromonecycletotheother,whilstontheotherhandanever-changingresearchdesigncanbeweak.Inthiscontext,McKenneyetal.(2006:84)refertothenotionofevolutionaryplanning,i.e.“aplanningframeworkthatisresponsivetofielddataandexperiencesasacceptablemomentsduringthecourseofthestudy”.Thisisalreadyalludedtointhediscussionofformativeevaluation(seeFigure6fromTessmerandtheexampletakenfromNieveen,1999).Theneedforadaptabilitypertainsalsototheroleoftheresearcher.AccordingtoVandenAkker(2005,inMcKenneyetal.,2006),thesynergybetweenresearchandpracticecanbemaximizedwhenresearchersdemonstrateadaptabilityby:
an introduction to educational design research32
(i) beingprepared,wheredesirable,totakeontheadditionalroleofdesigner,advisor,andfacilitator,withoutlosingsightoftheirprimaryroleasresearcher,
(ii) beingtolerantwithregardtotheoftenunavoidablyblurredroledistinctionsandremainingopentoadjustmentsintheresearchdesignifprojectprocesssodictates,
(iii)allowingthestudytobeinfluenced,inpart,bytheneedsandwishesofthepartners,duringwhatisusuallyalong-termcollaborativerelationship.
Suchadaptabilityrequiresstrongorganizationalandcommunicativecapabilitiesonbehalfoftheresearcher,aswellassoundunderstandingtheresearchprocesssothatcarefulchangesandchoicesthatmaximizevalueandminimizethreatstoqualityaremade.(McKenneyetal.,2006:84).
Toaddressthechallengesmentioned,McKenneyetal.(2006:85,86)presentafewguidelines for conducting design researchthatmayhelpresearchersmonitoringthescientificcharacterofhis/herresearch:- haveanexplicitconceptualframework(basedonreviewofliterature,interviewsof
experts,studyingotherinterventions)- developcongruentstudydesign,i.e.applyastrongchainofreasoningwitheachcycle
havingitsresearchdesign- usetriangulation(ofdatasource,datatype,method,evaluatorandtheory)toenhance
thereliabilityandinternalvalidityofthefindings- applybothinductiveanddeductivedataanalysis- usefull,context-richdescriptionsofthecontext,designdecisionsandresearchresults- membercheck,i.e.takedataandinterpretationsbacktothesourcetoincreasethe
internalvalidityoffindings.Itisbeyondthescopeofthispapertoelaborateontheseguidelinesfurther–seeMcKenneyetal.(2006;85,86)andresearchmethodologybooks.
Concluding remarks
Inthefieldofeducationthereismuchneedforresearchrelevantforeducationalpractice.Wehavearguedthatforcomplexpracticalproblemsandforresearchquestion(s)callingforthedesignanddevelopmentofaninterventiondesignresearchistheappropriateresearchapproach.
Givenitsfocusonpracticalproblemsanditsnatureofconductingtheresearchinareal-worldsettingwithactiveinvolvementofpractitioners,designresearchmaylooklikeactionresearch.Soonemaywonderhowdesignresearchisrelatedtoactionresearch.Indeed,actionresearchisalsodealingwithreal-worldproblems,aimingatimprovingpractice,cyclicalinnatureandparticipative(Denscombe,2007),buttheessentialdifferenceisthat
an introduction to educational design research 33
actionresearchisnotaimedatgeneratingdesignprinciples–ithasaparticularnicheamongprofessionalswhowanttouseresearchtoimprovetheirpractices(o.c.:122).
Wediscussedhowdesignresearchersshouldstriveforgeneralizabledesignprinciplesinthemeaningofgeneralizingtoabroadertheory.Whendesignresearchisconductedwithintheframeworkofaprogramofresearchaddressingfundamentalproblemsineducationalpractice,itwillresultinaspecificbodyofknowledge,vizsubstantiveandproceduraldesignprinciplesthatmaycontributetoimproveeducation.Ontheotherhandmanyquestionsarestilltobeaddressedastherearemanytypesofpracticalproblemsandthereforemanytypesofresearchgoalsforwhichdesignresearchmaybethebestapproach(e.g.,Reeves(2000)mentionssixdifferenttypesofgoals).
VandenAkker,Gravemeijer,McKenneyandNieveen(2006)reportthepresentationsanddiscussionsataseminardedicatedtoeducationaldesignresearch.Theirbookpoints-nexttodiscussinganumberofapproachestodesignresearchbyGravemeijerandCobb(2006),Reeves(2006)andMcKenneyetal.(2006)-toissueslikeassessingthequalityofdesignresearchproposals(chaptersbyPhillips,2006,andbyEdelson,2006)andthequalityofdesignresearch(chapterbyKelly,2006)whichneedfurtherreflectionandelaboration.
Finally,anumberofresearchreportsanddissertationshavebeenpublishedwhichareexemplaryforhowdesignresearchcanbeconducted(seechapter6forexamples).Butfordesignresearchtomaturefurthermoreresearchprojectsinavarietyofcontextsshouldnotonlybeconducted,butalsoreportedanddiscussedinresearchjournalsandatconferences.
OurhopeisthatthecommunityofeducationaltechnologistsinChinawillembarkonthisresearchendeavorandwillactivelycontributetothefurtherdevelopmentofeducationaldesignresearch.
Acknowledgement: in preparing this chapter, much use has been made of Van den Akker, Gravemeijer, McKenney and Nieveen (2006).
an introduction to educational design research34
References
Bannan-Ritland,B.(2003).TheRoleofDesigninResearch:TheIntegrativeLearningDesignFrameworkEducational Researcher, 32(1),21-24.
Barab,S.&Squire,K.(2004).Design-basedresearch:puttingastakeintheground.Journal of the Learning Sciences, 13(1),1-14.
Brown,A.L.(1992).Designexperiments:Theoreticalandmethodologicalchallengesincreatingcomplexinterventionsinclassroomsettings.The Journal of The Learning Sciences, 2(2),141-178.
Cronbach,L.J.(1975).Beyondthetwodisciplinesofscientificpsychology.American Psychologist, 30(2),116-27.
Denscombe,M.(2007,3rded)).The good research guide for small-scale social research projects. Buckingham(UK):OpenUniversityPress.
Design-BasedResearchCollective(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher, 32(1),5-8.
diSessa,A.A.&Cobb,P.(2004).Ontologicalinnovationandtheroleoftheoryindesignexperiments.Educational Researcher, 32(1),77-103.
Edelson,D.C.(2006).Balancinginnovationandrisk:assessingdesignresearchproposals.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,100-106.
Gravemeijer,K.&Cobb,P.(2006).Designresearchfromalearningdesignperspective.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research. London:Routledge,17-51.
Gustafson,K.L.&Branch,R.M.(2002).Survey of instructional development models(4thed).Syracuse(NY,USA):ERICClearinghouseonInformation&Technology.
Kelly,A.E.(2006).Qualitycriteriafordesignresearch:evidenceandcommitments.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,107-118.
Knippels,M.C.P.J.(2002).Coping with the abstract and complex nature of genetics in biology education.Utrecht(TheNetherlands):UniversityofUtrecht–CD-BètaPress.
Krathwohl,D.R.(1998,2nded).Methods of Educational and Social Science Research: An integrated approach.NewYork:Longman
Lijnse,P.L.(1995).“DevelopmentalResearch”asawaytoanempiricallybased“DidacticalStructure”ofScience.Science Education, 29(2),189-199.
Mafumiko,FideliceSimbagungileMbaruku(2006).Micro-scale experimentation as a catalyst for improving the chemistry curriculum in Tanzania.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/55448
McKenney,S.(2001).Computer-based support for science education materials developers in Africa: exploring potentials.Doctoraldissertation.Enschede:UniversityofTwente.
an introduction to educational design research 35
McKenney,S.,Nieveen,N.&VandenAkker,J.(2006).Designresearchfromacurriculumperspective.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research. London:Routledge,62-90.
Nieveen,N.(1999).Prototypingtoreachproductquality.InJ.vandenAkker,R.M.Branch,K.Gustafson,N.Nieveen,&T.Plomp(Eds),Design approaches and tools in education and training.Boston:KluwerAcademic,125-136.
Nieveen,N.,McKenney,S.&VandenAkker,J.(2006).Educationaldesignresearch:thevalueofvariety.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,151-158.
Phillips,D.C.(2006).Assessingthequalityofdesignresearchproposals:somephilosophicalperspectives.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,93-99.
Reeves,T.(2000).Enhancingtheworthofinstructionaltechnologyresearchthrough“designexperiments”andotherdevelopmentalstrategies.AERA.AERA.RetrievedOct.20,2006fromhttp://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf
Reeves,T.C.(2006).Designresearchfromatechnologyperspective..In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,52-66.
Shavelson,R.&Towne,L.(2002).Scientific research in education.WashingtonDC:NationalAcademicPress(website:http://books.nap.edu/books)
Tecle,A.T.(2006).The potential of a professional development scenario for supporting biology teachers in Eritrea.Doctoraldissertation.Enschede(TheNetherlands):UniversityofTwente.
Tessmer,M.(1993).Planning and conducting formative evaluations.London:KoganPage.Thijs,A.(1999).Supporting science curriculum reform in Botswana: The potential of peer
coaching.Doctoraldissertation.Enschede:UniversityofTwente.VandenAkker,J.(1999).PrinciplesandMethodsofDevelopmentResearch.InJ.vanden
Akker,R.M.Branch,K.Gustafson,N.Nieveen,&T.Plomp(Eds),Design approaches and tools in education and training.Boston:KluwerAcademic,1-14.
VandenAkker,J.(2005).Hoekansamenwerkingleidentotmeersuccesenwijsheidinonderwijsontwikkeling?[Howcancollaborationleadtomoresuccessandwisdomineducationdevelopment?].Pedagogische Studiën, 82(4),343-7.
VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge.ISBN10:0-415-39635-2(pbk)
Wademan,M.(2005).Utilizing Development Research to Guide People Capability Maturity Model Adoption Considerations.Unpublisheddoctoraldissertation,SyracuseUniversity(NewYork,USA).
Yin,R.K.(2003,3eded).Case study research: design and methods.NewburyPark(CA,USA):Sage–AppliedSocialResearchMethodsSeries,volume5.
an introduction to educational design research 37
2. CurriculumDesignResearch Jan van den Akker
Introduction
Thetitleofthischapter (Curriculum Design Research)intentionallycombinestwofields:‘curriculumdesign’and‘designresearch’.Itsymbolizestheaimofthistexttodiscussthefunctionandformsofdesignresearchfromacurricularperspective.Inparticular,itfocusesonhowdesignresearchcanincreasethequalityofcurriculumdesignanddevelopment.Also,itillustrateshowtherelevanceofeducationalresearch-awidelydebatedissue-canbenefitfromaconnectiontocurriculumpoliciesandpractices.Giventhisaimithelpstohaveanumberofbasicconceptsandanalyticalperspectivesavailablethatcanstructurecurriculardeliberationsandreducethecomplexityofcurriculumtasks.Thusmyinitialfocusinthischapter(buildingonvandenAkker,2003)isonsummarizingasetofconceptsandperspectivesthathelptoincreasethetransparencyandbalanceofcurriculumanalysis,developmentanddiscourse.Then,thefocuswillshifttowards(curriculum)designresearch(buildingonvandenAkker,1999,2006,andonvandenAkker,Gravemeijer,McKenneyandNieveen,2006).First,Iwillsketchthepotentialandcharacteristicsofdesignresearchinaddressingcomplexcurriculumchallenges.Second,Iwilladdressanumberofmethodologicalissues.Finally,Iwillpayattentiontoaclassicprobleminalleducationalresearch:generalizationoffindings.
Curriculum, what’s in a name?
Whenthereisamyriadofdefinitionsofaconceptintheliterature(aswithcurriculum),itisoftendifficulttokeepaclearfocusonitsessence.Inthosecasesitoftenhelpstosearchfortheetymologicaloriginoftheconcept.TheLatinword‘curriculum’(relatedtotheverb‘currere’i.e.running)referstoa‘course’or‘track’tobefollowed.Inthecontextofeducation,wherelearningisthecentralactivity,themostobviousinterpretationofthewordcurriculumisthentoviewitasacourse,trajectory,or‘plan for learning’(cf.Taba,1962).Thisveryshortdefinition(reflectedinrelatedtermsinmanylanguages)limitsitselftothecoreofallotherdefinitions,permittingallsortsofelaborationsforspecificeducationallevels,contexts,andrepresentations.Obviously,contextualspecificationisalwaysneededincurriculumconversationstoclarifytheperspective.Giventhissimpledefinition,adifferentiationbetweenvariouslevelsofthecurriculumhasproventobeveryusefulwhentalkingaboutcurricularactivities(policy-making;designanddevelopment;evaluationandimplementation).Thenextdistinctionappearstobehelpful:• International/comparative(orsupralevel)• System/society/nation/state(ormacro)level(e.g.nationalsyllabiorcoreobjectives)
an introduction to educational design research38
• School/institution(ormeso)level(e.g.school-specificcurriculum)• Classroom(ormicro)level(e.g.textbooks,instructionalmaterials)• Individual/personal(ornano)level.
Thesupralevelusuallyreferstointernationaldebatesoragreementsonaimsandqualityofeducation,sometimesfuelledbyoutcomesofinternationallycomparativestudies(cf.PISAorTIMSS1).Curriculumdevelopmentatthesupralevelisusuallyofa‘generic’nature,while‘site-specific’approachesaremoreapplicableforthelevelsclosertoschoolandclassroompractice.Moreover,theprocessofcurriculumdevelopmentcanbeseenasnarrow(developingaspecificcurricularproduct)orbroad(alongterm,ongoingprocessofcurriculumimprovement,oftenincludingmanyrelatedaspectsofeducationalchange,e.gteachereducation,schooldevelopment,testingandexaminations).Inordertounderstandproblemsofcurriculumdecision-makingandenactment,abroaderdescriptionofcurriculumdevelopmentisoftenmostappropriate:usuallyalongandcyclicprocesswithmanystakeholdersandparticipants;inwhichmotivesandneedsforchangingthecurriculumareformulated;ideasarespecifiedinprogramsandmaterials;andeffortsaremadetorealizetheintendedchangesinpractice.
Moreover,curriculacanberepresentedinvariousforms.Clarificationofthoseformsisespeciallyusefulwhentryingtounderstandtheproblematiceffortstochangethecurriculum.Acommonbroaddistinctionisbetweenthethreelevelsofthe‘intended’,‘implemented’,and‘attained’curriculum.Amorerefinedtypology(vandenAkker,2003)isoutlinedinbox1.
INTENDED Ideal Vision (rationale or basic philosophy underlying a curriculum)
Formal/Written Intentions as specified in curriculum documents and/or materials
IMPLEMENTED Perceived Curriculum as interpreted by its users (especially teachers)
Operational Actual process of teaching and learning (also: curriculum-in-action)
ATTAINED Experiential Learning experiences as perceived by learners
Learned Resulting learning outcomes of learners
Box 1: Typology of curriculum representations
Traditionally,theintendeddomainreferspredominantlytotheinfluenceofcurriculumpolicymakersandcurriculumdevelopers(invariousroles),theimplementedcurriculum
1) PISAistheOECDProgrammeforInternationalStudentAssessment,asurveyeverythreeyearsofthe15-year-olds.TIMSSistheTrendsInMathematicsandSciencesStudy,conductedevery4yearsbytheInternationalAssociationfortheEvaluationofEducationalAchievement(IEA)inprimaryandsecondaryeducation.
an introduction to educational design research 39
relatesespeciallytotheworldofschoolsandteachers,andtheattainedcurriculumhastodowiththestudents.
Besidesthisdifferentiationinrepresentations,curriculumproblemscanbeapproachedfromvariousanalyticalangles.Forexample,Goodlad(1994)distinguishesthefollowingthreedifferentperspectives:• substantive,focusingontheclassicalcurriculumquestionaboutwhatknowledgeisof
mostworthforinclusioninteachingandlearning;• technical-professional,referringtohowtoaddresstasksofcurriculumdevelopment;• socio-political,referringtocurriculumdecision-makingprocesses,wherevaluesand
interestsofdifferentindividualandagenciesareatstake.Somemightarguethatthislististoolimitedasitrefersespeciallytocurriculumissuesfor‘traditional’planningforlearninginschools,anddoesnotincludethemore‘critical’perspectivesthatareamplypresentincurriculumtheoryliterature(e.g.Pinar,Reynolds,Slattery&Taubman,1995).However,fromaprimaryinterestincurriculumimprovement,thethreeperspectivesseemusefulandappropriate.
The vulnerable curriculum spider web
Oneofthemajorchallengesforcurriculumimprovementiscreatingbalanceandconsistencybetweenthevariouscomponentsofacurriculum(i.e.planforlearning).Whatarethosecomponents?TherelativelysimplecurriculumdefinitionbyWalker(2003)includesthreemajorplanningelements:content,purposeandorganizationoflearning.However,curriculumdesignandimplementationproblemshavetaughtusthatitiswisetopayexplicitattentiontoamoreelaboratedlistofcomponents.Elaboratingonvarioustypologies,wehavecometoadheretoaframework(seeBox2)oftencomponentsthataddresstenspecificquestionsabouttheplanningofstudentlearning.
Rationale or Vision Why are they learning?
Aims & Objectves Toward which goals are they learning?
Content What are they learning?
Learning activities How are they learning?
Teacher role How is the teacher facilitating learning?
Materials & Resources With what are they learning?
Grouping With whom are they learning?
Location Where are they learning?
Time When are they learning?
Assessment How to measure how far learning has progressed?
Box 2: Curriculum components
an introduction to educational design research40
The‘rationale’(referringtooverallprinciplesorcentralmissionoftheplan)servesasmajororientationpoint,andthenineothercomponentsareideallylinkedtothatrationaleandpreferablyalsoconsistentwitheachother.Foreachofthecomponentsmanysub-questionsarepossible.Notonlyonsubstantiveissues(seethenextsection),but,forexample,alsoon‘organizational’aspectsas:• Grouping: - Howarestudentsallocatedtovariouslearningtrajectories? - Arestudentslearningindividually,insmallgroups,orwhole-class?• Location: - Arestudentslearninginclass,inthelibrary,athome,orelsewhere? - Whatarethesocial/physicalcharacteristicsofthelearningenvironment?• Time: - Howmuchtimeisavailableforvarioussubjectmatterdomains? - Howmuchtimecanbespentonspecificlearningtasks?Therelevanceofthesecomponentsvariesacrossthepreviouslymentionedcurriculumlevels(supra,macro,meso,micro,nano)andrepresentations.Afewexamplesmayillustratethis.• Curriculumdocumentsatthemacro-levelwillusuallyfocusonthefirstthree
components(rationale,aims&objectives,content;ofteninratherbroadterms),sometimesaccompaniedbyanoutlineoftimeallocationsforvarioussubjectmatterdomains.
• Whenonetakestheoperationalcurriculuminschoolsandclassroomsinmind,alltencomponentshavetobecoherentlyaddressedtoexpectsuccessfulimplementationandcontinuation.
• Thecomponentsoflearningactivities,teacherrole,andmaterials&resourcesareatthecoreofthemicro-curriculumintheclassroom.
• Thecomponentofassessmentdeservesseparateattentionatalllevelsandrepresentationssincecarefulalignmentbetweenassessmentandtherestofthecurriculumappearstobecriticalforsuccessfulcurriculumchange.
Ourpreferentialvisualizationofthetencomponentsistoarrangethemasaspiderweb(Figure1),notonlyillustratingitsmanyinterconnections,butalsounderliningitsvulnerability.Thus,althoughtheemphasisofcurriculumdesignonspecificcomponentsmayvaryovertime,eventuallysomekindofalignmenthastooccurtomaintaincoherence.AstrikingexampleisthetrendtowardintegrationofICTinthecurriculum,withusuallyinitialattentiontochangesinmaterialsandresources.Manyimplementationstudieshaveexemplifiedtheneedforamorecomprehensiveapproachandsystematicattentiontotheothercomponentsbeforeonecanexpectrobustchanges.
an introduction to educational design research 41
Thespiderwebalsoillustratesafamiliarexpression:everychainisasstrongasitsweakestlink.Thatseemsanotherveryappropriatemetaphorforacurriculum,pointingtothecomplexityofeffortstoimprovethecurriculuminabalanced,consistentandsustainablemanner.
Figure 1: Curricular spider web
Perspectives on substantive choices
Aclassicapproachtotheeternalcurriculumquestionofwhattoincludeinthecurriculum(orevenmoredifficultaswellasurgent:whattoexcludefromit)istosearchforabalancebetweenthreemajorsourcesororientationsforselectionandprioritysetting:• Knowledge:whatistheacademicandculturalheritagethatseemsessentialfor
learningandfuturedevelopment?• Society:whichproblemsandissuesseemrelevantforinclusionfromtheperspectiveof
societaltrendsandneeds?• Learner:whichelementsseemofvitalimportanceforlearningfromthepersonaland
educationalneedsandinterestsofthelearnersthemselves?
Asse
ssm
ent
Aims & Objectives
Content
Learning activitiesTe
ache
r rol
e
Materials &
Resources
Location
Time
Rationale
Grouping
an introduction to educational design research42
Answerstothesequestionsusuallyconstitutetherationaleofacurriculum.Inevitably,choiceshavetobemade,usuallyinvolvingcompromisesbetweenthevariousorientations(andtheirrespectiveproponentsandpressuregroups).Oftentimes,effortsfailtoarriveatgenerallyacceptable,clearandpracticalsolutions.Theresultofaddingupallkindsofwishesisthatcurriculatendtogetoverloadedandfragmented.Implementationofsuchincoherentcurriculaeventuallytendstoleadtostudentfrustrations,failure,anddropout.Howtocreateabettercurriculumbalance?Easyanswersarenotavailable,butafewalternativesseemtohavesomepromise.First,inviewofthemultitudeof(academic)knowledgeclaims,itsometimeshelpstoreducethebignumberofseparatesubjectdomainstoamorelimitednumberofbroaderlearningareas,combinedwithsharperprioritiesinaimsforlearning(focusingonbasicconceptsandskills).Second,referringtotheavalancheofsocietalclaims,moreinteractionbetweenlearninginsideandoutsidetheschoolmayreducetheburden.However,themosteffectiveresponseisprobablytobemoreselectiveinreactingtoallsortsofsocietalproblems.AsCuban(1992)phraseditclearly:schoolsshouldnotfeelobligedtoscratchthebackofsocietyeverytimesocietyhasanitch.Andthird,aboutthelearners’perspective:worldwide,manyinterestingeffortsareongoingtomakelearningmorechallengingandintrinsicallymotivatingbymovingfromtraditional,teacher-andtextbook-dominatedinstructiontowardsmoremeaningful,activity-basedandautonomouslearningapproaches.
Development strategies
Tosketchcurriculumdevelopmentasaproblematicdomainisactuallyanunderstatement.Fromasocio-politicalstance,itseemsoftenmoreappropriatetodescribeitasawarzone,fullofconflictsandbattlefieldsbetweenstakeholderswithdifferentvaluesandinterests.Problemsmanifestthemselvesinthe(sometimesspectacularandpersistent)gapsbetweentheintendedcurriculum(asexpressedinpolicyrhetoric),theimplementedcurriculum(reallifeinschoolandclassroompractices),andtheattainedcurriculum(asmanifestedinlearnerexperiencesandoutcomes).Atypicalconsequenceofthosetensionsisthatvariousfrustratedgroupsofparticipantsblameeachotherforthefailureofreformorimprovementactivities.Althoughsuchblaminggamesoftenseemratherunproductive,therearesomeseriouscriticalremarkstobemadeonmanycurriculumdevelopmentapproachesworldwide.Firstofall,manycurriculumreformeffortscanbecharacterizedbyoverlybiginnovationambitions(especiallyofpoliticians)withinunrealisticallyshorttimelinesandwithverylimitedinvestmentinpeople,especiallyteachers.Second,oftentimesthereisalackofcoherencebetweentheintendedcurriculumchangeswithothersystemcomponents(especiallyteachereducationandassessment/examinationprograms).And
an introduction to educational design research 43
lastbutnotleast,timelyandauthenticinvolvementofallrelevantstakeholdersisoftenneglected.Fromastrategicpointofview,theliteraturehasofferedusmany(technical-professional)modelsandstrategiesforcurriculumdevelopment.ThreeprominentapproachesareTyler’srational-linearapproach,Walker’sdeliberativeapproach,andEisner’sartisticapproach.Asitdoesnotfitwithinthepurposeofthischaptertoexplainthosemodelsinparticular,thereaderisreferredtoeducativetextsasfromMarshandWillis(2003).Obviously,thecontextandnatureofthecurriculumdevelopmenttaskathandwilldeterminetoalargeextentwhatkindofstrategyisindicated.Itisnoteworthythatwearebeginningtoseemoreblendedapproachesthatintegratevarioustrendsandcharacteristicsofrecentdesignanddevelopmentapproachesinthefieldofeducationandtraining(foranoverviewandaseriesofexamples:seevandenAkker,Branch,Gustafson,Nieveen&Plomp,1999).Somekeycharacteristics:• Pragmatism:Recognitionthatthereisnotasingleperspective,overarchingrationaleor
higherauthoritythatcanresolvealldilemmasforcurriculumchoicestobemade.Thepracticalcontextanditsusersareintheforefrontofcurriculumdesignandenactment.
• Prototyping:Evolutionaryprototypingofcurricularproductsandtheirsubsequentrepresentationsinpracticeisviewedasmoreproductivethanquasi-rationalandlineardevelopmentapproaches.Gradual,iterativeapproximationofcurriculardreamsintorealitiesmaypreventparalysisandfrustrations.Formativeevaluationoftentative,subsequentcurriculumversionsisessentialtosuchcurriculumimprovementapproaches.
• Communication:Acommunicative-relationalstyleisdesirableinordertoarriveattheinevitablecompromisesbetweenstakeholderswithvariousrolesandinterestsandtocreateexternalconsistencybetweenallpartiesinvolved.
• Professional development:Inordertoimprovechancesonsuccessfulimplementation,thereisatrendtowardsmoreintegrationofcurriculumchangeandprofessionallearninganddevelopmentofallindividualsandorganizationsinvolved.
Designordevelopment(al)researchisaresearchapproachthatincorporatessomeofthesecharacteristics,anditbecomesevenmorepromisingbyaddingtheelementofknowledgegrowthtoit(vandenAkker,1999).Suchresearchcanstrengthentheknowledgebaseintheformofdesignprinciplesthatofferheuristicadvicetocurriculumdevelopmentteams,when(morethanincommondevelopmentpractices)deliberateattentionispaidtotheoreticalembeddingofdesignissuesandempiricalevidenceisofferedaboutthepracticalityandeffectivenessofthecurricularinterventionsinrealusersettings.However,thereareseveralpersistentdilemmasforcurriculumdevelopmentthatcannoteasilyberesolved,letalonethroughgenericstrategies.Forexample:howtocombineaspirationsforlarge-scalecurriculumchangeandsystemaccountabilitywiththeneedfor
an introduction to educational design research44
localvariationsandownership?Thetensionbetweentheseconflictingwishescanbesomewhatreducedwhenoneavoidsthealltoocommon‘onesizefitsall’approach.Moreadaptiveandflexiblestrategieswillavoiddetailedelaborationandover-specificationofcentralcurriculumframeworks.In-stead,theyoffersubstantialoptionsandflexibilitytoschools,teachers,andlearners.Althoughstrugglesaboutprioritiesinaimsandcontentwillremaininevitable,theprincipleof‘lessismore’shouldbepursued.However,whatisincorporatedinacorecurriculumshouldbeclearlyreflectedinexaminationandassessmentapproaches.The‘enactment’perspective(teachersandlearnerstogethercreatetheirowncurriculumrealities)isincreasinglyreplacingthe‘fidelity’perspectiveonimplementation(teachersfaithfullyfollowcurricularprescriptionsfromexternalsources).Thistrendputsevenmoreemphasisonteachersaskeypeopleincurriculumchange.Bothindividualaswellasteamlearningisessential(Fullan,2001).Teachersneedtogetoutoftheircustomaryisolation.Collaborativedesignandpilotingofcurricularalternativescanbeveryproductive,especiallywhenexperiencesareexchangedandreflecteduponinastructuredcurriculumdiscourse.Interactionwithexternalfacilitatorscancontributetocarefulexplorationsofthe‘zoneofproximaldevelopment’ofteachersandtheirschools.Cross-fertilizationbetweencurriculum,teacher,andschooldevelopmentisaconditiosinequanonforeffectiveandsustainablecurriculumimprovement.Theincreasinglypopularmissionstatementsofschoolstobecomeattractiveandinspiringenvironmentsforstudentsandteacherscanonlyberealizedwhensuchintegratedscenariosarepractised.
The potential of curriculum design research
Variousmotivesforinitiatingandconductingcurriculumdesignresearchcanbementioned.Abasicmotivestemsfromtheexperiencethatmanyresearchapproaches(e.g.experiments,surveys,correlationalanalyses),withtheirfocusondescriptiveknowledge,hardlyprovideprescriptionswithusefulsolutionsforavarietyofdesignanddevelopmentproblemsineducation.Probablythegreatestchallengeforprofessionaldesignersishowtocopewiththemanifolduncertaintiesintheircomplextasksinverydynamiccontexts.Iftheydoseeksupportfromresearchtoreducethoseuncertainties,severalfrustrationsoftenarise:answersaretoonarrowtobemeaningful,toosuperficialtobeinstrumental,tooartificialtoberelevant,and,ontopofthat,theyusuallycometoolatetobeofanyuse.Curriculumdesignersdoappreciatemoreadequateinformationtocreateasolidgroundfortheirchoicesandmoretimelyfeedbacktoimprovetheirproducts.Moreover,theprofessionalcommunityofdevelopersasawholewouldbehelpedbyagrowingbodyofknowledgeoftheoreticallyunderpinnedandempiricallytesteddesignprinciplesandmethods.
an introduction to educational design research 45
Anotherreasonforcurriculumdesignresearchstemsfromthehighlyambitiousandcomplexnatureofmanycurriculumreformpoliciesineducationworldwide.Thesereformendeavorsusuallyaffectmanysystemcomponents,areoftenmulti-layered,includingbothlarge-scalepoliciesandsmall-scalerealization,andareverycomprehensiveintermsoffactorsincludedandpeopleinvolved.Thoseradical‘revolutions’,ifpromisingatall,cannotberealizedonthedrawingtable.Thescopeofdiverseneedsisoftenverywide,theproblemstobeaddressedareusuallyill-specified,theeffectivenessofproposedinterventionsismostlyunknownbeforehand,andtheeventualsuccessishighlydependentonimplementationprocessesinabroadvarietyofcontexts.Therefore,suchcurriculumreformeffortswouldprofitfrommoreevolutionary(interactive,cyclic,spiral)approaches,withintegratedresearchactivitiestofeedtheprocess(bothforwardandbackward).Suchanapproachwouldprovidemoreopportunitiesfor‘successiveapproximation’oftheidealsandformorestrategiclearningingeneral.Inconclusion:curriculumdesignresearchseemsawiseandproductiveapproachforcurriculumdevelopment.
Features of curriculum design research
Curriculumdesignresearchisofteninitiatedforcomplex,innovativetasksforwhichonlyveryfewvalidatedprinciplesareavailabletostructureandsupportthedesignanddevelopmentactivities.Sinceinthosesituationstheimageandimpactoftheinterventiontobedevelopedisoftenstillunclear,theresearchfocusesonrealizinglimitedbutpromisingexamplesofthoseinterventions.Theaimisnottoelaborateandimplementcompleteinterventions,buttocometo(successive)prototypesthatincreasinglymeettheinnovativeaspirationsandrequirements.Theprocessisoftencyclicorspiral:analysis,design,evaluationandrevisionactivitiesareiterateduntilasatisfyingbalancebetweenidealsandrealizationhasbeenachieved.
Towhatextentdothesedesignresearchactivitiesdifferfromwhatistypicalfordesignanddevelopmentapproachesinprofessionalpractices?Whataretheimplicationsoftheaccountabilityofresearcherstothe‘scientificforum’?Attheriskofexaggeratingthedifferences,letusoutlinesomeofthem,basedonwhatisknownaboutroutinizedstandard-patternsincurriculumdevelopmentpractices.Ofcourse,alotofactivitiesaremoreorlesscommonforbothapproaches,sothefocuswillbeonthoseadditionalelementsthataremoreprominentindesignresearchthanincommondesignanddevelopmentpractices.
(1)PreliminaryinvestigationAmoreintensiveandsystematicpreliminaryinvestigationofcurriculumtasks,problems,andcontextismade,includingsearchingformoreaccurateandexplicitconnectionsofthat
an introduction to educational design research46
analysiswithstate-of-the-artknowledgefromliterature.Sometypicalactivitiesinclude:literaturereview;consultationofexperts;analysisofavailablepromisingexamplesforrelatedpurposes;casestudiesofcurrentpracticestospecifyandbetterunderstandneedsandproblemsinintendedusercontexts.
(2) TheoreticalembeddingMoresystematiceffortsaremadetoapplystate-of-the-artknowledgeinarticulatingthetheoreticalrationaleforcurriculumdesignchoices.Moreover,explicitfeedbacktoassertionsinthedesignrationaleaboutessentialcharacteristicsoftheintervention(substantivedesignprinciples)ismadeafterempiricaltestingofitsquality.Thistheoreticalarticulationcanincreasethe‘transparency’and‘plausibility’oftherationale.Becauseoftheirspecificfocus,thesetheoreticalnotionsareusuallyreferredtoas‘mini’-or‘local’theories,althoughsometimesconnectionscanalsobemadeto‘middle-range’theorieswithasomewhatbroaderscope.
(3) EmpiricaltestingClearempiricalevidenceisdeliveredaboutthepracticalityandeffectivenessofthecurriculumfortheintendedtargetgroupinrealusersettings.Inviewofthewidevariationofpossibleinterventionsandcontexts,abroadrangeof(direct/indirect;intermediate/ultimate)indicatorsfor‘success’shouldbeconsidered.
(4) Documentation,analysisandreflectiononprocessandoutcomesMuchattentionispaidtosystematicdocumentation,analysisandreflectionontheentiredesign,development,evaluationandimplementationprocessandonitsoutcomesinordertocontributetotheexpansionandspecificationofthemethodologyofcurriculumdesignanddevelopment.
Morethanmostotherresearchapproaches,designresearchaimsatmakingbothpracticalandscientificcontributions.Inthesearchforinnovative‘solutions’forcurriculumproblems,interactionwithpractitioners(invariousprofessionalroles:teachers,policymakers,developers,andthelike)isessential.Theultimateaimisnottotestwhethertheory,whenappliedtopractice,isagoodpredictorofevents.Theinterrelationbetweentheoryandpracticeismorecomplexanddynamic:isitpossibletocreateapracticalandeffectivecurriculumforanexistingproblemorintendedchangeintherealworld?Theinnovativechallengeisusuallyquitesubstantial,otherwisetheresearchwouldnotbeinitiatedatall.Interactionwithpractitionersisneededtograduallyclarifyboththeproblematstakeandthecharacteristicsofitspotentialsolution.Aniterativeprocessof‘successiveapproximation’or‘evolutionaryprototyping’ofthe‘ideal’interventionisdesirable.Directapplicationoftheoryisnotsufficienttosolvethosecomplicatedproblems.Onemightstate
an introduction to educational design research 47
thatamore‘constructivist’developmentapproachispreferable:researchersandpractitionerscooperativelyconstructworkableinterventionsandarticulateprinciplesthatunderpintheeffectsofthoseinterventions.Anotherreasonforcooperationisthatwithoutinvolvementofpractitionersitisimpossibletogainclearinsightinpotentialcurriculumimplementationproblemsandtogeneratemeasurestoreducethoseproblems.Newinterventions,howeverimaginativetheirdesign,requirecontinuousanticipationatimplementationissues.Notonlyfor‘social’reasons(tobuildcommitmentandownershipofusers)butalsofor‘technical’benefits:toimprovetheirfitnessforsurvivalinreallifecontexts.Therefore,rigoroustestingofpracticalityisaconditiosinequanonindesignresearch.
Emphasis on formative evaluation
Ashasbecomeclearintheprevioussections,formativeevaluationholdsaprominentplaceincurriculumdesignresearch.Themainreasonforthiscentralroleisthatformativeevaluationprovidestheinformationthatfeedsthecycliclearningprocessofcurriculumdevelopersduringthesubsequentloopsofadesignanddevelopmenttrajectory.Itismostusefulwhenfullyintegratedinacycleofanalysis,design,evaluation,revision,etcetera,andwhencontributingtoimprovementofthecurriculum.Thebasiccontributionofformativeevaluationistoqualityimprovementofthecurriculumunderdevelopment.Quality,however,isanabstractconceptthatrequiresspecification.Duringdevelopmentprocesses,theemphasisincriteriaforqualityusuallyshiftsfromrelevance,toconsistency,topracticality,toeffectiveness2.Relevancereferstotheextentthattheintendedcurriculumisperceivedtobearelevantimprovementtopractice,asseenfromthevariedperspectivesofpolicymakers,practitionersandresearchers.Consistencyreferstotheextentthatthedesignofthecurriculumisbasedonstate-of-the-artknowledgeandthatthevariouscomponentsoftheinterventionareconsistentlylinkedtoeachother(cf.thecurricularspiderweb).Practicalityreferstotheextentthatusers(andotherexperts)considertheinterventionasclear,usableandcost-effectivein‘normal’conditions.Effectivenessreferstotheextentthattheexperiencesandoutcomeswiththeinterventionareconsistentwiththeintendedaims.
Themethodsandtechniquesforevaluationwillusuallybeattunedtothatshiftincriteria.Forexample,validitycanadequatelybeevaluatedthroughexpertappraisal,practicalityviamicro-evaluationsandtry-outs,andeffectivenessinfieldtests.Inlaterstagesofformativeevaluation,methodsofdatacollectionwillusuallybelessintensivebutwithanincreasingnumberofrespondents(e.g.achievementtestformanystudentscomparedtoin-depthinterviewwithafewexperts).
2) Seeforthesecriteriaalsochapters1and5.
an introduction to educational design research48
Formativeevaluationwithindevelopmentresearchshouldnotonlyconcentrateonlocatingshortcomingsoftheinterventioninitscurrent(draft)version,butespeciallygeneratesuggestionsonhowtoimprovethoseweakpoints.Richnessofinformation,notablysalienceandmeaningfulnessofsuggestionsinhowtomakeaninterventionstronger,isthereforemoreproductivethanstandardizationofmethodstocollectandanalyzedata.Also,efficiencyofproceduresiscrucial.Thelowerthecostsintimeandenergyfordatacollection,processing,analysisandcommunicationwillbe,thebiggerthechancesonactualuseandimpactonthedevelopmentprocess.Forexample,samplesofrespondentsandsituationsfordatacollectionwillusuallyberelativelysmallandpurposivecomparedtosamplingproceduresforotherresearchpurposes.Theaddedvalueofgetting‘productive’informationfrommoresourcestendstodecrease,becausetheopportunitiesfor‘rich’datacollectionmethods(suchasinterviewsandobservations)arelimitedwithbignumbers.Toavoidanoverdoseofuncertaintyindatainterpretation,oftentriangulation(ofmethods,instruments,sources,andsites)isapplied.Theseargumentsespeciallyholdtrueforearlystagesofformativeevaluation,whentheinterventionisstillpoorlycrystallized.
Generalization of curriculum design research findings
Thepracticallymostrelevantoutcomeofcurriculumdesignresearchisitscontributiontowardsoptimizationofthecurricularproductanditsactualuse,leadingtobetterinstructionalprocessesandlearningresults.However,amajorcontributiontoknowledgetobegainedfromdesignresearchisintheformof(bothsubstantiveandmethodological)‘designprinciples’tosupportdevelopersintheirtask.Thoseprinciplesareusuallyheuristicstatementsofaformatsuchas:“IfyouwanttodesigncurriculumX[forthepurpose/functionYincontextZ],thenyouarebestadvisedtogivethatcurriculumthecharacteristicsA,B,andC[substantiveemphasis],andtodothatviaproceduresK,L,andM[proceduralemphasis],becauseoftheoreticalandempiricalargumentsP,Q,andR.”Obviouslythoseprinciplescannotguaranteesuccess,buttheyareintendedtoselectandapplythemostappropriate(substantiveandprocedural)knowledgeforspecificdesignanddevelopmenttasks.Itisnotuncommonindesignresearchthatsuchknowledge,especiallythesubstantiveknowledgeaboutessentialcurriculumcharacteristics,canpartlybeextractedfromaresultingprototypeitself.Thatisoneofthereasonsthatmakeitsoprofitabletosearchforandcarefullyanalyzealreadyavailablecurriculatogenerateideasfornewdesigntasks.However,thevalueofthatknowledgewillstronglyincreasewhenjustifiedbytheoreticalarguments,well-articulatedinprovidingdirections,andconvincinglybacked-upwithempiricalevidenceabouttheimpactofthoseprinciples.Moreover,thoseheuristicprincipleswillbeadditionallypowerfuliftheyhavebeenvalidatedinsuccessfuldesignofmoreinterventionsinmorecontexts.Chancesforsuchknowledgegrowthwillincrease
an introduction to educational design research 49
whendesignresearchisconductedintheframeworkofresearchprograms,becauseprojectscanthenbuildupononeanother.Sincedatacollectionindesignresearchisoftenlimitedtosmall(andpurposive)samples,effortstogeneralizefindingscannotbebasedonstatisticaltechniques,focusingongeneralizationsfromsampletopopulation.Insteadonehastoinvestin‘analytical’formsofgeneralization(cf.Yin,2003):readers/usersneedtobesupportedtomaketheirownattemptstoexplorethepotentialtransferoftheresearchfindingstotheoreticalpropositionsinrelationtotheirowncontext.Reportsondesignresearchcanfacilitatethattaskofanalogyreasoningbyacleartheoreticalarticulationofthedesignprinciplesappliedandbyacarefuldescriptionofboththeevaluationproceduresaswellastheimplementationcontext.Especiallya‘thick’descriptionoftheprocess-in-contextmayincreasethe‘ecological’validityofthefindings,sothatotherscanestimateinwhatrespectsandtowhatextenttransferfromthereportedsituationtotheirownispossible.Anotheroptionthatmaystimulateexplorationofpossibilitiesfor(virtual)generalizationistoorganizeinteractivemeetingswithexpertsfromrelatedcontextstodiscusstheplausibilityoftheresearchfindingsandrecommendationsforrelatedtasksandcontexts.Lastbutnotleast,designresearchmayofferdraftsofvariousrelevantcurriculumversions(withprovenconsistencyandpracticality)thatcanbecomparedinmorequantitative,large-scalestudies.
References
Cuban,L.(1992).Curriculumstabilityandchange.InP.Jackson(Ed.),Handbook of research on curriculum(pp.216-247).NewYork:Macmillan.
Goodlad,J.(1994).Curriculumasafieldofstudy.InT.Husén,&T.Postlethwaite(Eds.),The international encyclopedia of education(pp.1262-1276).Oxford:PergamonPress.
Fullan,M.(2001).The new meaning of educational change.NewYork:TeachersCollegePress.Marsh,C.,&Willis,P.(2003).Curriculum: Alternative approaches, ongoing issues(third
edition).UpperSaddleRiver,NJ:Merrill/PrenticeHall.Pinar,W.,Reynolds,W.,Slattery,P.,&Taubman,P.(1995).Understanding curriculum. An
introduction to the study of historical and contemporary curriculum discourses.NewYork:PeterLang.
Taba,H.(1962).Curriculum development: Theory and practice.NewYork:Harcourt,Brace&World.
vandenAkker,J.(1999).Principlesandmethodsofdevelopmentresearch.InJ.vandenAkkeretal.(Eds.),Design approaches and tools in education and training(pp.1-14).Dordrecht:KluwerAcademicPublishers.
an introduction to educational design research50
vandenAkker,J.(2003).Curriculumperspectives:Anintroduction.InJ.vandenAkker,W.Kuiper&U.Hameyer(Eds.),Curriculum landscapes and trends(pp.1-10).Dordrecht:KluwerAcademicPublishers.
vandenAkker,J.,Branch,R.,Gustafson,K.,Nieveen,N.&Plomp,T.(Eds.)(1999).Design approaches and tools in education and training.Dordrecht:KluwerAcademicPublishers.
vandenAkker,J.,Gravemeijer,K.,McKenney,S.,&Nieveen,N.(Eds.)(2006).Educational design research.London:Routledge.
Walker,D.(2003).Fundamentals of curriculum: Passion and professionalism.Mahwah,NJ:LawrenceErlbaumAssociates.
Yin,R.K.(2003).Case Study Research: Design and Methods.London:Sage.
an introduction to educational design research 51
an introduction to educational design research 53
3. TheIntegrativeLearningDesign Framework:AnIllustratedExample fromtheDomainofInstructional Technology Brenda Bannan
Articulatingacleardefinitionandprocessofdesignresearchisacurrentandprominenttopicamongeducationalresearchers(Kelly,Lesh&Baek,2008;vandenAkker,Gravemeijer,McKennyandNieveen,2006).Designresearchstudiesinvolvecomplexinteractionsandfeedbackcyclesthatcansignificantlyblurtherolesofresearchers,teachers,curriculumdevelopers,instructionaldesignersandassessmentexperts(Kelly,Lesh,Baek&Bannan-Ritland2008).Aseducationalresearchersstruggletoclarifythisresearchmethod,theycontinuetoraisesignificantquestionssuchashowisdesignresearchdifferentfromtheprocessofdesign?Whatareappropriatemethodsandprocessesthatcanbeusedindesignresearch?Howdowesystematicallycreate,testanddisseminatedesignorteachinginterventionsthatwillhavemaximumimpactonpracticecapitalizingondesignresearch?Howdowegenerateboththeoreticalandpracticalknowledgerelatedtocomplexeducationalsettings?Kelly(2006)andothers(Fishman,Marx,Blumenfeld,Krajcik&Soloway,2004;Zaritsky,Kelly,Flowers,Rogers&O’Neill,2003;Rogers2003;Collins,1999;Design-basedResearchCollective,2003)advocatethattheseemergingmethodscallforthearticulationofnewprocessesandcriteriaincludingfactorssuchastheusefulnessandusabilityofknowledge,itsshareability,andmarketability,howwellitdisseminatesandtheextenttowhichitpositivelyimpactspractice.Sabelli(personalcommunication,May16,2002)citesaneedfororganizationalstructureandprotocolforthediffusionofresearchintopracticeandstatesthateducationalresearchsituationsareextremelycomplexsystemsthatcanbenefitfromintegratedsystemresearchstrategies.Thereisaneedforcomprehensivemodelstoguidedesignresearchaddressingtheprocessofdesigning,developingandassessingtheimpactofaneducationalinnovation.Inthischapter,Ipresentanintegrationofexistingdesignandresearchprocessesofferingaguidingframeworkthatgoesbeyondtheindividualdomainsofsocialscience,behavioralscienceandcommunicationtheoryandattemptstointegratethesystematicprocessesoftherelatedfieldsofinstructionaldesign,softwareengineering,productdesign,hencethenameIntegrativeLearningDesignFramework(ILDF).Buildingontheintegrationofprocessesfrommultiplefieldssuchasinstructionaldesign,objectorientedsoftwaredevelopment,productdevelopmentanddiffusionofinnovationsandeducationalresearch,theILDFpresenta“meta-methodological”viewthatattemptstointegratethebestofdesign,researchanddiffusionofeducationalinnovations.Thisframeworkconsistingoffourphases(seeFigure1)challengesresearcherstoprovideimprovedarticulationofdesignresearchprocessesbyphaseandtoconsidertheentirescopeofresearchfrominitialconceptualizationtodiffusionandadoption.
an introduction to educational design research54
Figure 1: Questions and Methods for Design Research by ILDF Phase
Web-enabled Proto-D
iffusion
ConsequensesD
iffusion Adoption
Adaptation
Publish Results
Evaluate Results
Implem
ent.Theory/System
Refinem.
Formative
TestingD
etailed D
esignArticulated Prototype
Research/System
D
esign
AudienceCharacteriza-
tion
TheoryD
evelop.Survey
LiteratureN
eedsAnalysis
Informed Exploration
Enactment
Evaluation: Local Impact
Evaluation: Broader Impact
Questions:
- What are identified gaps/problem
s in theory, practice and/or the m
arketplace?- W
hat information can be gleaned from
existing data or research?- H
ow can w
e characterize the problem or
leaner need?- W
hat are the systemic social, cultural, and
organizational influences or constraints on design?- W
hat are characteristics of the audience?
Methods:
- Benchmarking
- Performance/needs analysis
- Interviews
- Survey of Experts- Focus G
roups- O
bservation/Role Modeling
- Case Studies
Questions:
- What are the learning targets for
innovation?- W
hat design principles or strategies may
be applicable?- H
ow to identify and operationalize
cognitive and performance processes
in design?- To w
hat extent does the design embody
the theoretical model?
Methods:
- Task Analysis- Contextual Analysis- D
esigner Logs- Expert Review- Audience Review
Questions:
- Is the enacted design usable, valid and
relevant?- Is the design instance accessible and efficient in delivering instruction or supporting learning?- W
hat is the local impact or effectiveness
of the design instance?- H
ow effective is the design solution in
achieving learning targets at its highest fideliy in full context?
Methods:
- Usability Testing
- Expert Review- O
bservation or Video records- Interview
s- Form
ative Evaluation- Pre-post Com
parative Studies- Q
uasi-experimental studies
Questions:
-What factors influence diffusion, adoption
and adaption of innovation?- W
hat are the pragmatic dem
ands of the learning environm
ent that influences adoption of design?- W
hat policies and cultures shape participants use of innovation?
Methods:
- Analysis of computer log files
- Multi-site Interview
s, Surveys and O
bservations- D
ata mining
- Correlational studies- Q
uasi-experimental studies
ILD
Guiding
Questions
for Research
ApplicableResearchM
ethods
Adaptation
an introduction to educational design research 55
ThefourphasesofInformedExploration,Enactment,LocalEvaluationandBroadEvaluationpresentedintheILDFencompassaprocessmodelforconductingdesignresearchbasedonseveralyearsofattemptstoincorporateprogressivelymorerigorous,research-basedcycleswithinatechnology-basedinstructionaldesigneffort.Thistypeofeffortisdifferentthantraditionalinstructionaldesignastheiterativecyclesareessentiallymicro-cyclesofresearch(morecomprehensivequalitativeandquantitativeresearcheffortsthanformativeevaluationcycles)conductedtolearnmorethanhowtoimprovethetechnologysystem,althoughthestudiesmayalsoresultinthatoutcome.ThereferencetolearningwithintheILDFistoplaceemphasisonthelearningthatcanresultinthecontextandactivityofdesign.Forexample,asresearchersorinstructionaldesignerswemaygenerateinformationabouttheteachingandlearningprocess,participants,context,andculturethatisoftennotattendedto,discardedandcapturedinarigorousmannerforotherstolearnfromandcapitalizeon.Whetherourdesignactivityinvolvesclassroom-basedinterventions,technologyorsomecombinationofboth,theinterconnecteddesignresearchcyclescangenerateknowledgeaboutdesignprinciplesbutalsoproviderichinformationonaspectsoflearning,cognition,expertandnoviceperspectives,aswellasstakeholderpositionstodirectdesignanddesigndecision-making.Thecoreissueathandisthattherich,complex,designprocessmayoffermultipleopportunitiestogenerateresearch-basedknowledgehowever,muchofitislostandnotdocumentedinthecreativedesignprocess.Ourchallenge,asdesignresearchers,istotrytosystematicallygather,analyze,reportandcodifythisinformationinarigorousmannerthatstrivestowardsometypeoflogical,argumentativegrammarworthyofstringentresearchprocesses(Kelly,2006).
Connectedcyclesofresearchcyclesanddesignprocessesresultinimproveddecision-makingbasedondata-drivenresultsfordesign,developmentandresearchpurposes.Thoughclearlyinterventionistandprimarilyformativeinnature,theILDFprocessstandsapartfromtraditionalinstructionaldesignandresearchefforts.Throughoutthemultiplephasesandcyclesofintegratedresearchanddesignprocessesvaluableknowledgeinthecontextofuseisgenerated.Weneedtominewhatislearnedaboutimportantfactorsrelatedtolearning,context,culture,andtechnologywithinthedesignprocess(notseparatefromitinacontrolledsettingasevidencedintraditionalresearch).Ifdesignresearcherscanarticulateanintegrativeresearchanddesignprocess,itmayhavethepotentialtosignificantlyimproveourunderstandingofteaching,learningandtrainingin-situ.Themultiplemacroandmicro-cyclesofdatacollection,analysisandmostimportantly,results-drivendesigndecision-makingiswhatsetsdesignresearchapartfromtraditionalformativeevaluationininstructionaldesignwhichisoftenconductedinaverylimitedmannerorasinglecycleofdata-gatheringandanalysis.
an introduction to educational design research56
Tessmer(1993)referstoformativeevaluationasa“judgmentofthestrengthsandweakenessesofinstructioninitsdevelopingstages,forpurposesofrevisingtheinstructiontoimproveitseffectivenessandappeal(p.11)”.Althoughmultiplemethodsmaybeusedincludingexpertreview,one-to-oneevaluation,smallgroupandfieldtesting,formativeevaluationcyclesintraditionalinstructionalsystemsdesignmaynotalwaysemployresearchmethodsthatarespecifictoparticularphasesofanintegrativeandconnecteddesignresearchcycle.Formativeevaluation,despiteitsmostrigorousandcomprehensiveapplicationdoesnotprogressivelygenerateknowledgeaboutcognition,contextandcultureofusebutprovidesalimitedfocusonaparticulartechnologysystemofinstructionandjudgesitseffectiveness,appealandefficiency.Incontrast,designresearchcyclesarebasedonathorough,systematicprocessintegratedmultipledesignandresearchprocessestoprogressivelyimproveunderstandingaboutlearners,learning,context,orcultureaswellasiterativelyimproveanintervention.Therefore,formativeevaluationmethodsaresubsumedasoneselectedmethodinwhatcouldbedescribedasa“meta-methodological”orinvolvingmultipleresearchmethodsacrossthedesignresearchprocess.Whatiscriticalindesignresearchisthetheoreticalyieldoftheefforttobeviewedasimportantastheimprovementoftheintervention(seePlompchapter1).
Designresearchcyclesaredynamicandintegratemultipleexploratory,constructiveand/orempiricalresearchmethodsaswellasmultipledesign/developmenttechniques(seeFigure1).Exploratoryresearchmethodsstructureandidentifynewproblemssuchasfeasibilitytesting,benchmarkingandqualitativeresearchapproaches.Constructiveresearchdevelopssolutionstoproblemsandmayincludetestingofaconstructortheoryagainstapredefinedcriteriaandmay,forexample,includeformativeevaluationtestingofaninstructionaltechnologysystem.Incontrast,empiricalresearchteststhefeasibilityofasolutionusingempiricalordirectorindirectobservationorevidenceinthetraditionofthescientificmethod.Designresearchmayemployallthreeformsofresearchmethodsaswellasincorporateformativeevaluationmethodsatdifferentphasesintheprocess.However,traditional,formativeevaluationperspectiveswhileofferingvaluableiterativeprocesses,donotinisolation,addressthecomplexityinherentineducationalpractice.Mosteducationalresearchprojectsadvocateonlyonecycleofqualitativeorquantitativeempiricaltestingatafixedpointintimeforagiveninstructionalinterventionforthesolepurposeofgeneratingknowledge.Incontrast,designresearchattemptstoprogressivelyanddynamicallygenerate(exploratoryresearch),improve(constructiveresearch)andlearnabout(empiricalresearch)aparticularphenomenonfrominterconnectedresearchanddesigncycles.
Inresponsetothischallenge,theILDFmodelattemptstoprovideacomprehensiveyetdynamicandflexibleguidingframeworkthatpositionsmultiple,microandmacrodesign
an introduction to educational design research 57
researchcyclesasprimarilysocially-constructed,contextualizedprocessofproducingthemosteducationallyeffectiveproductthathasthebestchancetobeusedintheclassroomwhilealsogeneratingknowledgeaboutteachingandlearningwithintheactivityofdesign.Themodelorframeworkattemptstomovepastisolated,individualeffortsofeducationalresearchbyclearlyarticulatingalogically-orderedstructuralframethatconsidersthefullspectrumofresearchmethodologyinadvancingtowardsystemicimpactineducationandmaybeappliedinavarietyofcontexts.Collins(1990;1993)advocatesforasimilarovert,systematicmethodologyforconductingdesignexperimentsandstates:
“When designing a learning environment, whether computer based or not, there are a multitude of design decisions that must be made. Many of these
design decisions are made unconsciously without any articulated view of the issues being addressed or the tradeoffs involved. It would be better if these
design decisions were consciously considered, rather than unconsciously made (1993, p.1).”
TheILDFprocesspresentsonesteptowardasystematicframeworkforthearticulationanddocumentationofcommonphasesandcomplementarystagesbasedonmultipledesignandresearchprocessespromotingmoreconsciousdesignresearch(Collins,1990;1999).Althoughtherearethousandsofdecisionsmadeinadesignresearchcontext,themajorconjectures,learningtargets,taskanalysis,designprinciplesandevaluationorresearchdecision-makingresultingfromexploratory,constructiveand/orempiricalresearchcyclesmaybeuncoveredbyexaminingarichcasestudyaspresentedhereentitledtheLiteracyAccessOnline(LAO)project.TheILDFispresentedhereasastartingpointforresearcherstoconsideraswiththegoalofelicitingquestions,suggestions,limitationsandcriteriathatmayneedtobeconsideredasresearchersstrugglewiththeimplicationsofthisemergingformofeducationalresearch.Inthischapter,IbrieflydescribetheprogressionoftheLAOdesign-basedresearchstudythatencompassedfouryearsofeffortandillustratestheapplicationoftheILDF.TheLAOcasestudyexampleisdescribedaccordingtobroadphasesincluding1)theinformedexplorationphase;2)theenactmentphase;3)thelocalimpactphase;and4)thebroadimpactphaseaswellasthemultiplepotentialappliedandempiricalresearchprocessesthatalignwitheachphase(seeFigure1).ItishopedthattheLAOexamplewillprovideenoughdetailtopotentiallyimproveunderstandingofconductingcyclesofdesignresearchrelatedtoatechnology-basededucationalintervention.
an introduction to educational design research58
LiteracyAccessOnline - An integrative learning design study
TheLiteracyAccessOnline(LAO)project1providesanexampleofanintegrativelearningdesignstudybasedontheILDFspecificallyillustratingtheintersectionandsystematicexpressionofmultipledesignandresearchmethods.LiteracyAccessOnlineisanefforttoutilizeWeb-basedtechnologytoprovidesupportforteachers,tutors,andparents(literacyfacilitators)inaddressingliteracygoalsforallchildrenwithaparticularfocusonthosewithdisabilities.Afterfouryearsofdesignresearchanddevelopment,LAO(http://literacyaccessonline.com)nowprovidesatechnology-basedlearningenvironmentthatpromotestheuseofspecificliteracystrategiesfortheimprovementoftutoringandreadingperformanceasthechildandliteracyfacilitatorcollaborativelyengageintheprocessofreadingonline.
TheInformedExplorationPhaseTheexploratoryresearchobjectivesoftheLAOintegrativelearningdesignstudyweretwo-fold;1) toinvestigatethenatureandeffectivenessofaconsistenttechnology-based,
collaborativeliteracyenvironmentaswellas;2) togenerateknowledgeabouthowliteracyfacilitatorsandchildrenunderstandand
employreadingsupportstrategies.Theseobjectiveswereoriginallyconceivedasresearch/evaluationquestionsandevolvedfromanextended,progressiveinvestigationintotheprovisionofliteracysupportforfacilitatorsandchildren.This“meta-methodological”designresearchprocessconsistingofmultipleresearchmethods(e.g.survey,focusgroups,interviews,expertreviews,etc)resultedinclearlyarticulatedlearningtargets,taskanalysesoflearningobjectives,theoreticalmodelembeddedinatechnologysystemdesignandcongruentresearch/evaluationquestionsthatdrovemorerigorousqualitativetestingoftheinterventionwhoseresultscontributedtotheoryofliteracysupportforchildrenwithdisabilitiesfurtherelaboratedinsectionsbelow.Tobeginexploratoryresearchcycles,initialexplorationsintotargetaudienceandstakeholderperceptions,relatedproductsandliteratureanddocumentationofthecomplexnatureofsupportingliteracyrevealedmanyplausiblepathsfordesignresearch.TheinterdisciplinaryresearchteaminvolvedintheLAOprojectwerechargedwithdeterminingtheresearchdirectionandconsistedofeducationalresearchers,teachers,graduatestudents,contentexpertsinliteracy,specialeducationandassistivetechnologyaswellasparentsinvolvedinanadvocacygroupforchildrenwithdisabilities.Thebroaddesignresearch
1) TheLiteracyAccessOnline(LAO)projectissupportedbytheOfficeofSpecialEducationProgramsintheDepartmentofEducationSteppingstonesofTechnologyInnovationforStudentswithDisabilitiesGrantCFDA84.327A
an introduction to educational design research 59
focusevolvedfromtheteam’sperceivedlackofsupportforchildrenwhowerestrugglingwiththeliteracyprocess,basedondirectobservationsofthisprobleminbothclassroomandhomeenvironmentswhichthenmanifesteditselfintoseveralindividualbutconnectedresearchstudies.Forexample,weconductedmultipleinterviewswithparentswithchildrenwithdisabilitieswhowerestrugglingwiththereadingprocess.Wealsoinvitedseveralparentstoparticipateonourdesignresearchteam.Theteams’analysisoftheinterviewdataaswellasthedesignresearchteamdiscussionsrevealedourinitialapproach.Wehadinitiallydecidedtodesignatutorial-basedinterventiononlyforthechild’sbenefit,however,acommentinateammeetingdramaticallychangedourdesigndirection.InlinewithCollins’notionofconsciousconsiderationsofdesignasdemonstratingcoreunderlyingdesigndecision-making,oneparentmember/stakeholderontheteamstatedthatsheprimarilyreadinconjunctionwith(notto)hersonandwantedtodosoonlinebutwithadditionalsupportofhigherlevelreadingstrategies.Basedonthatinputandfollow-upmicro-cyclesofinterviewdatacollectionandanalysistoconfirmationtheviabilityofthisdesignapproachwithparents,weconductedaseriesofinterviewsandsurveystodeterminethefeasibilityofthisdesigndirection.Theresultsofourinvestigationanddiscussionsevolvedintoanonlinecollaborativeperformancesupportsystemtosupportbothliteracyfacilitatorsandtheirchildrenintheliteracyprocessasthedetermineddesigndirection.Thiswasadesigndecisionbasedondataanalyzedfrommultipleinterviewsandsurveyprocedures.AlignedwithConfreyandLachance’s(2000)notionofdrawingkeyinferencesfromdissatisfactionwithcurrenteducationalpracticesanddirectexperienceswithchildren,initialtheoreticalconjecturesweredevelopedbasedontheanalyzeddatathatadvocatedforreading,writingandassistivetechnologysupportforchildrenwithorwithoutdisabilitiestoincreasetheirengagementandperformanceinliteracy.
Whiletheseinitialtheoreticalconjecturesprovidedacentralpremiseandbroaddirectionfordesignresearch,moreinformationwasneededtorefinetheseconjecturesresultinginacomprehensiveneedsanalysisandliteraturereviewthatprovidedafirmandcomplementarytheoreticalfoundationfortheintendeddesign.Extensiveexplorationintoappropriateliteracystrategies,tutorialprogramsandprocesses,surveysofexperts,teachersandparentsaswellasqualitativeobservationofchildrenandfacilitatorsengagedinaliteracyexperienceallinformedthisphaseoftheresearch.Thisprovidednotonlywell-defineddesigndirectionsbutalsoaddedtotheresearchliteratureregardingchildrenwithdisabilitiesandtheirparentsunderstandingofassistivetechnologyandliteracylearning(seeJeffs,Behrman&Bannan-Ritland,2006).Manypotentialdesignresearchdirectionswereconsideredbasedontheinitialconjectures,however,datadrawnfromconductedinterviews,directexperiencewithpotentialresearchparticipantsandliteraturereviewconvergedandpointedtheteaminaparticulardirection.
an introduction to educational design research60
Aprominentthemethatemergedacrossinitialinterviews,surveysandobservationswithexperts,parents,teachersandchildrenrevealedthatliteracyfacilitatorshadacrucialroleinprovidingsupportforchildrenstrugglingtogainliteracyskillsandthequestionremainedhowtobestsupportthisrole.Thesefindingsandrelatedliteratureprovidedinsightforinformedtheoryandimprovedconjecturesbasedontheaforementionedqualitativeinterviewsandliteraturereviews.Resultsfromdatacollectionandliteraturereviewmethodsintheinformedexplorationphaseindicatedthat:1) childrencan,butoftendonotuseeffectivemetacognitivereadingstrategies;2) explicitlyteachingthesestrategiescangreatlyenhancechildren’scomprehensionof
text;3) teachers(aswellasotherliteracyfacilitators)needtobetrainedinhowtoprovide
cognitivestructurefortheirstudentssothatchildrencanlearntoguidetheirowngenerativeprocessesinreading;and
4) one-to-onetutoringisoneofthemosteffectiveformsofinstructionforimprovingreadingachievementbutincreasedsuccessoftendependsupontheskillofthetutororfacilitatorandtheestablishmentofconsistentrolesandexpectations(Wittrock,1998;Wasik,1998).
Thisexplorationintotheliteratureandperspectivesofthoseinvolvedintheseissuesgreatlyrefinedourinitialtheoreticalconjecturesandresultedinadramaticchangeofourintendeddesigndirectionforthisresearchfromadidactic,tutorial,child-focusedinterventiontoacollaborative,story-basedreadingexperienceprovidingembeddedmetacognitivestrategysupportforboththeliteracyfacilitatorandthechild’suse.Therationaleforthisresearchdirectionwasdocumentedinacomprehensiveneedsanalysisthatdetailedthedatacollection,conclusionsandrelatedliteraturereview.
ThenextstageofourdesignresearchinvolvedtheanalysisanddescriptionoftherangeoflearnersandfacilitatorsthatwouldpotentiallyusetheLAOsystem.Directexperiencewith4th-8thgradechildrenwithorwithoutdisabilities,teachers,tutors,andparentsprovideddatathatcharacterizedouraudience.Thesedescriptionsweredepictedasrolemodels(Constantine&Lockwood,1999)orpersonas(Cooper,1999)thatcomprisedabstractcompositeprofilesofaudiencecharacteristicsgleanedfromactualinterviewsandobservationsandprovidedafocalpointfordesign.RolemodelsorpersonasaresimilartoGraueandWalsh’s(1998)qualitativevignettesthatstrivetocapturethesubstanceofasetting,personoreventtocommunicateacentralthemeofqualitativedata,basedonmultipledirectobservationsandareemployedhereasalsoafocalpointfordesign.Exploringthenatureoftheidentifiededucationalproblem,relatedproductsandliteratureaswellascreatingandrefiningtheoreticalconjecturesanddescriptionsoftheaudienceprovidedaninformedperspectiveforgroundeddesignofalearningenvironmentbasedon
an introduction to educational design research 61
articulatedtheory.Theseactivitiesresultedinspecificresearchartifactsincludinganeedsanalysisthatcontainedanextensiveliteraturereview,anarticulatedandcongruentdesignandresearchdirectionanddetailedaudienceanalysisbasedonqualitativeandquantitativedata.ThesedocumentswerehousedonaprojectWebsitethatprovidedacommunicationmechanismbetweenteammembersaswellasanarchiveofshareabledesignresearchprocesses,productsandevidentiarydata.
TheEnactmentPhaseTheembodimentoftheresultsofourinformedexplorationandtheoriesaboutprovidingliteracysupportforchildrenandliteracyfacilitatorsinausablelearningenvironmentwerecollaborativelyconstructedacrossseveralstagesandconstructiveresearchcyclesthatdevelopsolutionstoproblemsculminatinginaWeb-basedprototype.TheinitialdesignoftheLAOlearningenvironmentresulteddirectlyfromthedesignimplicationsarticulatedinthepreviousphaseofexploratoryresearch,analysesandreview.Theseimplicationsweretranslatedintoanarticulatedprototypeinitiallydevelopedbybuildinganabstract,paper-basedmodelofthesystemforresearcherandteacherinputaccordingtoproceduresadaptedfromusage-centereddesignprocessespreviouslymentionedasrolemodels(Constantine&Lockwood,1999).Rolemodelsareatechniquetocharacterizeprimaryandsecondarytargetaudiencesforthepurposesofdesign.Forexample,wecreatedrolemodelsandpersonasforchildrenwithlearningdisabilities(suchasattentiondeficitdisorder)basedonourdirectexperiencewithachildwhowasstrugglinginthereadingprocessandhismotherwhodidnothaveanyknowledgeofadvancedreadingsupportstrategies.Thesetechniquesarebasedonreal-worldexperienceswithrepresentativesofthetargetaudiencesyourinterventionorsystemisbeingdesignedforbutevolveintoaarchetypalcompositeoftheattributesofmanyindividuals.Therefore,rolemodelsandpersona’sbecomeaqualitativeprofiletocontinuallytargetdesigneffortstomaintaintheaudience(s)oruser(s)perspectives.
Abstractorlow-fidelitymodeling/prototypingoftheinstantiatedorenacteddesignprovidedopportunitiesforinputandco-constructionofLAOwithseveralaudiencememberspriortothemoretime-intensivecomputer-basedproductionofthelearningenvironment.WeutilizedConstantineandLockwood’s(seeforuse.com)proceduresofusage-centereddesignthatencompassedlow-fidelityrepresentationandorganizationofallthefeaturesofthedatabase-drivenWebsite.ForLAO,wedeliberatelyultimatelydesignedaWebdatabasesystemthatwouldpermitperformancesupportfortheparent-childdyadinprovidingmeta-cognitivepromptsforbothparticipantsbasedonresearch-basedreadingstrategiesthroughoutacollaborativeandgenerativeprocessofengagingwithtext.
an introduction to educational design research62
Inthecontextofaconstructiveresearchapproachthatattemptstovalidateaparticularconstruct(e.g.theory,model,softwareorframework)againstidentifiedcriteriaorbenchmarks,theteamconductedseveraliterativecyclesofdata-gatheringandanalysisofexpertreviewsandtargetaudiencereviews.Theseprogressive,micro-cyclesofdatacollectionandanalysisresultedindata-drivencyclicalrevisionsofthearticulatedprototypewhichwerereflectedindetaileddesigndocumentationincludingtheproductionofflowcharts,technicalspecificationsandstoryboards.Thedesignresearchprocessofemployingmicro-cyclesofconstructiveresearchdatagatheringandanalysiselicitedfeedbackateachcycleanddesignrevisionsagreeduponbytheteamwhichresultedintheinitialcreationandthenprogressiveimprovementofaWeb-basedprototypevalidatedbydatacollectedinaconstructiveresearchapproach.Asateam,weconstructedspecificcriteriarelatedtousabilityofsystemandobservationsandvideoanalysisofactualuseofthesystembyliteracyfacilitatorsandchildren.ThespecificmethodsofdatacollectionemployedatthisstageincludeddesignerlogspostedontheprojectWebsite,expertpanelreviewsofthedesignanddocumentedreviewsofthedesignbycontentexperts,audiencemembersandtheresearchteam.
TheLocalImpactPhaseOnceaphysicalWeb-basedprototypewasinplace,theincorporationofformativeevaluationandqualitativemethodsinanempiricalmannercouldcommenceandbegantocharacterizetherich,highlyiterativenatureofthelocalimpactphaseasitprogressivelyinformed,revisedandrefinedourtheoreticalconstructsaswellastheWeb-basedinstructionaldesignapproachandredesignefforts.Thecomplexinteractionsbetweenfacilitatorsandchildrenthatcanoccurinmultiplesettingsformedtheseriesofmicro-cyclesinLAOexaminingthesespecificconstructsthatgroundedrelatedresearchquestions:1)parent-childdyadsinaninformalsettingwithextensiveinvolvementbyresearchers;2)parent-childdyadsinastructuredworkshopexperiencesupportedbyresearchersand;3)pre-service-teacherdyadsinafieldtrialprogressingtowardmorecloselymodelingauthenticconditionsexperiencedwiththeprototype.Whenafullyfunctioningprototypewasnotyetavailable,studiesattemptedtocloselymimicthetasksthatwouldbeembeddedinLAO.Thedatagatheringacrossthesethreestudiesincorporatedobservations,interviews,childandparentjournalentries,videotapeduseofsystemandpre-andpost-onlinesurveys(seeJeffs,Behrmann&Bannan-Ritland,2006).Thismulti-tiered,multi-methodevaluationschemegeneratedusefulknowledgeandsubsequentresultsfromeachstageofinquirywerethencycledintochangesofourtheoreticalconjectures,researchdesignaswellassystemdesign.Thisprocessrevealedinsightsintothecoredesignprinciples(vandenAkker,etal.,2006)thatmaysupportthecollaborativelearningandimplementationofmetacognitiveprocessesbyliteracyfacilitatorsandchildreninatechnology-basedenvironment.
an introduction to educational design research 63
OurcoredesignprinciplesthatevolvedandwererefinedincludedthefollowingthatwhenengagedinacollaborativeliteracyprocessthatprovideshighlevelreadingstrategymetacognitivesupportinaWeb-basedcontext:1) parentliteracyfacilitatorscoulddevelopgreaterawarenessandskillinimplementing
readingactivitiesandidentifysupportsfortheirchildinastructuredsetting;2) childrenshowedimprovementinliteracyskillsusingtechnology-basedsupportwhen
participatinginaguidedworkshopenvironment;and3) pre-serviceteachersfeltthatthestrategiesandactivitiesembeddedintheLAO
environmentfacilitatedchildren’scomprehension,motivationandinterestwhenworkingwiththeminthisenvironment(seeJeffs,et.al.2006).
MorerigorousevaluationsareplannedforthefuturetosystematicallyincreasingnumberofparticipantsandvaryingcontextsfortheuseofLAOinschool,homeandtutoringenvironments.Thesestudiesinvolvedetailedtrackingofcomputer-basedactivitiesofthedyadsinschoolandhomesettings,assessmentoffacilitatorandchilduseofmetacognitivestrategiespriortousingLAOandpre-andpostcomprehensionmeasuresafterseveralweeksofusingthesystem.
Inaddition,aseriesofexpertreviews,usabilitytesting,one-to-to-one,smallgroup,andfieldtestingwereimplementedinprogressivelymoreauthenticsettings.Themoreintensivestudiesfocusedprimarilyonqualitativestudiesthatcharacterizedthetargetaudienceinteractionwiththeenactedtheoreticalmodelinrelationtothelearningtargets.Specifically,theteamwasinterestedinhowfacilitatorsandlearnersperceivedandinteractedwithWeb-basedsupportinthecollaborativeliteracyprocess(whichincludedbothreadingandwritingtasks).ApilotstudywasinitiallyconductedthatsimulatedsometaskswithinLAOandprovidedfeedbackontheemergingsitewithfivedyadsofmostlyparentfacilitatorsandonesiblingfacilitator.Methodsincludedcollectingdatathroughsemi-structuredinterviewsandobservationsofparent-childinteractionwiththeprototypeandcomplementaryassistivetechnologies(e.g.text-to-speech,etc.)thatpromotedinreadingandwritingactivities.
ThepreliminarystudyrevealedthatthechildrenweremotivatedtocompletereadingandwritingactivitiesontheWebandthatfacilitatorsdevelopedawarenessforimplementingreadingactivitiesinacollaborativeprocessbutdesiredadditionalsupportforchildren’sdisabilities.WhiletheWeb-basedactivitiesandsupportsforthereadingprocesswereusefulforprovidingmoreauthenticandself-initiatedreadingandwritingactivities,theresearchalsorevealedthatinteractionbetweenparentandchilddyadsduringtheseactivitiesoftencreatedtensionsthatwerenotpresentwhenchildrenwereworkingwithnon-familymembers.RevisionstothetheoreticalmodelandenacteddesignofLAObased
an introduction to educational design research64
onthiscycleofevaluationincludedamongothers,behavioralpromptsdirectedtowardtheparent-childdyadtopotentiallyreleasetension(suchaspromptstotakeabreak,positivereinforcementtechniques,etc.)whenengagedincollaborativereadingandwritingtasksandadditionalreadingstrategysupportsandactivities.
Tofurtherinvestigatetheenactedtheoreticalmodel,afollow-upsmallgroupqualitativestudywasconductedwitheightparent/childdyadsthatrepresentedavarietyofskilllevelsanddisabilities(Jeffs,2000).Thespecificgoalsofthiscycleofresearchwastoidentifythecharacteristicsofparent/childdyadsworkingtogetherspecificallyinliteracyskilldevelopment,depicttheinteractionsofthedyadandinvestigatetheimpactofvariousformsoftechnology(Internet,EPSSandanyassistivetechnology)onattitudesoftheparticipants.Participantsincludedparentsandchildrenwithvariousdisabilitiesingrades4ththrough6thwhowerereadingatleasttwogradesbelowgradelevelandhadatendencytoavoidreadingandwritingtaskspriortoparticipationinthestudy.Thestudyrevealedthatparentsrecognizedtheimportanceofimmediatefeedbackandassistivetechnologyfeaturesintheprovidedtools.Otherresultsrevealedthatwiththesupportoftheirparents,childrencanselectappropriatetechnologiesandwithintegrateduseoftheInternetandassistivetechnologies,children’swritingsamplesimprovedinbothquantityandquality.SuggestedrevisionsfortheLAOprototypebasedontheseresultsincludedbuilt-inassistivetechnologyfeatures(insteadofmerelyreferencestooutsideresources)suchastext-to-speechcapabilitiesandreadingselectionsreflectingvaryingabilitiesandareasofinterest–featuresthatweresubsequentlyincorporatedintotheLAOdesign.
Ineachofthesecyclesofproblem-state,datacollection,analysisandsubsequentdesignmoveorformativeevaluationprocess,thetheoreticalmodelenactedwithintheLAOprototypeexpandedtoincorporatenewandrevisedelementsbasedontargeteddatacollectionandresearchresults.Atthispoint,traditionalresearchanddesignprocessessomewhatdivergeinthattheanalyzedresultsarenotanendinandofthemselves,butareusedfordata-drivendecisionmakingorproblemsolvingtobuilduponorrevisetheoreticalassumptionsandimprovedesign.Often,basedontestingresults,wewouldneedtothrowoutpreviousprototypefeaturesandtotallyredesign,reviseoraddnewfeatures.Theteam’sinformeddesignjudgmentandcollaborativesocialnegotiationwaskeytothisdecision-making.
Thelocalimpactphaseisatime-intensivephasewithmultiplecyclesthatstrivestoyieldausableandinternallyvalidintervention.Testingtheinterventioninprogressivelymorerealisticsettingsprovidesvaluableinformationtoinformtheoreticalassumptionsrelatedtothedesignbutalsotobegintoisolatevariablesthatmightbefurtherempiricallytested.IntheLAOresearchconductedtodate,theintegrationofreadingstrategyscaffoldsand
an introduction to educational design research 65
assistivetechnologysupportsinthecollaborativeliteracyprocessbetweenfacilitatorsandchildrenwitharangeofdisabilitieswasidentifiedasonefactor,ofmany,thatseemtoholdpromiseforimprovingliteracyskills.Conductingadditionalresearchtofurtherinvestigatethecollaborativeprocesspromotedbythetechnologicalenvironmentaswellasisolatingtheeffectsofthemultiplereadingsupportsandassistivetechnologiesaffordedbytheprototyperemainsanimportantobjectiveinthisresearch.
AlthoughthefundingcycleforLAOhasceased,inordertoprogressfromlocaleffectstomoreexternallygeneralizableeffects,additionalcyclesoftestingareneededtoisolateandtestparticularvariablesusingmultiplesites,diverseparticipantsandsettingsprogressivelylimitingtheresearcher-participantinteraction.Basedonavailablefunding,fieldtestsortrialsareplannedforLAOtocollectsignificantamountsofquantitativeandqualitativedatafromseveralsitesandover50participantdyadsusingselectedmeasurements,onlinesurveysandinterviewsincludingparentsandchildreninhomeschoolenvironments,pre-serviceteachersandin-serviceteachersthatcouldrepresentotherliteracyfacilitatorsinseveralgeographicallocationsinteractingwithchildrenwitharangeofdisabilities.Thisdatawouldprovideadditionalevidencefortheeffectivenessofenactedtheoreticalassumptionsforthecollaborativereadingandliteracyprocessaswellasprovideevidencefortheeffectivenessoftheprototypeatitshighestfidelityinfullcontextoftheintendeduse.
TheBroadEvaluationPhaseThelastphaseofthisdesign-basedresearcheffortinvolvesdisseminatingLAOintothebroadeducationalsystem.AlthoughtheLAOresearchhasnotyetfullyprogressedthroughthisstage,initialexplorationsinthisareahaveyieldedsomeuniqueinsightsintothedisseminationprocess.However,thereadershouldnotethatthedisseminationprocesscanencompassanentireresearcheffortinitself.Forexample,Fishman(2006)hasappliedadesignresearchframeworkrelatedtothesustainabilityoftechnology-basedcurriculuminterventionswithinanentireschooldistrictorsystem.LAO,asaWeb-basedlearningenvironment,affordstheopportunitytopublishcurrentworkingprototypesonlineforopenuseandinputthathasresultedinanearlyanduniquediffusionandadoptionprocessbegunpriortothecompletionofafullyfunctioningsystem.Whilestillindevelopment,wehavetrackedover100potentialadoptersthathavediscoveredandexploredtheLAOsite.Theprofilinganddata-basecapabilitiesofthesitepermittrackingandanalysisofthisinformationthathasprovideddetailedinformationonpotentialadoptersofthesystemprovidingsignificantinsightandimpactonsourcesforourlaterdiffusionefforts.Weplantoincorporatemoresophisticatedcomputer-baseddatacollectionandanalysistechniquessuchasdatamining(Tsantis&Castellani,2001)thatmayyieldevenmoreinsightsintoearlyadopters’behaviors,profilesanduseofthisnewtool.We
an introduction to educational design research66
havejustbeguntopublishourresultsofthedesignbasedresearchconductedrelatedtoLAOintraditionalacademicjournalsandnon-traditionalWebpublishingthatprovideavenuesforadditionalformsofreviewandevaluation.TheresultsofourinitialstudieshavepromptednewresearchdirectionssuchasexploringtheinteractionofanonlinecommunityforparentsofchildrenwithdisabilitiesincorporatedintheLAOenvironment.Giventheiterativenatureofthistypeofresearch,itishighlylikelythatdeterminingtheconsequencesoftheLAOdesignresearcheffortwillyieldnewtheoreticalandappliedquestionsthatwillprompttheentireprocessonceagain.
Theoretical yield of literacy access online design research study
GiventhedesignresearchprocessbasedontheIntegrativeDesignLearningFrameworkdescribedabove,whatdidwelearn?ThecharacteristicsofaninterventionorasvandenAkker,et.al.(2006)describethe“designprinciples”areanimportantyieldofdesignresearch.IntheLAOproject,thesedesignprinciplesincludedprovidingmetacognitivereadingstrategysupportwhileaparent,teacherortutorisengagedinthecollaborativereadingprocesswiththechilddeliveredthroughacomprehensiveWeb-basedperformancesupportsystem.
Designresearchisoftenemployedtobegintogeneratetheory(Design-basedResearchCollective,2003).WithLAO,therewerenoliteraturesources,theoreticalprinciplesorresearchstudiesdirectlyapplicabletoaWeb-supportedcollaborativereadingprocesssotheteamintegratedinsightsfromtutoring,readingstrategiesandreal-timeperformancesupport.Zaritskyet.al.(2003)speaktogoing“…beyondsimpledevelopmentofaninterventionandbeyondstandardcognitiveanalysesallowingtheoryandmodelingthataccountsforthecontent,thecognitionandtheenactmentbyrealpeopleinrealandrichcontextswithreallimitsonresources(p.11).TheLAOdesignresearchteamwentbeyondtraditionaldevelopmentwithintensivecyclesofinterviews,surveys,observationalstudiesaswellasdeepinvestigationoftheone-on-onetutoringandreadingstrategiesliteraturetobuildanewtheoreticalmodelofreal-timemetacognitivereadingstrategyandassistivetechnologysupportforboththeliteracyfacilitatorandthechildwithdisabilities.Muchoftheseinsightswereanintegrationofdataanalyses,directexperiencewithtargetaudiencemembersandagroundedliteratureinreadingprocesses,tutoringandcollaborativeperformancesupport.Thedesignresearchprocesswasconductedsystematicallyto:1) uncovertheinitialconjecturesabouthowlearningmightoccurinthistypeofsetting;2) statedlearningtargets,taskanalyses(inthiscasebasedonActivityTheory);3) thedesignedinterventionwhichembodiesthecoredesignprinciples(metacognitive
readingstrategysupportinacollaborativeperformancesupportcontext);
an introduction to educational design research 67
4) localimpactorevaluationquestionsthatdrovethemoreintensiveresearchcycles(seeFigure2).
Thisprogressiondemonstratesanalignmentorcongruencyfrominitialconjecturesthroughlocalimpactorevaluationquestionsthatevolvedduringthedesignresearchstudy.ThespecifictheoreticalinsightsthatweretestedandrevealedbasedonthisprocessareincludedinFigure2.ThemultiplephasesoftheIDLFprocessuncoveredmanyinformalandformaltheoreticalinsightsbasedonmacroandmicrodatacollectionandanalysiscyclesconductedwithintheprocessofdesignthatcanbetypicallyoverlookedinthetraditionalinstructionaldesignprocess.Forexample,extendingbeyondatraditionallearneranalyses,weconductedmultiplecyclesofsurveys,interviews,andobservationsoftargetaudiencememberinteractionthatrevealedtheoreticalinsightsthatgobeyondjustthedesignoftheintervention.Ourstudiesrevealedthatparentshavelittleformalknowledgeanduseofgoodreadingstrategieswhenengagedwiththeirchildinthereadingprocess.ThisinsightparlayedintothedesignprinciplesofLAObutalsostandapartfromitasafindingthatmaycontributetotheliteratureinthereadingfield.Byformalizingandextendingthemethodsoftraditionalinstructionaldesigntopromoterichcyclesofdatacollectionthatthencaninformourknowledgeofparticularaudiences,learningcontextsandprocesses–separatebutconnectedtothedesignofaparticularintervention,wecanbegintoprogresstowardgeneratingknowledgeandusefultheoreticalinsightsthataretypicallyoverlookedindesign.Thisbecomesaninformation-lossprocessoflearningaboutlearners,contexts,andprocesseswithintheactofdesignthatdesignresearchcanrecapture,whichreferstothenotionthatinthecontextofbothisolateddesignandresearchefforts,wedonottakeadvantageofformalizingmuchofourlearninginanexploratory,confirmatoryorempiricalmanner(Bannan-Ritland&Baek,2008).
an introduction to educational design research68
Figure 2: Initial Progressive Formulation from Conjectures to Local Evaluation Questions in LAO
ConjecturesLearning Targets
Task Analysis of Learning Task,
(based on Activity Theory)
Designed
InterventionLocal Evaluation
Questions
Provinding a consistent environm
ent and reading support strategies for literacy facilitators
and children would collaboratively
engage the dyad in higher level literacy processes
1) Literacy facilitators w
ill acknowledge the im
portance of and dem
onstrate their ability to im
plement reading strategies w
hen provided technology-based support
in a collaborative reading session w
ith a child
2) Children with of w
ithout disabilities w
ill demonstrate their
abilities to access information,
activities and assisitive technology support related to reading as w
ell as interact w
ith literacy facilitators in a collaborative reading session
3) The facilitator-child dyad will
be able to explore and select appropriate assistive technology intergrated w
ith Internet-based supports that can facilitate
performance in reading and w
riting
4) Children, regardless of disability w
ill be able to capitalize on technology-based supports and a collaborative process to im
prove their literacy skills
Subject: facilitator-child dyad
Object: LAO
system
Tools: literacy strategies, assisitive technologies
Division of Labor
Comm
unity: dyads in school, hom
e and tutoring contexts
Theoretical Model Em
bedded in Electronic Perform
ance Support System
(EPSS)
How
do facilitators and learners perceive and interact w
ith W
eb-based support in the collaborative literacy process?
What are the specific
characteristics of parent/child dyads w
orking toward literacy
skill development?
What im
pact does various forms
of technology support have on the attitudes of facilitators and children w
hen engaged in the collaborative literacy process?
How
should the interaction betw
een literacy facilitators and children w
hen engaged in the collaborative literacy process
be depicted?
an introduction to educational design research 69
LimitationswillcertainlyalsoexistfortheILDFframework,astheknowledgegeneratedisonlyasgoodastherigoroftheresearchmethodsemployed.Integratingbasicqualitativeandquantitativeresearchcyclestoinformdesignatparticularpointsandgeneratebothdesignprinciplesbutalsoknowledgeaboutlearners,learningandlearningcontextsistheultimategoal.Limitationsmayexistintime,qualityofinformationuncoveredindatacyclesthatmayimpactdesign,smallNtoprovidemostlyqualitativeinsightsinitiallyandthefailureinherentinthegenerationoftheoryinthediscoveryresearchprocess.However,itisthroughapplicationindifferentdesignresearchcontextsthatmoreformalizedprocesseswillbegintobeunveiled.TheIDLFandLAOexampleareonecaseofafewcurrentlyfordesignresearcherstouncoverthelogicandwarrantsofthisnewformofresearch(Kelly,2006).Therearemanychallengesthatremainbutcapitalizingonthedesignprocesstogenerateresearch-baseddata-driveninsightsisaworthygoal,indeed.
Conclusion
ThischapterhaspresentedabriefexampleandintroductiontotheILDFframeworkthatcomprisesameta-methodologicalviewofthedesignresearchprocessinanattempttoelucidatecommonphasesandstagesinthisspecificresearchmethodology.Theframeworkispresentedtobegintoestablishcommonterminologyandprocessesthatcanpromoteconsciousdesignresearch.Mostimportantly,theILDFframeworkisanattempttoprovidearoadmapforfuturedesignresearcherstoinvestigate,articulate,documentandinformeducationalpractice.
* IamgreatlyindebtedtoDr.AnthonyE.Kellywhoseinsightsandfeedbackonthischapterwereinvaluableinextendingmythinkinginthisarea.MyappreciationalsogoestoDr.TjeerdPlomp,Dr.NienkeNieveenandDr.JanvandenAkker,esteemedcolleaguesandreviewersofthismanuscriptfortheirsuggestionsforrevision.
References
Akker,J.vanden,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge.
Bannan-Ritland,B.&Baek,J.(2008).Investigatingtheactofdesignindesignresearch:Theroadtaken.InA.E.Kelly,R.A.Lesh,&J.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, mathematics and engineering.Mahway,NJ:Taylor&Francis.
Cooper,A.(1999).The inmates are running the asylum: Why high-tech products drive us crazy and how to restore the sanity.Indianapolis,IN:SAMS.
an introduction to educational design research70
Collins,A.(1990).Toward a design science of education. Technical Report Issue No. 1 Center for Technology in Education.NewYork,NY.ResourcesinEducation(ERIC)ED326179.
Collins,A.(1993).Design issues for learning environments.TechnicalReportNo.27CenterforTechnologyinEducation,NewYork,NY.ResourcesinEducation(ERIC)EDED357733.
Collins,A.(1999).Thechanginginfrastructureofeducationresearch.InE.C.Lagemann&L.S.Shulman(Eds.),Issues in education research: Problems and possibilities (pp.289-298).SanFrancisco:Jossey-Bass.
Confrey,J.&Lachance,A.(2000).Transformativeteachingexperimentsthroughconjecture-drivenresearchdesign.InA.E.Kelly&R.ALesh(Eds.),Handbook of research design in mathematics and science education (pp.231-265).Mahway,NJ:LawrenceErlbaumAssociates.
Constantine,L.L.&Lockwood,L.A.(1999)Software for use: A practical guide to the models and methods of usage-centered design.Addison,WesleyLongman,Inc.
TheDesign-BasedResearchCollective(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher 32(1) 21-24.
Fishman,B.,Marx,R.,Blumenfeld,P.,Krajcik,J.S.&Soloway,E.,(2004).Creatingaframeworkforresearchonsystemictechnologyinnovations.Journal of the Learning Sciences, 13(1),43-76.
Graue,M.E.&Walsh,D.(1998).Studying children in context.ThousandOaks,CA:SageJeffs,T.(2000).Characteristics, interactions, and attitudes of parent/child dyads and their use
of assistive technology in literacy experience on the internet.Unpublisheddissertation,GeorgeMasonUniversity,Fairfax,VA.
Jeffs,T.,Behrmann,M.&Bannan-Ritland,B.(2006).Asssistivetechnologyandliteracylearning:Reflectionsofparentsandchildren.Journal of Special Education Technology 21(1), 37-44.
Kelly,A.E.,Lesh,R.A.&Baek,J.Y.(Eds).(2008).Handbook of Design Research Methods in Education Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching.NewYork:LawrenceErlbaumAssociates.
Kelly,A.E.(2006).Qualitycriteriafordesignresearch.In:J.J.H.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge.
Kelly,A.E.,Baek,J.,Lesh,R.,&Bannan-Ritland,B.(2008).Enablinginnovationsineducationandsystematizingtheirimpact.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.
LiteracyAccessOnlinePhaseFourProjectWebSite(2000).RetrievedAugust30,2002,fromGeorgeMasonUniversity,ImmersionProgramWebSite:http://chd.gse.gmu.edu/immersion/lao/deliver/needs.htm
Rogers,E.M.(2003).Diffusion of innovations.NewYork:TheFreePress.Sabelli,N.(personalcommunication,May15,2002).
an introduction to educational design research 71
Tessmer,M.(1993).Planning and conducting formative evaluations.London:KoganPage.Tsantis,L.&Castellani,J.(2001).Enhancinglearningenvironmentsthroughsolutionbased
dataminingtools:Forecastingforself-perpetuatingsystemicreform. Journal of Special Education Technology, 16(4).
Wasik,B.A.(1998).Usingvolunteersasreadingtutors:Guidelinesforsuccessfulpractices.Reading Teacher,51(7),562-70.
Wittrock,M.C.(1998).Cognitionandsubjectmatterlearning.InN.M.LambertandB.L.McCombs(Eds.),How Students Learn(pp.143-152).Washington,DC:AmericanPsychologicalAssociation.
Zaritksy,R.,Kelly,A.E.,Flowers,W.,Rogers,E.,&O’Neill,P.(2003).Clinicaldesignsciences:Aviewfromsisterdesignefforts.Educational Researcher, 32(1), 32-34.
an introduction to educational design research 73
4. WhenisDesignResearchAppropriate Anthony E. Kelly
Introduction
Designresearchhasbeendescribedindetailinmanypublications,mostrecentlybytheDutch(VandenAkker,Gravemeijer,McKenney&Nieveen,2006,withe.g.,Kelly,2006),andtheKelly,LeshandBaek(2008)collectionofpapers(e.g.,Kelly,Lesh,Baek&Bannan-Ritland,2008;Middleton,Gorard,Taylor&Bannon-Ritland,2008).Plomp(chapter1ofthisbook)alsoprovidesanoverview.Forthatreason,Iwillnotreiteratethedescriptionhere.Rather,Iwillassumethatthereaderisfamiliarwiththesesourcesandthespecialissuesofjournals(e.g.,Barab&Squire,2004;Kelly,2003,2004)thathaveappeared.
Instead,Iwishtoplacedesignresearchwithintheframeofalargercontextforresearchoninterventions.Inherseminalpiece,Bannan-Ritland(2003)describedaportfolioofresearchactivitiesusingthefollowingcategories:
• InformedExploration• Enactment• Evaluation:LocalImpact - Quasi-experimentaldesigns - Randomizedtrials - HierarchicalLinearModeling• Evaluation:BroaderImpact - Implementationinnewcontexts(DesignandResearch) - ImplementationatScale - ScalingupDesignandResearch - Web-enabledprotodiffusion - DiffusionofInnovations(Rogers)• Adoption,adaptation,acceptance,rejection
Ofcourse,thislargerframeworkcallsformanydifferentresearchmethods.Inhispaper(Plomp,chapter1),brieflycapturesthefunctionsofresearchmethods:
• survey:todescribe,tocompare,toevaluate• case studies: todescribe,tocompare,toexplain• experiments: toexplain,tocompare• action research:todesign/developasolutiontoapracticalproblem
an introduction to educational design research74
• ethnography:todescribe,toexplain• correlational research:todescribe,tocompare• evaluation research: todeterminetheeffectivenessofaprogram
HethenprovidesexamplesrelatedtotheChinesecontext:
1. to describe:e.g.whatistheachievementofChinesegrade8pupilsinmathematics;whatbarrierstostudentsexperienceinthelearningofmathematicalmodelling
2. to compare:e.g.whatarethedifferencesandsimilaritiesbetweentheChineseandtheNetherlandscurriculumforprimaryeducation;whatistheachievementinmathematicsofChinesegrade8pupilsascomparedtothatincertainothercountries
3. to evaluate:e.g.howwelldoesaprogramfunctionintermsofcompetencesofgraduates;whatarethestrengthsandweaknessesofacertainapproach;etc
4. to explain or to predict:e.g.whatarethecausesofpoorperformanceinmathematics(i.e.insearchofa‘theory’predictingaphenomenonwhencertainconditionsorcharacteristicsaremet)
5. to design and develop:e.g.whatarethecharacteristicsofaneffectiveteachingandlearningstrategyaimedatacquiringcertainlearningoutcomes;howcanweimprovethemotivationoflearners.
BothBannan-RitlandandPlompprovideabroadercontextforresearch.Withinthislargerframework,wemayask,therefore:Whenisdesignresearchappropriate?Wemayapproachananswerbyasking,first,whenisdesignresearchinappropriate?
When is design research inappropriate?
Areviewofthemanypublishedexamplesofdesignresearch(e.g.,Kelly,Lesh&Baek,2008)demonstratetheheavyinvestmentoftimeandresourcesnecessarytomakeprogressinthefaceofsometimesdauntingcircumstances.Designresearchrequiresinvestmentofsubstantialresourcesatmanylevels:schooldistrictadministrators,teachers,students,andthedesignresearchteam(whichmayincludeeducationresearchers,softwaredevelopers,curriculumspecialists,andsoforth).Thus,designresearchisinappropriateiftheeducationalproblemisfairlysimple.
Iftheproblemhasaknownorstandardsolution,andthereisgeneralagreementonwhentoapplythesolution,andthesolutionhasbeenregularlysuccessfullyappliedinvarioussettings,designresearchisprobablyapooruseofresources.
an introduction to educational design research 75
Evenformorechroniclearningproblemssuchasreading,ifthereareadequatetrainingprograms,andclearmeasuresofsuccessorprogress(e.g.,useofphonicstoteachdecodingskills),designresearchisprobablynotindicated.If,however,newresearchsuggestsapowerfulinnovation,designresearchmaybeareasonablechoice(seebelow,andMcCandliss,Kalchman&Bryant,2003)
Generally,designresearchisprobablynotrecommendedforclosedproblems(e.g.,improvingmathematicscalculationfluency),wherethe:
• Initialstate(s)areknown(e.g.,twonumbersaretobemultiplied;achessboardisreadytoplay).
• Goalstate(s)areknown(e.g.,aproductoftwonumbersistoproduced;checkmateorstalemateinchess).
• Operatorstomovefrominitialstatestogoalstatesareknownandcanbeapplied.(e.g.,theproceduresofmultiplication;therulesofchess).
When is design research appropriate?
Designresearchisrecommendedwhentheproblemfacinglearningorteachingissubstantialanddauntinghow-to-doguidelinesavailableforaddressingtheproblemareunavailable.Further,asolutiontotheproblemwouldleadtosignificantadvancesinlearningoratleastasignificantreductioninmalfunctionintheeducationalsystem.
Thereshouldbelittleagreementonhowtoproceedtosolvetheproblem,andliteraturereviewstogetherwithanexaminationofothersolutionsappliedelsewhere(i.e.,benchmarking)shouldhaveprovenunsatisfactory.Designresearchisfurthersuggestedifpriortrainingorinterventionshaveconsistentlyprovenunsuccessful.Designresearchisoftenindicatedforcriticaleducationalgoals,evenwhenthereisnotacleardefinitionofsuccess,ordesigningadequateindicatorsofsuccessispartoftheoverallproblem.
Inotherwords,designresearchismostappropriateforopen,ormoreappropriately,wickedproblems.TheconceptofawickedproblemwasdescribedbyRittelandWebber(1977)todescribeproblemsthatsharethefeaturesofopenproblems,butthatalsoengageelementsthatmaketheirsolutionfrustratingorpotentiallyunattainable.
an introduction to educational design research76
Followingfromthedescriptionofclosedproblems,above,inopenproblems,someormoreofthefollowingapply:
• Initialstate(s)areunknownorareunclear.• Goalstate(s)areunknownorareunclear.• Operatorstomovefrominitialstatestogoalstatesareunknownorhowtoapplythe
operatorsisunclear.
Forwickedproblems(e.g.,Camillus,2008;Horn&Weber,2007;Richey,2007),thecharacterofopenproblemspertain.Plus,therearetypicallyinadequateresources,unclear“stoppingrules”(conditionsthatindicateasolutionisathandortheprojectshouldbeabandoned),uniqueandcomplexcontexts,andinter-connectedsystemicfactorsthatimpingeonprogress.Mostfrustrating,theseotherfactorsmaythemselvesbesymptomsofproblemsofassociatedwickedproblems.Forexample,attemptingtoteachnumeracyinasocietywithhighpovertyandHIVrates.
Therefore,oneofthebroadgoalsofdesignresearchistodynamicallyclarifytheinitialandgoalstatesandtheoperators,andtoilluminatethenatureoftheproblem–i.e.,to“tame”awickedproblembybetterspecifyingitscharacterandmakingitopentointervention.Ineducationalsettings,designresearchisrecommendedwhenoneormoreofthefollowingconditionsoperatetomaketheproblemmorewickedandopenthansimpleandclosed,forexample:
• Whenthecontent knowledgetobelearnedis new or being discoveredeven by the experts.
• Whenhowtoteachthecontentisunclear:pedagogical content knowledge is poor.• Whentheinstructional materialsarepoorornotavailable.• Whentheteachers’ knowledgeandskillsareunsatisfactory.• Whentheeducational researchers’ knowledgeofthecontentandinstructionalstrategies
orinstructionalmaterialsarepoor.• When complex societal, policy or political factors may negatively affect progress.
AnumberofexamplesofmaybefoundinKelly,LeshandBaek(2008).Someotherexamplesfrommathematics,science,andreadingarebrieflypresentedinthenextsection.
Examples from mathematics, science and reading
Thissectionpresentsbrieflyanumberofexamplesofwhenapplyingdesignresearchistheappropriateresearchapproach.
an introduction to educational design research 77
1. IntroducingExistingScienceorMathematicsatEarlierGradeLevelsForexample,someeducationauthoritieshaveadvocatedtheteachingofalgebrainearliergrades(asearlyasthe8thgradeintheUS),seeFoundationsforSuccess:ReportoftheNationalMathematicsAdvisoryPanel(http://www.ed.gov/about/bdscomm/list/mathpanel/index.html).Afewpolicymakershaveevenadvocatedstartingalgebrainstructionintheearlyelementary.
Howshouldoneproceedtointroduceideasofalgebraicreasoningintheearlyelementarygrades?Isthisrecommendationadvisable?Thisissueclearlymeetsthecriteriasetout,above.SomeofthecomplexitiesassociatedwithansweringthisquestioncanbegaugedbyreadingsomeoftherecentworkonthistopicbyCarraherandcolleagues(e.g.,Carraher&Schliemann,2007;Carraher,Schliemann,&Schwartz,2007;Peled,&Carraher,2007;Schliemann,Carraher&Brizuela,2007).
2. Learningneworemergingsciencecontent(e.g.,genetics)Researchinmicrobiologyisinrevolutionwithstunningfindingsappearingonfrontpagesofnewspapers,almostdaily,worldwide.Howcanhistoricalscienceeducationbeupdatedtopreparehighschoolteachersandstudentstomeetthischallengeandopportunity?Moreover,howcanhighschoolspreparestudentstobesuccessfulinemergingintegratedbiologyprogramssuchastheoneatPrincetonUniversity(http://www.princeton.edu/integratedscience/)?RutgersUniversityhasexploredthischallengethroughitsmicrobiologyprogram(http://avery.rutgers.edu/WSSP/Begin/index.html).Areviewofitsvariedsolutionstothischallengeexemplifiesthisrichcontextfordesignresearch.
3. UncoveringthePotentialContributionsofNeuroscienceforMathematicsLearningTheauthorhasjoinedotherwriters(e.g.,Varma,McCandliss&Schwartz,2008)inoutliningthecaseforcultivatingtheintersectionofneuroscienceandmathematicslearning.(e.g.,Kelly,2002,2008).
Whyisthereagrowinginterestinneuro-mathematicseducation?Anumberoffactorshavecoincidedtosupportasurgeininterestinbrain-basedmathematicseducationresearch(seeOECD,2007foracomprehensivereviewofbrain-relatedresearchineducation):• Confidenceduetorecentgainsinunderstandingthebrainbasesforprocessesof
decodinginreading.• Emergentfindingsintheneuralbasesformathematicalthought.• Decadesofbehavioralandcognitivesciencefindingsonlearningmathematicsand
relatedhigher-orderprocessesfromwhichtodraw.
an introduction to educational design research78
• Adesiretodisambiguateandconstrainresearchhypothesesatthebehavioral,cognitiveandsociallevelsofanalysis.
• Adesiretosharpenandgrounddiagnosisandremediationofmathematicallearningdifficultieswithimprovedassessments.
• Adesiretoconstructnewmixed-methodsresearchmethodologiesforthesocialsciences.
• Adesiretoscientificallydebunklearningandteaching“neuromythologies”.• Asenseofurgencytobringscientificdiscourse,evidenceandreasoningtotheslateof
ethicalissuesthatareemergingthatpertainbothtolearningandteaching.• Agoaltoimprovemethodsofteachingofmathematics.• Agoaltoimproveeducationalmaterials,includingthosethatusecomputerhardware
andsoftware.• Morecomprehensiveandtestablemodelsoflearningemergingfromcognitivescience
(e.g.,Bruer,1997).• Adesiretounderstandandpromotesignificantmathematicalcreativity.• Tochallengeneuroscientiststocontinuetopushtheboundariesofimaging
technologies,andtoco-formulateclinicallearningtasks.Thepointtobedrawnhereisthatthecoincidenceofthesefactors,alone,doesnotdictateteachingorlearningstrategiesorevenprovideprinciples,materials,curricula,interventions,orassessmentapproachestosupporteitherlearningorteaching.How,then,shouldresearchersproceedtobringthelaboratoryfindingsofcognitiveneuroscienceintotheclassroominviableways?Again,theproblemmeetstheaboverequirementsforusingdesignresearch.
4. CyberinfrastructureCyberinfrastructureencompassestheuseofdistributedinternetresourcessuchascomputingsystems,data,informationresources,networking,digitallyenabled-sensors,instruments,virtualorganizations,andobservatories(NSF,2007).Itallowstolinkgroupsofscientiststoattackmulti-levelcomplexproblems.Theseproblemswillhaveassociatedchallengesforlearning,teaching,andassessment.Importantquestionsarehoweducationshouldcapitalizeoncyberinfrastructureresources.Whatitmeanstostudysciencecontentwithinacyberinfrastructureframework,andwhatthecurricular,instructionaldesign,assessment,teacherprofessionaldevelopment,andpolicyquestionsthatareraised,andhowtheymustbeansweredtofullyexploitthehigh-technologyinvestmentinscienceatthislevel.Asimportant,whatarethemethodologicalchallengesinstudyinglearningwithinacyberinfrastructureproject?Forexample,howareclaimsofcausalityhandledinacomplexnetworkedandnestedlearningenvironment,andwhatevidencewouldmakesuchclaimscredible(e.g.,Kelly&Yin,2007)?Thisisaclearexample,spanningmanysciencedisciplines,forwhichdesignresearchisanappropriateinvestment.
an introduction to educational design research 79
TheAppendixtothischapterdiscussesinmoredetailthemeaningandpossibilitiesofcyberinfrastructureore-scienceingeneralandforeducation.
5. ReadingandInquiryScienceThereader’sattentionisdrawntotwoexamplesfromBrendaBannan-Ritland,currentlyatGeorgeMasonUniversity.Heranalysisofhowdesignresearchworkswithinherintegrativelearningdesignframework(Bannan-Ritland,2003;Bannan-Ritland&Baek,2008;seealsopaperinthisvolume)providesexamplesinnarrower,ifnolessimportant,applications.
Forexample,designresearchisappropriatewhendevelopingcreativeorinnovativeeducationalproducts,blueprintsordesignsthataredirectedatchroniceducationalproblems.Inanumberofpapers(seeLiteracyAccessOnline,Bannan-Ritland&Baek,2008;http://immersion.gmu.edu/lao/spring2003/projectResources.htm),andBannan-Ritland’schapter(thisvolume)describestheprocessesundertakentoaddressachronicprobleminmostcountries,howtoteachreadingtostrugglingreaders.
FollowingherworkonLiteracyAccessOnline,Bannan-Ritlandextendedherworkonreadingdesignintothelearningofinquiryscienceatthe4thgrade.Basedonthisexperience,Bannan-Ritlandsignificantlyaddedtothebroadeninguseofdesignresearchprinciplesbymethodologicallyincorporatingteachersasdesignersintheoveralldesignresearchparadigm.Thisexcitingnewdirection,calledteacherdesignresearch(whichdovetailswithworkbyZawojewskietal.,2008),isdescribedinBannan-Ritland(2008).Theareaofapplicationinthereportisearthsciencesintheearlyelementaryschool.
The growing need for design assessment research
Arecentreviewofcontributionstodesignresearchshowanincreasingawarenessoftheneedfortacklingtheproblemofhowtoassesslearninginemergingareasoflearning,particularlywhenthereisanemphasisoninnovationininstructionalpractices(Kelly,Baek,Lesh&Bannan-Ritland,2008).Theynote:
In design research as currently practiced, assessment is not directed at some summative sense of learning, though a summative measure of student
learning would be central to later attempts at confirmatory studies, i.e. to show local impact (Bannan-Ritland, 2003). . . . Design research also differs from forma-
tive assessment with regard to the student’s knowledge end state and how feedback loops are enacted. Formative assessment is the gathering of data
relative to some predetermined fixed point, providing feedback that informs the students and teacher of their current knowledge state in relation to some
an introduction to educational design research80
end state (see Black & Williams, 1998). In design research, assessment may be used formatively in order to dynamically determine progress toward mastery of disciplinary knowledge (e.g., Cobb & Gravemeijer, [Kelly, Lesh & Baek, 2008])
or to guide the design of a prototype and to inform its iterative re-design as necessary or both. In fact, sensitivity to assessment practices themselves may
inform changes to the act of assessment itself (e.g., Lobato, [Kelly, Lesh & Baek, 2008]; Lesh et al., [Kelly, Lesh & Baek, 2008]). Ultimately, design researchers are
challenging the assumptions about learning, teaching, and knowing that underlie available assessment techniques, not only in terms of the psychome-tric assumptions (like item response theory), but also the function of assess-
ment itself within and across the stages of design research (see Sloane & Kelly, [Kelly, Lesh & Baek, 2008]).
Inotherwords,whenasuitablecontextfordesignresearchisidentified,totheextentthattheapplicationisnovel(e.g.,teachingalgebraicconceptsintheearlyelementarygrades,readingcomprehension)ortheknowledgeunfolding(e.g.,genomics,cyberinfrastructure),therewillbearequirementandaresponsibilityforresearchersnotonlytoiterativelyinvestigatetheimpactoflearningprototypes,butalsotoaddressdirectlythequestionofhowthisimpactwillbemeasured.Thepointhereisnotthatassessmentisnecessary,ratherthatthetargetsforassessmentmayarisedynamicallyinthecourseofdesignresearchandmeasuresmaynotbeavailableapriori.Asaresult,manyofthequestionsaboutthevalidityandreliabilityofmeasureshavetobeactivelyreconsidered.Inpractice,toooften,prototypesareredesignedwithoutspecifyingtheevidencebase(viaassessmentdesign)fortheredesign.Inmanycases,designresearchersappeartorelyonjudgmentorsubjectivefactors.Addingtotheunfoldingneedfornewmethodsindesignassessmentwillbeamajorchallengeandopportunityforscholarsinthenextdecade(e.g.,Kelly,2005a,2005b).
Whatistheevidencetosupportclaimsofeffectivenessduringiterations,andlater,astheinnovationissubjecttomorerigoroustests?
Design research in general practice
Thegoalofthischapterwastocharacterizedesignresearchatabroadlevel,andtoprovidesomeexamplesofwherethesignificantresourcesassociatedwithdesignresearchmightbespent.Iwillfinishwithageneraloutlineofhowdesignresearchcyclesunfoldwithinalargerframeworkofresearch(Bannan-Ritland,2003;Plomp,thisvolume).Usingcognitivescience,cognitivepsychologyandothersocialsciencemethodssuchassurveys,casestudies,clinicalinterviews,ethnography:
an introduction to educational design research 81
• Identifyorcharacterizetheinitialstates.Clarifytheinitialknowledgeandgoalknowledgestates(ofstudents,teachers,researchers,experts)usingtheinterventions.
• Identifyorcharacterizethegoalstates.Designformativeassessmentstomonitorprogresstowardthegoalstate.
• Identifyorcharacterizetheoperators.Dynamicallyusingthecognitiveandotheranalyses,iterativelydesignandspecifytheoperators(interventions,supports,environments)tosupportlearning.See,inparticular,theworkofBannan-Ritland(2008)andZawojewskietal.,(2008).
• Informre-designcyclesoriterationsusingdatagatheredfromunfolding,andparallelworkindesignassessment.
• Worktowarddevelopingamatureprototypethatcanbesubjecttoamoredefinitivetest(e.g.,randomizedclinicaltrial),seeBannan-Ritland’s(2003)localimpactphase.
One final note: Prototyping and theory building
Byperturbingthesystemusingtheinterventionsinthisiterativeresearchprocess,designresearchtranscendseachofthelocalmethodsused.Inotherwords,designresearchinvolvesnotonlytheuseofdifferentmethods(e.g.,surveys,casestudies,clinicalinterviews),butcombinesthefruitsofeachmethod,overtime,tospecifytheoryandmodelsrelatedtolearning,teachingandassessingthetargetknowledge(seeCobb&Gravemeijer,2008).Thus,designresearchgoes beyond simple development of an interventionandgoes beyond standard cognitive analysesandallowstheoryandmodelingthataccountsforthecontent,thecognition,andtheenactmentbyrealpeopleinrealandrichcontextswithreallimitsonresources(seeZaritskyetal.,2003).
Thequestionofthe“theoreticalyield”ofdesignresearchisnotasimpleone.Notethatthischapterwasframedintermsofcomplex,openandwickedproblems.Forsuchproblems,thereexistsnosimpletheoreticalmodel(atleastnoneisperceivedatthetime).Forthatreason,if“theory”issomethingthatisassumedtobeinformedbyhypothesistestingofasomewhatdefinitivequestion,thendesignresearch(inearlystages)willlikelynotposeoreasilyanswersimplehypotheses,andthusnothavesimpletheoreticalyield.Schwartz,ChangandMartin(2008;inKelly,Lesh&Baek)viewsthedesignresearchcyclesaspreparatoryfortheoreticalyieldfromlaterrandomizedclinicaltrialsorotherlaboratorytests.Iftheobservationisborneoutthatmuchofeducationalinterventionoccursincomplexsystems,thenthetheoreticalyieldwillnotbeassociatedwithonetheory,butmany(perhapsinterdependent)subtheories.Ifso,thentheyieldmaybediffuseandobfuscatedbytheinfluenceofmanyfactorsthatarenotcontrolledindesignresearchsettings.Someresearchershaveattemptedtoframedesignresearchwithinanoverarchingtheory(say,“variation”theory,Holmqvist,Gustavsson,&Wernberg,2008).Thepay-offofthisapproachwillinformusgreatlyabouttheroleoftheoryindesignresearch.
an introduction to educational design research82
Somewritersusetheword“theory”moregenerallytoencompass“designprinciples,”anditmaybethecasethatsuchprinciplescanindeedbeidentified(seeKali,2008).Suchrecommendationsfordesignpracticeareusefulheuristics.Iftheseheuristicsshowevidenceofdurableapplicabilityacrossmanyprojectsandcontexts,itislikelythatsomenecessary(asopposedtocontingent)principlesarebeingevoked(seeKelly,2004),whichwouldopentheseheuristicstotheoreticalanalysis.
References
Akker,J.J.H.vanden,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds.)(2008).Educational design Research.London:Routledge.
Bannan-Ritland,B.(2003).Theroleofdesigninresearch:Theintegrativelearningdesignframework.Educational Researcher 32(1) 21-24.
Bannan-Ritland,B.(2008).Teacherdesignresearch:Anemergingparadigmforteachers’professionaldevelopment.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.
Bannan-Ritland,B.,&Baek,J.Y.(2008).Investigatingtheactofdesignindesignresearch.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.
Barab,S.&Squire,K.(2004).Design-basedresearch:puttingastakeintheground.Journal of the Learning Sciences, 13(1),1-14.
Camillus,JohnC.,“StrategyasaWickedProblem,”HarvardBusinessReview,May2008.Carraher,D.W.&Schliemann,A.D.(2007).Earlyalgebraandalgebraicreasoning.InF.Lester
(ed.)Second handbook of research on mathematics teaching and learning: A project of the National Council of Teachers of Mathematics.VolII.Charlotte,NC:InformationAgePublishing,pp.669-705.
Carraher,D.W.,Schliemann,A.D.&Schwartz,J.(2007).Earlyalgebraisnotthesameasalgebraearly.InJ.Kaput.D.Carraher,&M.Blanton(Eds.),Algebra in the early grades. Mahwah,NJ,Erlbaum,pp.235-272(nowTaylor&Francis).
Chin,G.,Jr.,&Lansing,C.S.(2004).Capturingandsupportingcontextsforscientificdatasharingviathebiologicalsciencescollaboratory,Proceedings of the 2004 ACM conference on computer supported cooperative work,409-418,NewYork:ACMPress.
Cobb,P.&Gravemeijer,K.(2008).Experimentingtosupportandunderstandlearningprocesses.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.
an introduction to educational design research 83
Cogburn,D.L.(2003).HCIintheso-calleddevelopingworld:what’sinitforeveryone,Interactions,10(2),80-87,NewYork:ACMPress.
Holmqvist,M.,Gustavsson,L.,&Wernberg,A.(2008).Variationtheory:Anorganizingprincipletoguidedesignresearchineducation.InA.E.
Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.
Horn,RobertE.,Knowledge Mapping for Complex Social Messes,apresentationtothe“FoundationsintheKnowledgeEconomy”conferenceattheDavidandLucilePackardFoundation,July16,2001.
Kali,Y.(2008).TheDesignPrinciplesDatabaseasmeansforpromotingdesign-basedresearch.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.
Kelly,A.E.(2003).Theroleofdesigninresearch.Educational Researcher, 32(1),3-4.Themeissueguesteditor.
Kelly,A.E.(2004).Designresearchineducation:Yes,butisitmethodological?Journal of the Learning Sciences, 13(1),115-128.
Kelly,A.E.(2005a).“How do modeling perspectives inform program assessment?”Presentationatthe27thannualmeetingoftheNorthAmericanChapteroftheInternationalGroupforthePsychologyofMathematicsEducation.Roanoke,VA.
Kelly,A.E.(2005b).“Theintersectionofdesignandassessmentformathematicslearning.”Presentation,NationalScienceFoundationWorkshop,ParkCity,UT.
Kelly,A.E.(2006).Qualitycriteriafordesignresearch.In:J.J.H.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge.
Kelly,A.E.(2008).Brainresearchandeducation:Potentialimplicationsforpedagogy.InEducation, sciences cognitives et neurosciences.Paris:PressesUniversitariesdeFrance.
Kelly,A.E.,Baek,J.,Lesh,R.,&Bannan-Ritland,B.(2008).Enablinginnovationsineducationandsystematizingtheirimpact.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.
Kelly,A.E.&Yin,R.(2007).Strengtheningstructuredabstractsforeducationalresearch:Theneedforclaim-basedstructuredabstracts.Educational Researcher.133-138.
McCandliss,B.D.,Kalchman,M.,&Bryant,P.(2003).Designexperimentsandlaboratoryapproachestolearning:Stepstowardcollaborativeexchange.Educational Researcher, 32(1),14-16.
Middleton,J.,Gorard,S.,Taylor,C.,&Bannan-Ritland,B.(2008).The“compleat”designexperiment:Fromsouptonuts.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.
an introduction to educational design research84
NationalScienceFoundation(2007).Cyberinfrastructure vision for 21st Century discovery.OfficeofCyberinfrastructure.March.
OECD(2007).Understanding the brain: The birth of a learning science.Paris:OECD.Olson,G.M.,Teasley,S.,Bietz,M.J.,&Cogburn,D.L.(2002).Collaboratoriestosupport
distributedscience:theexampleofinternationalHIV/AIDSresearch,Proceedings of the 2002 annual research conference of the South African institute of computer scientists and information technologists on enablement through technology,44–51.
Peled,I.&Carraher,D.W.(2007).Signednumbersandalgebraicthinking.InJ.Kaput.D.Carraher,&M.Blanton(Eds.),Algebra in the early grades.Mahwah,NJ,Erlbaum,pp.303-327(nowTaylor&Francis).[608k]
Richey,Tom;Wicked Problems: Structuring Social Messes with Morphological Analysis,SwedishMorphologicalSociety,lastrevised7November2007
Schliemann,A.D.,Carraher,D.W.,&Brizuela,B.(2007).Bringing out the algebraic character of arithmetic: From children’s ideas to classroom practice.LawrenceErlbaumAssociates(nowTaylor&Francis).
Schwartz,D.L.,Chang,J.,&Martin,L.(2008).Instrumentationandinnovationindesignexperiments:Takingtheturntowardsefficiency.InA.E.Kelly,R.Lesh,andBaek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.
Thomas,J.J.,&Cook,,K.A.(Eds.),(2005).Illuminating the path: The research and development agenda for visual analytics. LosAlamitos,CA:IEEEComputerSociety.
Varma,S.,McCandliss,B.D.,Schwartz,D.L.(2008).Scientificandpragmaticchallengesforbridgingeducationandneuroscience.Educational Researcher, 37(3),140-152.
Wulf,W.(1989,March).Thenationalcollaboratory.InTowards a national collaboratory. UnpublishedreportofaNationalScienceFoundationinvitationalworkshop,RockefellerUniversity,NewYork.
Zaritksy,R.,Kelly,A.E.,Flowers,W.,Rogers,E.,&O’Neill,P.(2003).Clinicaldesignsciences:Aviewfromsisterdesignefforts.Educational Researcher, 32(1),32-34.
Zawojewski,J.,Chamberlin,M.,Hjalmarson,M.A.,&Lewis,C.(2008).Developingdesignstudiesinmathematicseducationprofessionaldevelopment:Studyingteachers’interpretivesystems.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.
an introduction to educational design research 85
Appendix
FortheUSNationalScienceFoundation(NSF,2007),theopportunitiesinthecomplementaryareasthatmakeupcyberinfrastructure:computingsystems,data,informationresources,networking,digitallyenabled-sensors,instruments,virtualorganizations,andobservatories,alongwithaninteroperablesuiteofsoftwareservicesandtoolsprovidechallengesalongthreelines:(a)data,dataanalysis,andvisualization;(b)virtualorganizationsfordistributedcommunities;and(c)learningandworkforcedevelopment.
AmajorparallelactivityincyberinfrastructureisunderwayinEurope,whichislabeled“e-science”.e-sciencedescribessimilaractivitiestotheUScyberinfrastructure.NotunlikeearlyvisionsofUScyberinfrastructure,theUKlaunchingdocument,(http://www.nesc.ac.uk/documents/OSI/report.pdf),didnotexplicitlylisteducationasoneofthekeyareasofconcerninsettingupacyberinfrastructure.Itfocused,rather,onnetworks,middleware,digitallibraries,andcomputationalresources.AsintheUS,thisimbalanceisbeingrecognized.InEurope,itbeingaddressedbythecreationofICEAGE:“Theinternationalcollaborationtoextendandadvancegrideducation”(http://www.iceage-eu.org/v2/partners.cfm).ICEAGE,whileinternational,isprimarilyaEuropeaneffort,withbranchesinEdinburgh,Scotland,UniversityofCatania,Sicily,SPACI(SouthernPartnershipforAdvancedComputationalInfrastructures),anItalianuniversity-basedeffort(http://www.spaci.it/),CERN,nearGeneva(http://public.web.cern.ch/Public/Welcome.html),theRoyalInstituteofTechnologyinSweden(http://www.kth.se/?l=en_UK),andTheComputerandAutomationResearchInstitute,HungarianAcademyofSciences(http://www.sztaki.hu/institute).
Cyberinfrastructuredescribestheuseofdistributedinternetresourcestolinkgroupsofscientiststoattackmulti-levelcomplexproblems.Theseproblemswillhaveassociatedchallengesforlearning,teaching,andassessment.Forexample,adesignresearchproblemwouldbehowtodescribeandcreditastudent’slearninginacyberinfrastructureresearchcollaboratoryingeosciences:Scientifically,acrucialconcernindetectingearthquakesistomeasureminutechangesinelevation.Traditionalradar,whichusesradiowavesasthemeansofdetectingdistancesfromthesource,areoflimitedvalueinprecisemeasurementsduetothelengthoftheradiowaves.TheuseofLiDAR(LightDetectionandRanging)technologyallowstheuseofwavelengthsintheultraviolet,visible,ornearinfraredrange(fromabout10micrometerstotheUV(ca.250nm).Theseshorterwavelengthsallowdetectionofsmokeandotherdiffuseparticulates,whichhasledtotheuseofLiDARinmeteorology.Forearthquakeprediction,LiDARcanbeusedtolocatefaults,andtomeasureuplift.Faultsdescribethelineoffractureanddemarcationbetweenplates(McKnight&Hess,2000).
an introduction to educational design research86
Upliftistypicallyduetotectonicplateactivity(Kearney&Vine,1990),technically“orogenicuplift”orduetotheremoval(duetoerosion)ofheavymaterial,technically“isostaticuplift.”ThesignificantadvantageofLiDARoverradaristhatLiDARcangeneratedigitalelevationmodels(DEMs)oftheshapetheearth’ssurfaceatresolutionsnotpreviouslypossible.Complexifyingtheproblem,anearthquakeissometimesassociatedwithvolcanicactivity.Forexample,the“PacificRimofFire”isassociatedwithcollidingtectonicplates.Insuchcases,LiDARmaybeusednotonlytomakeprecisemeasurementsofelevation,butalsotocharacterizethedensityandeventhechemicalmakeupofthegasesandashemittedbyavolcano.LiDARdataonMountSt.Helen’svolcanomaybefoundathttp://wagda.lib.washington.edu/data/type/elevation/lidar/st_helens/.
LearningaboutgeomorphologyusingLiDARiscomplex,andsomepubliclyavailablewebsiteshaveattemptedtoprovideinstruction(e.g.,http://lidar.cr.usgs.gov/andhttp://gisdata.usgs.net/website/lidar/viewer.php).ThemostcomprehensiveactivityhasbeenconductedbytheGEONnetwork(http://www.geongrid.org/).Thisnetworkispartofacyberinfrastructureresearchcollaborator.TutorialsontheuseofLiDARwithinandoutsideofgeoscience(e.g.,coastalerosion,flooding,rivercourses,forestmappingandmining)maybefoundhere;http://home.iitk.ac.in/~blohani/LiDAR_Tutorial/Airborne_AltimetricLidar_Tutorial.htm.
WecannowseejustafractionoftheassociatedscientificconceptsthatarepertinentinunderstandingtheuseofLiDARinunderstandinggeoscience:e.g.,radartechnologyvsLiDARtechnology,thescienceofplatetectonics,digitalelevationmodels,readingandunderstandingcomputervisualizations,modelingcomplexinter-relatedscientificprocesses,reasoningaboutimplicationsforhumanactivity,includingurbangrowth,andsoforth.Whichofthese(orotherrelatedconcepts)aremostpertinentforscientistsinacyberinfrastructureresearchcollaboratorywillbeanempiricalquestion.Howtoidentifythecentralconstructspertinenttoahigh-schoolscienceeducationwillprovideasignificantmeasurementchallenge,includinghowtodesignauthenticassessmentstomeasureunderstandingoftheseconcepts.Identifyingandmappingoutthecontentandcognitivedemandsofsuchmeasurementcouldbeamajorfocusofthedesignresearchwork.Ofparticularinterestwillbehowtoestablishcontent,construct,predictive,concurrentandotherformsofvalidityforthesemeasures.
Factorsconvergingtosupportthedevelopmentofcyberinfrastructure.
1. ExistingcomputingdatagridsintheUSandoverseas a. TheTeraGridproject(http://www.teragrid.org/about/)combinesthepowerofNCSA,
SDSC,ArgonneNationalLaboratory,CACR,PSC,ORNL,TACC,andvariousuniversity
an introduction to educational design research 87
partnersintegratedbytheGridInfrastructureGroupattheUniversityofChicago.Europeane-sciencelinksfacilitiesontheContinentwiththoseintheUK.SimilaractivitiesoccurinJapan.IndustrypartnersincludeIBM,Intel,Hewlett-PackardandOracle.
2. Theavailabilityofmassivedatastoragecapacityandspeed a. TheTeraGridcurrentlyoffersover100teraflopsofcomputingpower;andover3
petabytesofrotatingstorage3. Thedevelopmentofmiddlewareandsoftwaretogatherandanalyzestoreddata a. TheTeraGridsupportsdataanalysisandvisualizationproductioninterconnectedat
10-30gigabits/second.4. Theemergenceoflargeteamsofscientistsdedicatedtosolvingsharedscience problems(actingthroughscience“collaboratories”and“gateways”) a. Acollaboratory(Wulf,1989)is“morethananelaboratecollectionofinformationand
communicationstechnologies;itisanewnetworkedorganizationalformthatalsoincludessocialprocesses;collaborationtechniques;formalandinformalcommunication;andagreementonnorms,principles,values,andrules”(Cogburn,2003,p.86).Collaboratoriesexistinmanyareasofscience,includingbiology,chemistry,geoscienceandastronomy(e.g.,Chin&Lansing,2004;Olson,Teasley,Bietz,&Cogburn,2002).
b. Sciencegatewaysareweb-basedportalsorinterfacesforthestructuresanddataofthecyberinfrastructureinmanyscienceareas(foralistingof24gateways,seehttp://www.teragrid.org/programs/sci_gateways/).
5. Developmentsinscientificvisualization. Scientificvisualizationdrawsonhumanspatialandvisualprocessinginordertomodel
andanalyzecomputationallyintensethegraphicdisplayofcomplexdata(foracomprehensivereview,seeThomas&Cook,2005).Existingmethodsandmodelsforscientificvisualizationaresignificantlychallengedbycyberinfrastructure(e.g.,http://www.teragrid.org/userinfo/data/vis/vis_gallery.php;Chinetal.,2006).
6. Funding. Theestablishmentandfundingofnationalandinternationaleffortstocoordinateand
developtheinfrastructuretobetterservescienceand,morerecently,education(e.g.,theOfficeofCyberinfrastructure–NSF;CERN,Dutch(VL-e)andUKinitiatives).ThepromiseofcyberinfrastructureforeducationisthatthevastinvestmentbyUSagencies(upwardsof$250Moverthenext5years,alone)willprovidetest-bedsforexploration.
an introduction to educational design research 89
5. FormativeEvaluationinEducational DesignResearch Nienke Nieveen
Introduction
Inthischapterandinlinewiththegeneralintroductionofthisbook,wedefineeducationaldesignresearchas:the systematic study of analyzing, designing and evaluating educational interventions in order to solve complex educational problems for which no ready-made solutions are available and to gain insight in key design principles.Designresearchprojectsstriveaftertwotypesofmainresults.Thefirstaimcompriseshigh-qualityinterventions(suchasprograms,productsandprocesses)designedtosolvecomplexeducationalproblems.Thistypeofoutputputsforwardthepracticalrelevanceofdesignresearch.Itisforthatreasonthatdesignresearchisalsolabeledasbeinguse-inspired,appliedorientedand/orsociallyresponsibleresearch(vandenAkker,1999;Reeves,2000).Thesecondmainoutputofdesignresearchistheaccompanyingsetofwell-articulateddesignprinciples(Linn,Davis&Bell,2004;vandenAkker,1999)thatprovideinsightinthe:• purpose/functionoftheintervention;• keycharacteristicsoftheintervention(substantiveemphasis);• guidelinesfordesigningtheintervention(proceduralemphasis);• itsimplementationconditions;• theoreticalandempiricalarguments(proof)forthecharacteristicsandprocedural
guidelines.Thesecomprehensivedesignprinciplesserveseveralpurposesforavarietyoftargetgroups.Fromaresearchperspective,theseprinciplesshowthecontributionofdesignresearchtotheexistingknowledgebasewithinformationonhowtheinterventionworksinpractice,theeffectsofusingtheinterventionandexplanationoftheworkingmechanisms.Foreducationaldesigners,theseprinciplescarryrichinformationonhowtodesignsimilarinterventionsforsimilarsettings.Fromtheperspectiveoffutureusers,theprinciplesprovideinformationneededforselectingandapplyinginterventionsinthespecifictargetsituationandprovideinsightsintherequiredimplementationconditions.Finally,forpolicymakers,theseprinciplesassistinmakingresearch-baseddecisionsforsolvingcomplexeducationalproblems.
Inordertoreachbothtypesofoutput(highqualityinterventionsanddesignprinciples),designresearcherscarefullycombinedesignandresearchactivitiesresultinginaniterativedevelopmentapproach.Inthiscontributionwewillfirstexplorethisiterativenatureofdesignresearch(herelabeledwiththetermprototypingapproach)andthenelaborateon
an introduction to educational design research90
therolethatformativeevaluationplaysindesignresearchprojectsinordertooptimizeinterventionsanddesignprinciples.Thecontributionwillendwithsomeremarksontheroleofdesignresearchersconcerningformativeevaluationactivities.
Prototyping approach
Designresearchisbynaturehighlyiterative(Design-basedresearchcollective,2003;vandenAkker,1999).Eachiterationhelpstoimproveprototypesofbothendresultsofdesignresearchefforts:1.theeducationalinterventionunderdevelopment;and2.itsaccompanyingtentativedesignprinciples.Thissectiondiscussesbrieflythenotionofaniterativeorprototypingapproach.
Aprototypeisapreliminaryversionofthewholeorapartofaninterventionbeforefullcommitmentismadetoconstructandimplementthefinalproduct.Prototypesmaybeusedintwoways(cf.Smith,1991).Ontheonehand,aprototypemaybecontinuallyrefined(basedonformativeevaluationresultsandreflectionsofdevelopersontheprototype)andevolvetowardsafinaldeliverable.Thisrefiningapproachcanbereferredtowiththetermevolutionaryprototyping.Ontheotherhand,developerscandesignthrow-awayprototypes,suchasscenariosorpaper-basedmock-ups(Nieveen,1999).Ascenarioisanarrativedescriptionoftypicalandcriticalsituationsthatprospectiveusersparticipatein.Scenariosmaybeusedtomakethetentativedesignspecificationsmoreconcrete.Thismakesiteasiertocommunicatethepotentialsofasystemwiththetargetgroup.Apaper-basedmock-upcomprisesapileofpapersrepresentingallscreenswhichmayappearduringtheuseoftheintervention.Thiskindofprototypeisoftenusedinsoftwaredevelopmentprojects.Usersmay‘walkthrough’thescreenstogetanideaoftheintentionsofthesoftwareapplication.Paper-basedprototypesfocustheattentionoftheusermoreoncontentandoverallstructurethanonappearance.Afterbeingevaluated,athrow-awayprototypewillbediscardedanditsevaluationresultsaretakenintoaccountinthenextprototype.Thisprocesswillcontinueuntilalluncertaintiesarecoveredandthefinalproductorinterventioncanbedelivered.
Especiallyindesignresearchprojectsthataimatinnovativeandcomplexproducts,withfewexperiencesordesignprinciplesfromwhichtodraw,suchaprototypingapproach(eitherevolutionaryorthrow-away)isrecommendable.Somedesignresearchprojectscombinebothkindsofprototypes,forinstancebyfirstdesigningandevaluatingthrow-awayprototypesandthenshiftingtoanevolutionaryapproach(Nieveen,1999).Tomaketheprototypingapproachwiththrow-awayand/orevolutionaryprototypesfeasible,thenotionof‘thinkbig,butstartsmall’ishelpful.Byfirstdevelopingasmallpartoftheproposedintervention,onekeepsthedevelopmentprocessmanageableandonecan
an introduction to educational design research 91
learnfromfailuresandapplysuccesseswhendesigningthesubsequentparts.Inordertokeepanoverviewontheentiredevelopmentprocessitisoftenfunctionaltodecomposetheinterventionintoseveralcomponentsthatcouldbebuiltseparately.Educationalinterventionscanbedecomposedintoatleasttwokeyaspectswhichwillrequiremajorattentionduringthedesignprocess(cf.Nieveen,1999;Nieveen&vandenAkker,1999):• theconceptual framework oftheintervention,referringtoallnotionsthatare
underlyingtheintervention.Incaseofaneducationalintervention,itrefersforinstancetotheconceptualizationofall10curriculumcomponentspresentedinthecurricularspiderweb(vandenAkker,2003);
• thepresentation-modeoftheintervention,referringtotheformatthatassuresthattheinterventionisusableforitsusers.Toassistusersfindingthecontentoftheirpreference,allinterventions(paper-basedandcomputer-based)needasounduser-interfaceincludingconsistentlayoutandtransparantnavigation.
Thevariousconceptualandpresentationelementsmaybeindifferentstagesofdevelopmentineachprototype.However,towardsthefinaldeliverable,allelementsneedtobeconsistentwithoneanother.Forexample,inamathprojectaimedatsolvingproblemsoflow-achievingstudentswithmeasuringquantities,interventionsweredevelopedtohelpthesestudentstoacquiretherequiredmathematicalproblemsolvingskills.Thedesignresearchteamputmuchemphasisonthedevelopmentofinnovativelearningandteachingactivities(twocomponentsofthecurricularspiderweb/theconceptualframework)gearedtotheproblemsofthesestudents.Subsequently,thefirstversionofthelessonmaterialswasdesignedaccordingtothisspecificpedagogy.Duringtheformativeevaluationofthisfirstprototype,thedesignresearchteamwasespeciallyinterestedinthequalityofthenewlearningandteachingactivities(beingpartoftheconceptualframework)andlessinthelayout(beingpartofthepresentation-mode).However,towardstheendofthestudy,thelayoutofthematerialsgotspecificattentioninordertoimprovetheoverallpracticalityofthematerials.
Formative evaluation
Inaprototypingapproachempiricaldataareneededtogaininsightintothequalityofthetentativeinterventionanddesignprinciples.Forthatreason,formativeevaluationisacrucialfeatureofeachprototypingapproachandthusofeachdesignresearchproject.Itprovidesinsightinthepotentialsoftheinterventionanditskeycharacteristics.Resultsoftheformativeevaluationgivegroundforboth1.improvingtheprototypeoftheinterventiontowardsahigh-qualityfinaldeliverableand2.sharpeningtheunderlyingtentativedesignprinciplestowardsanelaboratedsetofdesignprinciples.Inthisway,eachprototypingcyclecontributestosuccessiveapproximationofbothoutputsofadesignresearchproject.IntheGenericDesignResearchModelofWademan(2005),seeFigure1,thisisnicelyillustratedintheprototypingandassessmentphase.
an introduction to educational design research92
Figure 1: Generic Design Research Model (Wademan, 2005)
Inthissection,theconceptofformativeevaluationwillbefurtherelaboratedandplacedinadesignresearchcontext.Asfarasthetermevaluationisconcerned,theJointCommitteeonStandardsforEducationalEvaluation(1994)usesthefollowingdefinition:“Evaluationisthesystematicassessmentoftheworthormeritofsomeobject.”Meritreferstotheobject’sinherent,intrinsicvalue,whileitsworthisdefinedasitscontextuallydetermined,place-boundvalue(Lincoln&Guba,1979).Scriven(1967)wasthefirstauthorwhomadethedistinctionbetweenformativeandsummativeevaluation.Formativeandsummativeevaluationsservedifferentfunctions.Thefunctionofformative evaluationis‘toimprove’.Itfocusesonuncoveringshortcomingsofanobjectduringitsdevelopmentprocesswiththepurposetogeneratesuggestionsforimprovingit.Thefunctionofsummative evaluationis‘toproof’.Asummativeevaluationiscarriedouttogainevidencefortheeffectivenessoftheinterventionandfindargumentsthatsupportthedecisiontocontinueorterminatetheproject.Summativeevaluationsarebeingcarriedoutwithoutthedirectintentiontoreveal
Practitioner and User Participation
Researchers
Other Sources
Collaboratives
Practitioners
ConsultExperts &
Practitioners
FocusedLiterature
Review
AnalyzePromisingExamples
AnalyzePracticalContext
Tentative Product
Approaches
Tentative Design
Principles
Reflection
FormativeEvaluation
PracticalProduct/Results
Contributionto
Theory
Refinement of Problem, Solution and Method
Refinement of Design Theory
Preliminary investigation of Problem, Context,
& Approaches
Problem in Context
Phases
Prototyping & Assessment of Preliminary Products & Theories
Identification of Tentative Products & Design Principes
Problem Identification Tentative Products& Theories
Problem Resolution & Advancing Theory
Successive Approximation of Theory
Successive Approximation of Product
Redesign & Refinementof Products & Theories
an introduction to educational design research 93
pointsofimprovement.However,itisnotalwayspossibletodrawasharplinebetweenformativeandsummativeevaluation.Theresultsofsummativeevaluationsareusuallytakenintoaccountwhiledevelopingasecondreleaseoftheproduct.
Basedoncomparingandsynthesizingdefinitionsofvariousscholarsinthefieldofformativeevaluation(cf.Brinkerhoff,Brethouwer,Hluchyj&Nowakowski,1983;Flagg,1990;Scriven,1967,Tessmer,1993)wedefineformativeevaluationinthecontextofdesignresearchas:a systematically performed activity (including research design, data collection, data analysis, reporting) aiming at quality improvement of a prototypical intervention and its accompanying design principles.Asstatedbefore,adesignresearchprojectusuallyneedsseveraliterationsbeforeanoptimalsolutionforthecomplexproblemcanbeenreached.Eachdesignresearchcycleoriterationconcentratesonspecificresearchquestionsandneedsanappropriateresearchdesign.Theremainderofthissectionwillelaborateonissuesrelatedtotheresearchdesignofformativeevaluationactivities.
FormulatingresearchquestionsThemainresearchquestionofaformativeevaluationisbuiltaroundthekindofvaluejudgmentthatisexpectedfromevaluatingtheprototypeandtwokeyattributesoftheprototypicalintervention:1.thestageofdevelopmentoftheprototype;and2.themainelementoftheprototypethattheevaluationwillfocuson.
Firstofall,itisnecessarytomakeclearthetypeofvaluejudgmentthattheevaluationneedstoresultin.Inthisrespect,wedistinguishfourqualitycriteriathatareapplicabletoawidearrayofeducationalinterventions(seeTable1).Attheendofadesignresearchproject,theinterventionshouldsufficeallofthesecriteria.However,usuallyeachiterationconcentratesononeortwoofthesecriteria.
an introduction to educational design research94
Criterion
Relevance (also referred to as content validity)
There is a need for the intervention and its design is based on state-of-the-art (scientific) knowledge.
Consistency (also referred to as construct validity)
The intervention is ‘logically’ designed.
Practicality ExpectedThe intervention is expected to be usable in the settings for which it has been designed and developed. ActualThe intervention is usable in the settings for which it has been designed and developed.
Effectiveness ExpectedUsing the intervention is expected to result in desired outcomes. ActualUsing the intervention results in desired outcomes.
Table 1: Criteria for high quality interventions
Itisimportanttopointheretothedistinctionbetweenexpectedandactualpracticalityandeffectiveness.Onlywhenthetargetusershavehadpracticalexperiencewithusingtheinterventiononewillbeabletogetdataontheactual practicalityoftheprototype.Similarly,onlywhentargetusershavehadtheopportunitytousetheinterventioninthetargetsetting,theevaluatorwillgetdataontheactual effectiveness.Inallotherinstances,suchasagroupdiscussionsbasedonthematerials,theresearcherwillonlygetdataontheexpected practicality and/or effectiveness.
Moreover,whenpreparingaformativeevaluationitisimportanttodescribetheboundariesoftheprototypethatwillbeevaluated.Inadesignresearchprojecta(throw-awayorevolutionary)prototypeisusuallyinoneofthefollowingdevelopmentstages:• Designspecifications:Afirstandgeneraldescriptionoftheinterventioninwhich
attentionispaidtoitssubstantiveparts.Thissketchhasbeenbasedonpreliminaryresearchactivities(includingproblemandcontextanalysisandliteraturereview).
• Globalintervention:Someorallcomponentsoftheinterventionaregivensomedetail.Thiscouldbetermedasahorizontalprototype.Itgivesanideaofhowtheinterventionwilleventuallyappear,howeveritcannotyetbeusedinpractice.Forexample,inthecaseofthedevelopmentofanewcurriculumatthisstagetheinterventioncouldtaketheformofatableofcontentswithabriefdescriptionofsub-componentsormodules.
• Partoftheinterventionindetail:Atthisstage,apartorcomponentoftheinterventionhasbeenelaboratedtoaconcretelevelforusebythetargetgroup.Thiscouldbecalledaverticalprototype.Onecanimaginevarioussub-stageswitheachofthemaddressingonlyaspecificpartofthetotalinterventionforuseinpractice.
an introduction to educational design research 95
• Completeintervention:Thetotalinterventionissufficientlydetailedthatitcouldbeusedintheintendeduser-setting.
Anotherissuethatneedstobeclearbeforestartingaformativeevaluationisthemainelementsoftheprototypethattheevaluationwillfocuson.Thesecanbealloroneelementsrelatedtotheconceptualframeworkandpresentation-modeoftheintervention.
Thethreecharacteristics(qualitycriteria,elementsandthestageofdevelopment)giveinputforthemainresearchquestions.Thesyntaxoftheseresearchquestionsis:‘Whatisthe[qualitycriteriuma,b,cand/ord]of[elementoftheconceptualframeworkorpresentationmode]oftheinterventionthatisin[developmentstagew,x,y,z].Instancesofthesequestionsare:• Whatistherelevancy[qualitycriterium]ofthecontent[conceptualelement]ofaquick
referencemanualforusingChinesecharactersthatisinaglobalstage[developmentstage]?
• Whatistheinternalconsistency[qualitycriterium]oftheattainmenttargets[conceptualelement]forscienceinuppersecondaryeducationofwhichthreeoutofsevendomainsareelaboratedindetail[developmentstage]?
• Whatisthepracticality[qualitycriterium]ofthelayout[elementofpresentationmode]oftheMathtextbookmodulesthatisinacompletedstage[developmentstage]?
SelectingappropriatemethodsDesignresearchersneedtoselectthoseformativeevaluationmethodsthatfittheresearchquestions.Buildingonearlierresearch(Nieveen,1997,1999),Table2providesanoverviewoftherelationshipbetweentheresearchquestions(withonthevericalaxisthequalitycriteriaconcerningtheelementsofaninterventionandonthehorizontalaxisthedevelopmentstages,seeprevioussection)andsuitableformativeevaluationmethods(indicatedinthecells).Herewedistinguishthefollowingmethods.PleasereferforanextensiveoverviewforinstancetoTessmer(1993)orBrinkerhoff,etal.(1983):• Screening:membersofthedesignresearchteamcheckthedesignwithsomechecklists
onimportantcharacteristicsofcomponentsoftheprototypicalintervention.• Expertappraisal:agroupofexperts(forinstance,subjectmatterexperts,instructional
designexperts,teachersreviewthematerials)reactsonaprototypeofanintervention,usuallyonthebasisofaguidelinewithcentralquestionsofthedesignresearchteam.Usuallythisisdonebyinterviewingtherespondents.
• Walkthrough:thedesignresearcherandoneorafewrepresentativesofthetargetgrouptogethergothroughthesetupoftheintervention.Usuallythisiscarriedoutinafacetofacesetting.
an introduction to educational design research96
• Micro-evaluation:asmallgroupoftargetusers(e.g.learnersorteachers)usespartsoftheinterventionoutsideitsnormalusersetting.Here,themainactivitiesoftheevaluatorareobservingandinterviewingtherespondents.
• Try-out:alimitednumberoftheusergroup(e.g.teachersandlearners)usesthematerialsinthedaytodayusersetting.Iftheevaluationfocusesonpracticalityoftheintervention,thefollowingevaluationactivitiesarecommon:observation,interviewing,requestinglogbooks,administeringquestionnaires;iftheevaluationhasitsfocalpointontheeffectivenessoftheintervention,evaluatorsmaydecidetorequestlearningreportsand/orgiveatest.
Summativeevaluationmethods,suchas(quasi-)experiments,surveysandaccompanyingcase-studies,followtheseformativeevaluationactivitiesassoonastheinterventionhasbecomefullygrownandhasbeenimplementedineducationalpractice(seeforinstanceRossi,Freeman&Lipsey,1999).
Table 2: Table for selecting formative evaluation methods
Wheninterventionsbecomemoredetailed,thefocusoftheformativeevaluationwillgraduallyshiftwithrespecttotheaforementionedqualitycriteria.Inanearlystage,themainfocuswillbeontherelevancyandconsistencyofaprototype.Assoonasaglobalinterventionhasbeendesigned,designresearchersalsowouldliketoassesstheexpectedpracticalityoftheintervention.Whentheinterventionisevenmoreelaborated,thenthefocuswillshifttowardstheactualpracticalityandeffectiveness.Intable2,itisindicatedingreythatwiththisshiftinfocusalsoother,moresuitable,evaluationmethodswillcomeintoplay.Moreover,eachdevelopmentstagemayconsistofseveralcyclesofanalysis,designandformativeevaluationbeforetheprototypewillgrowintoanextdevelopmentstage.
Table 6: Table for selecting formative evaluation methods
Design stage
Quality criterion
Designspecifications
Global design Partly detailedintervention
Completeintervention
Implementedintervention
Relevance - Screening- Expert appraisal
- Screening- Expert appraisal
- Screening- Expert appraisal
- Screening- Expert appraisal
Consistency - Screening- Expert appraisal
- Screening- Expert appraisal
- Screening- Expert appraisal
- Screening- Expert appraisal
Practicality expected - Screening- Expert appraisal
- Screening- Expert appraisal
- Expert appraisal- Walkthrough
- Expert appraisal- Walkthrough
actual - Micro-evaluation - Micro-evaluation- Try-out
Survey, (Quasi)experiment, Case-study
Effective-ness
expected - Screening- Focus group
- Screening- Focus group
- Expert appraisal - Expert appraisal
actual - Micro-evaluation - Micro-evaluation- Try-out
Survey, (Quasi)experiment, Case-study
an introduction to educational design research 97
Sampling-selectingrespondentsTobeabletoanswertheresearchquestionswiththechosenevaluationmethods,therequiredtypeandnumberofrespondentsneedtobediscussed.Thetypeandsamplesizedependontheresearchquestions.Withrespecttothetypeofrespondents,oneneedstoselectthoserespondentsthatcanhelpansweringtheresearchquestions.Forinstance,incasedesignresearcherswanttogaininsightintherelevancyofthedesignfromasubjectmatterperspectivetheywillselectanumberofexpertsinthatspecificdomaintodoanexpertappraisal.Incaseinsightsareneededintheactualpracticalityofalearningpackageforlearnersbyperformingamicro-evaluation,studentsneedtobesampledwhowillhavetoworkwiththeintervention.Moreover,themainpurposeoftheevaluationalsoinfluencesthesamplesize.Incaseofaformativeevaluationduringearlystagesoftheproject,themainpurposeistolocateshortcomingsintheinterventionandtogeneratesuggestionsforimprovement(seealsodefinitionofformativeevaluation),thenumberofrespondentsislesscritical:aremarkofonlyonerespondentcouldbehighlyvaluablebecauseofitssalience.Smallsamplesofrespondentsareusuallysufficientiftheyarecarefullyselected.Samplesareusuallydeliberatelychosen(alsoreferredtoaspurposivesamplingwheresubjectsareselectedbecauseofsomecharacteristic),insuchawaythatthecommentsandreactionswillbeasinformation-richaspossible.Thismeansthatforinstancefororganizingamicro-evaluationinordertogaininsightsintothepracticalityofaprototypeofsomelearnermaterials,nexttohigh-achievingstudentsalsoagroupoflow-achievingandagroupofavaragestudentsneedtobeselected.Triangulationisimportanthereinordertoenhancethereliabilityandinternalvalidityofthefindings(cf.Miles&Huberman,1994).Onecouldtriangulatebyusingdifferenttypeofpersons,differenttimes,differentplaces.Theeffectivenessoftriangulationrestsonthepremisethattheweaknessesineachsingledatasourcewillbecompensatedbythecounterbalancingstrengthofanother.Incaseofasummativeevaluation,whenthemainpurposeofanevaluationisto prooftheactualpracticalityandeffectiveness,(quasi-)experimentalresearchdesignswithexperimentalandcontrolsettingsarerequiredwithlargesamplesizes.FormoreinformationonsamplingseeforinstanceCreswell(2008),Denscomb(2007)andMills,Gay,AirasianandAirasian(2008).
Wheninvitingrespondentsforaformativeevaluationitisnecessarytoilluminatetheirrole.Theycouldfulfilltheroleoflearner,criticand/orrevisor(Weston,McAlpine&Bordonaro,1995).Respondentswithalearnerrolearenotspecificallyexpertinthesubjectmatterwhichiscoveredbythematerials.Onecouldthinkofstudentswholearnanewsubject;butalsoteacherswhohavenottaughtinacertainmannerbefore.Inmanycasesexpertsrepresentthiscategoryaswell.Forinstance,educationaltechnologyexpertsdonotalwayshaveexpertiseinthesubjectmatterdomainoftheeducationalintervention.Theywilltaketheroleofalearnerfirst,beforetheywillgivecommentsonmattersrelatedto
an introduction to educational design research98
educationaltechnology(inwhichtheyareexperts).Criticsarerespondentswhoareaskedtocommentonthematerialsfromtheperspectiveoftheirexpertise.Thisgroupconsists,forinstance,ofsubjectmatterexpertsandteacherswhoareinvitedtomakestatementsaboutthedifficultyorreadabilityoflearnermaterials.Revisorswillnotonlygivecommentsonthematerials(likecriticsdo),buttheywillalsoprovidesuggestionsforimprovements.Forinstance,asubjectmatterexpertmayindicatewhattypeof‘state-of-the-artknowledge’ismissinginthelearnermaterialsandwherethisknowledgecouldbefound.Itisimportanttonotethatindividualsmayplayseveralrolessimultaneouslyduringtheformativeevaluation.Thenextsectionwillelaborateontheroleoftheresearchersduringaformativeevaluation.
Researchers’ role during formative evaluation
Sinceadesignresearchprojectcomesintoplaywhenaneedarisestosolveacomplexeducationalproblemforwhichnoready-madesolutionsareavailable,oftentimesamulti-disciplinedteamisbroughttogethertoworkonit.Suchteamsusuallycompriseofexpertsindomainsthatweredistinguishedwhendecomposingtheintervention(e.g.fromaconceptualpointofview:subjectmatterexperts,pedagogicalexperts,instructionaldesigners;fromapresentation-modepointofview:user-interfacedesigners)aswellasmembersofthetargetgroup.Monk,Wright,HaberandDavenport(1993,p.5)stressthat“Itrequiresaccesstopeopletypicalofthosewhowillactuallyusethesystem,nottheirrepresentativesormanagement.”Involvingfutureusersinadesignresearchteamhasseveraladvantages(cf.Moonen,1996;Shneiderman,1992):moreaccurateinformationaboutcomplexityoftheproblemathand,moreintensivediscussionsabouttherequirementsoftheintervention,increaseofusercommitmentandownershipofthefinaldeliverable,increaseofinsightsintotherequirementsofthecontextinwhichtheinterventionwillbeused,andstimulationoftheprofessionaldevelopmentofallparticipants.
Oneofthekeyresponsibilitiesofthedesignresearchteamistoworkontheformativeevaluationoftheprototypes.Forreasonsofscientificrigor,itisoftenrecommendedtolookforexternalevaluators.However,certainlyintheearlystagesofadesignresearchprojectitseemslegitimateorevenadvisablethatdesignresearchersthemselvescarryouttheformativeevaluationoftheprototype.Engaginginformativeevaluationactivitiestendtoleadtoimportantlearningexperiencesofthedesignresearchers.Theywillexperiencethemselvestheproblemsthatoccurandhearoutoffirsthandthesuggestionsforimprovementthatrespondentscomeupwithduringtheiruseofaprototype(forexample,byobservingorinterviewingteachersorstudents).Thisusuallyhasstrongerandmore
an introduction to educational design research 99
directimpactontheirthinkinganddesignactivities,comparedtocaseswhereexternalevaluatorsreporttheresultstothedevelopers.Ofcourse,designresearchersneedtobeawareofseveralpitfallswhentheyareinvolvedintheformativeevaluationoftheinterventiontheyarealsodesigning(cf.McKenney,Nieveen&vandenAkker,2006).Theymayeasilybecometoo‘attached’totheirprototypewhichcouldleadtoalessobjectiveviewtowardproblemsandcommentsfromtherespondents.Inthisrespect,Scriven(1991)warnsofa(co-)authorshipbias.Moreover,respondentscouldbebiasedduringtheevaluation,aswell.Forinstance,iftheyknowhowmucheffortthedesignresearchteamhasputintotheprototype,theymayhesitatetobefullycriticalofit.Toovercomethesebiases,itseemsessentialtoincludeformativeevaluationearlyoninthedevelopmentprocessandtoapplytriangulationofdatasources,methods(observation,interview,questionnaires,etc.),evaluators(differentevaluators)andtheories(differentconceptualframeworks).
Closing remark
Thischapterfocusesontheiterativenatureofeducationaldesignresearch.Eachiterationorcycleofanalysis,designandformativeevaluationgivesthedesignresearchteamfirmergroundandargumentsfortheinterventiontheteamisworkingoninordertosolveacomplexeducationalproblem.Theempiricaldatatheteamcollectsduringaformativeevaluationwillnotonlyprovidesuggestionsforimprovingtheintervention,butwillalsoassistinsharpeningtheaccompanyingdesignprinciples.Proceedingthroughseveraloftheseiterationswillendinafinalstageofthescientificcycleinwhichclaimsofcausalitycanbestudiedinsummativeevaluationsettings(cf.Nieveen,McKenney&vandenAkker,2006).Inthiscontributionweconcentratedontheresearchdesignforeachformativeevaluationperformedwithinsuchaniterativeorprototypingapproach.Weelaboratedontheresearchquestions,selectionofappropriatemethodsandrespondents.Weareawarethatthereismuchmoretosayaboutformativeevaluationingeneral,andintegratedindesignresearchprojectsinparticular.Forinstance,wecouldhavepaidattentiontoevaluationinstruments,datacollection,dataanalysisandreporting.Severalhelpfulbooksandarticlesareavailabletoassistinsystematicallyconductingformativeevaluationineducation(cf.Brinkerhoff,etal.,1983;Flagg,1990;Tessmer,1993).Althoughthesesourceswerenotwrittenwiththespecificneedsandwishesofdesignresearchersinmind,theycanprovideampleinspiration.
an introduction to educational design research100
References
Brinkerhoff,R.O.,Brethouwer,D.M.,Hluchyj,T.,&Nowakowski,J.R.(1983).Program evaluation: A practitioner’s guide for trainers and eduacators. Boston:Kluwer-Nijhoff.
Creswell,J.W.(2008).Educational research: planning, conducting and evaluating quantitative and qualitative research.UpperSaddleRiver,NJ:Pearson.
Denscomb,M.(2007). The good research guide for small-scale social research projects. Maidenhead(UK):OpenUniversityPress.
Design-basedresearchcollective.(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher, 32(1),5-8.
Flagg,B.N.(1990).Formative evaluation for educational technologies.Hillsdale,NJ:LawrenceErlbaumAssociates.
JointCommitteeonStandardsforEducationalEvaluation(1994).The program evaluation standards. ThousandOaks,CA:SagePublications.
Lincoln,Y.&Guba,E.G.(1979).Thedistinctionbetweenmeritandworthinevaluation.Paper presented at the Annual Meeting of the Evaluation Network(5th,Cincinnati,OH,September24-26,1979)retrievedOctober20,2008,http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/32/b5/c8.pdf
Linn,M.,Davis,E.A.,&Bell,P.(2004).Internet environments for science education.Hillsdale:LawrenceErlbaum.
McKenney,S.Nieveen,N.&Akker,J.vanden(2006).Designresearchfromacurriculumperspective.InJ.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.),Educational design research(pp.67-90).London:Routledge.
Miles,M.B.,&Huberman,A.M.(1994).Qualitative data analysis: An expanded source book.London:Sage.
Mills,G.,Gay,L.R.,Airasian,P.&Airasian,P.W.(2008).Educational research: competencies for analysis and applications.UpperSaddleRiver,NJ:PrenticeHall.
Monk,A.,Wright,P.,Haber,J.,&Davenport,L.(1993).Improving your human-computerinterface: A practical technique.NewYork:PrenticeHall.
Moonen,J.(1996).Prototypingasadesignmethod.InTj.Plomp,&D.P.Ely(Eds.),International encyclopedia of educational technology(pp.186-190).Oxford:Pergamon.
Nieveen,N.M.(1997).Computer support for curriculum developers: A study on the potential of computer support in the domain of formative curriculum development,Doctoraldissertation.Enschede:UniversityofTwente.
Nieveen,N.(1999).Prototypingtoreachproductquality.InJ.vandenAkker,R.Branch,K.Gustafson,N.Nieveen,andT.Plomp(Eds.),Design approaches and tools in education and training(pp.125–36).Dordrecht:KluwerAcademicPublishers.
Nieveen,N.M.,&Akker,J.J.H.vanden(1999).Exploringthepotentialofacomputertoolforinstructionaldevelopers.Educational technology research & development, 47(3),77-98.
an introduction to educational design research 101
Nieveen,N.,McKenney,S.,&Akker,J.vanden(2006).Educationaldesignresearch:Thevalueofvariety.InJ.vandenAkker,K.Gravemeijer,S.McKenney,andN.Nieveen(Eds.),Educational design research(pp.151-158).London:Routledge.
Reeves,T.(2000).Enhancingtheworthofinstructionaltechnologyresearchthrough“designexperiments”andotherdevelopmentalstrategies.RetrievedOct.20,2006fromhttp://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf
Rossi,P.H.,Freeman,H.E.&Lipsey,M.W.(1999).Evaluation: A systematic approach.ThousandOaks,CA:Sage.
Scriven,M.(1967).Themethodologyofevaluation.InR.W.Tyler,R.M.Gagné,&M.Scriven(Eds.),Perspectives of curriculum evaluation.AERAMonographseriesoncurriculumevaluation.nr.1.Chicago,MI:RandMcNally.
Scriven,M.(1991).Beyondformativeandsummativeevaluation.InM.W.McLaughlin,&D.C.Phillips(Eds.),Evaluation and education: At quarter century(pp.19-64).Chicago:UniversityofChicago.
Shneiderman,B.(1992).Designing the user interface: Strategies for effective human-computer interaction.Reading,MA:Addison-Wesley.
Smith,M.F.(1991).Software prototyping: Adoption, practice and management.London:McGraw-Hill.
Tessmer,M.(1993).Planning and conducting formative evaluations: Improving the quality of education and training.London:KoganPage.
VandenAkker,J.(1999).Principlesandmethodsofdevelopmentresearch.InJ.vandenAkker,R.Branch,K.Gustafson,N.Nieveen,andT.Plomp(Eds.),Design approaches and tools in education and training (pp.1–15).Dordrecht:KluwerAcademicPublishers.
VandenAkker,J.(2003).Curriculumperspectives:Anintroduction.InJ.vandenAkker, W.Kuiper&U.Hameyer(Eds.),Curriculum landscapes and trends(pp.1-10).Dordrecht:
KluwerAcademicPublishers.Wademan,M.(2005).Utilizing development research to guide people capability maturity
model adoption considerations. Unpublisheddoctoraldissertation. Syracuse:SyracuseUniversity.Weston,C.,McAlpine,L.,&Bordonaro,T.(1995).Amodelforunderstandingformative
evaluationininstructionaldesign.Educational training research & technology, 43(3),29-48.
an introduction to educational design research 103
6. ReferencesandSourcesonEducational DesignResearch Tjeerd Plomp and Nienke Nieveen
Introduction
Thisbibliographyhasbeencompiledtosupportresearchersandgraduatestudentsingettingaccesstokeypublicationsondesignresearch.Wedonotclaimthattheselectionofsourcesincludedinthischapteriscompleteandexhaustive–it is coloured by our background and bias as well as our knowledge and familiarity with publications.Importantcriteriaforustoincludetitlesinthisbibliographyare(i)provenusefulnessofsourcesforourownwork,and(ii)representingimportantperspectivesandgroupsthatare(orhavebeen)activelyworkinginthisdomain.Inthefirstsectionwepresentjustanoverviewofrelevantsourcesavailable.Thisisfollowedbyasectioninwhichwepresentthestructureandcontentoftheexcellentwebsite‘Design-basedResearchEPSS’(http://projects.coe.uga.edu/dbr/index.htm)–createdbyInstructionalTechnologyPh.D.studentsatTheUniversityofGeorgia(lastupdateNovember2006).Giventhequalityandcompletenessofthiswebsite(atleasttillNovember2006),wedecidedtointroduceitinaseparatesectionandintheothersectionsofthischapterwewillrefertopartsofthiswebsite,butalsointroduceanumberofothersources.Inthefollowingtwosectionswepointthereadertoselectedjournalarticlesandbookchaptersontheconceptandmethodologyofdesignresearchandondesignresearchindomainssuchascurriculum,instructionaltechnology,andthelearningofreadingandwriting,mathematicsandscience.Inthefinalsection,welisttheURLsofanumberofdoctoralthesesthathavebeendefendedinTheNetherlandsutilizingdesignresearchasaresearchapproach.Asstated,ourselectioniscolouredbyourbiasandexperience,butallthesepublicationsrefertoawiderangeofwritingsondesignresearchandwetrustthattheythereforeserveasausefulintroductiontothereader.
Overview of sources
Thissectionpresentstitlesandreferencestovariousspecialissuesofjournalsandbooksthathavebeenpublishedaboutdesign(-based)research.Besidesanumberofwebsiteswillbelisted.
an introduction to educational design research104
Specialissuesofjournals
• EducationalPsychologist,39(4),2004 Specialissue‘Design-basedresearchmethodsforstudyinglearningincontext’,edited
byW.Sandoval&P.Bell,including: - Sandoval,W.A.,&Bell,P.L.(2004).Design-BasedResearchMethodsForStudying
LearningInContext:Introduction.Educational Psychologist, 39(4),199-201. - Hoadley,C.(2004).Methodologicalalignmentindesign-basedresearch.Educational
Psychologist, 39(4),203-212. - Sandoval,W.A.(2004).Developinglearningtheorybyrefiningconjecturesembodied
ineducationaldesigns.Educational Psychologist, 39(4),213-223. - Tabak,I.(2004).ReconstructingContext:NegotiatingtheTensionbetween
ExogenousandEndogenousEducationalDesign.Educational Psychologist, 39(4),225-233.
- Joseph,D.(2004).ThePracticeofDesign-BasedResearch:UncoveringtheInterplayBetweenDesign,Research,andtheReal-WorldContext.Educational Psychologist, 39(4),235-242.
- Bell,P.L.(2004).Onthetheoreticalbreadthofdesign-basedresearchineducation.Educational Psychologist, 39(4),243-253.
- AlsoincludescommentarybyAngelaO’Donnell.
• EducationalResearcher32(1),January/February2003 SpecialissuepreparedbyA.E.Kelly,including: - Kelly,A.E.(2003).ThemeIssue:TheRoleofDesigninEducationalResearch.
Educational Researcher, 32,3-4. - TheDesign-BasedResearchCollectiveDesign-BasedResearch(2003).AnEmerging
ParadigmforEducationalInquiry.Educational Researcher, 32,5-8. - Cobb,P.,Confrey,J.,diSessa,A.,Lehrer,R.&Schauble,L.(2003).DesignExperimentsin
EducationalResearch.Educational Researcher, 32,9-13. - McCandliss,B.D.,Kalchman,M.&Bryant,P.(2003).DesignExperimentsand
LaboratoryApproachestoLearning:StepsTowardCollaborativeExchange.Educational Researcher, 32,14-16.
- Lobato,J.(2003).HowDesignExperimentsCanInformaRethinkingofTransferandViceVersa. Educational Researcher, 32,17-20.
- Bannan-Ritland,B.(2003).TheRoleofDesigninResearch:TheIntegrativeLearningDesignFrameworkEducational Researcher, 32,21-24.
- Shavelson,R.J.,Phillips,D.C.,Towne,L.,&Feuer,M.J.(2003).OntheScienceofEducationDesignStudies.Educational Researcher, 32,25-28.
an introduction to educational design research 105
- Sloane,F.C&Gorard,S.(2003).ExploringModelingAspectsofDesignExperiments.Educational Researcher, 32,29-31.
- Zaritsky,R.,Kelly,A.E.,Flowers,W.,Rogers,E.,&O’Neill,P.(2003).ClinicalDesignSciences:AViewFromSisterDesignEfforts.Educational Researcher, 32,32-34.
• EducationalTechnology,45(1),2005 SpecialissuepreparedbyC.Dede,including: - Dede,C.(2005).Whydesign-basedresearchisbothimportantanddifficult.
Educational Technology, 45(1),5-8. - Squire,K.D.(2005).Resuscitatingresearchineducationaltechnology:Usinggame-
basedlearningresearchasalensforlookingatdesign-basedresearch.Educational Technology, 45(1),8-14.
- Barab,S.A.,Arici,A.,&Jackson,C.(2005).Eatyourvegetablesanddoyourhomework:Adesign-basedinvestigationofenjoymentandmeaninginlearning.Educational Technology, 45(1),15–21.
- Nelson,B.,Ketelhut,D.J.,Clarke,J.,Bowman,C.,&Dede,C.(2005).Design-basedresearchstrategiesfordevelopingascientificinquirycurriculuminamultiuservirtualenvironment.Educational Technology, 45(1),21–28.
- Kafai,Y.B.(2005).Theclassroomas“livinglaboratory”:Design-basedresearchforunderstanding,comparing,andevaluatinglearningsciencethroughdesign.Educational Technology, 45(1),28–34.
- Hay,K.E.,Kim,B.,&Roy,T.C.(2005).Design-basedresearch:Morethanformativeassessment?AnaccountoftheVirtualSolarSystemProject.Educational Technology, 45(1),34-41.
- Hoadley,C.(2005).Design-basedresearchmethodsandtheorybuilding:AcasestudyofresearchwithSpeakEasy.Educational Technology, 45(1),42-47.
- Reeves,T.C.(2005).Design-basedresearchineducationaltechnology:Progressmade,challengesremain.Educational Technology, 45(1),48-52
• JournaloftheLearningSciences,13(1),2004 Specialissue,including: - Barab,S.,&Squire,K.(2004).Design-BasedResearch:PuttingaStakeintheGround.
Journal of the Learning Sciences, 13(1),1-14. - Collins,A.,Joseph,D.,&Bielaczyc,K.(2004).DesignResearch:Theoreticaland
MethodologicalIssues.Journal of the Learning Sciences, 13(1),15-42. - Fishman,B.,Marx,R.W.,Blumenfeld,P.,Krajcik,J.,&Soloway,E.(2004).Creatinga
FrameworkforResearchonSystemicTechnologyInnovations.Journal of the Learning Sciences, 13(1),43-76.
an introduction to educational design research106
- diSessa,A.A.,&Cobb,P.(2004).OntologicalInnovationandtheRoleofTheoryinDesignExperiments.Journal of the Learning Sciences, 13(1),77-103
- Dede,C.(2004).IfDesign-BasedResearchistheAnswer,WhatistheQuestion?ACommentaryonCollins,Joseph,andBielaczyc;diSessaandCobb;andFishman,Marx,Blumenthal,Krajcik,andSolowayintheJLSSpecialIssueonDesign-BasedResearch.Journal of the Learning Sciences, 13(1),105-114.
- Kelly,A.(2004).DesignResearchinEducation:Yes,butisitMethodological?Journal of the Learning Sciences, 13(1),115-128.
Books• VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational
design research.London:Routledge.ISBN10:0-415-39635-2(pbk)(163pages) Availableathttp://www.taylorandfrancis.co.uk/shopping_cart/products/product_
detail.asp?sku=&ppid=118302&isbn=9780415396356 ThisbookcomprisesthepaperspresentedataseminarorganizedbytheNetherlands
OrganizationforScientificResearch,inparticularbytheProgramCouncilforEducationalResearch.Theseminar,conductedinDecember2003,hasbeenameetingplaceofdesignresearchersfromtheUSAandTheNetherlands.Thebookreflectsthevariousanglesfromwhichresearchersinthedomainsofcurriculum,instructionaltechnologyand(mathematicsandscience)educationaddresstheneedtodevelopresearchbasedsolutions(interventions)toproblemsforwhichnoguidelinestosolutionsareavailable.Thebookillustratesthatauthorswithvariousbackgroundshaveclearlyacommongroundwhenreflectingondesignresearchasaresearchapproach.
Thebookhasfourparts: Part1.Whatandwhy 1. IntroducingEducationalDesignResearch-JanvandenAkker,KoenoGravemeijer,
SusanMcKenney,NienkeNieveen 2. TowardProductiveDesignStudies-DeckerWalker Part2.Examplesfromthefield 3. DesignresearchfromtheLearningDesignPerspective-KoenoGravemeijer,Paul
Cobb 4. DesignResearchfromtheTechnologyPerspective-ThomasReeves 5. DesignResearchfromaCurriculumPerspective-SusanMcKenney,NienkeNieveen,
JanvandenAkker
an introduction to educational design research 107
Part3.Quality 6. AssessingtheQualityofDesignResearchProposals:SomePhilosophicalPerspectives
-D.C.Phillips 7. BalancingInnovationandRisk:AssessingDesignResearchProposals-DanielC.
Edelson 8. QualityCriteriaforDesignResearch:EvidenceandCommitments-AnthonyE.Kelly Part4.Movingahead 9. FromDesignResearchtoLarge-ScaleImpact:EngineeringResearchinEducation-
HughBurkhardt 10. EducationalDesignResearch:TheValueofVariety-NienkeNieveen,SusanMcKenny,
JanvandenAkker
• Kelly,A.E.,Lesh,R.A.&Baek,J.Y.(Eds).(2008).Handbook of Design Research Methods in Education Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching.NewYork:LawrenceErlbaumAssociates.ISBN:978-0-8058-6059-7(pbk)(560pages)
Availableathttp://www.routledgeeducation.com/books/Handbook-of-Design-Research-Methods-in-Education-isbn9780805860597
Theannouncementofthebookstatesthatthehandbookpresentsthelatestthinkingandcurrentexamplesofdesignresearchineducation.Design-basedresearchinvolvesintroducinginnovationsintoreal-worldpractices(asopposedtoconstrainedlaboratorycontexts)andexaminingtheimpactofthosedesignsonthelearningprocess.Designedprototypeapplications(e.g.,instructionalmethods,softwareormaterials)andtheresearchfindingsarethencycledbackintothenextiterationofthedesigninnovationinordertobuildevidenceoftheparticulartheoriesbeingresearched,andtopositivelyimpactpracticeandthediffusionoftheinnovation.
TheHandbook of Design Research Methods in Educationismeanttofillaneedinhowtoconductdesignresearchbythosedoingsorightnow.Thechaptersrepresentabroadarrayofinterpretationsandexamplesofhowtoday’sdesignresearchersconceptualizethisemergentmethodologyacrossareasasdiverseaseducationalleadership,diffusionofinnovations,complexitytheory,andcurriculumresearch.
Thehandbookhaseightsections: - Designresearchanditsargumentativegrammar - Modelingstudentlearningduringdesignresearch - Modelingteacherlearningusingdesignresearch - Modelingstakeholderscommitmentsusingdesignresearch - Reflectingondesignresearchattheprojectlevel
an introduction to educational design research108
- Reflectingondesignresearchattheprogramlevel - Extendingdesignresearchmethodologically - Trackingthediffusionofdesignresearch.
• Reinking,D.&Bradley,B.A.(2008).On Formative and Design Experiments: Approaches to Language and Literacy Research.NewYork&London:TeachersCollege,ColumbiaUniversity.ISBN:978-0-8077-4841-1(pbk)(134pages)
Thisbookletprovidesaniceintroductionintoformativeanddesignexperiments,atermsynonymousforwhatwecalldesignresearchandothersdesign-basedresearch.Itprovidesathorough,butpracticalandusefuloverviewofdesignresearchaddressingthefollowingquestions:
- Whatareformativeanddesignexperiments(Ch1)? - Whatarethemethodsofformativeanddesignexperiments(Ch2)? - Whataresomegoodexamplesofformativeanddesignexperiments(Ch3)? - Isthereaformativeordesignexperimentinyourfuture(Ch4)?
• Richey,R.&Klein,J.D.(2007).Design and development research: methods, strategies, and issues.London:Routledge.ISBN080585732X,9780805857320(180pages)
Thisvolumediscussesmethodsandstrategiesappropriateforconductingdesignanddevelopmentresearch.Richwithexamplesandexplanations,thebookdescribesactualstrategiesthatresearchershaveusedtoconducttwomajortypesofdesignanddevelopmentresearch:1)productandtoolresearchand2)modelresearch.Commonchallengesconfrontedbyresearchersinthefieldwhenplanningandconductingastudyareexploredandproceduralexplanationsaresupportedbyawidevarietyofexamplestakenfromcurrentliterature.
Websites• http://projects.coe.uga.edu/dbr/index.htm(lastupdateNovember2006): titled‘Design-basedResearchEPSS’–createdbyInstructionalTechnologyPh.D.students
atTheUniversityofGeorgiaundersupervisionofTomReeves(comprehensivetilllastupdateofNovember2006).Thiswebsiteissummarizedinthenextsection.
• http://cider.athabascau.ca/CIDERSIGs/DesignBasedSIG/dbrreferences(lastupdateearly2005):
ThisbibliographyisdrawnupbyTerryAndersonoftheUnivesityofAthabasca(Edmonton,Alberta,Canada).Andersoncallsitasnapshotofmostcurrent(early2005)literaturerelatedtodiscussion,explorationandexamplesofdesign-basedresearch.ThereferencesarepresentedwithURLs(ifavailable)alongwithabstractsandoccasionallyquotationsorannotationsbyAnderson.HasmuchoverlapwiththeUniversityofGeorgiawebsite.
an introduction to educational design research 109
• http://www.designbasedresearch.org/index.html(lastupdatenotclear,butnoreferenceslaterthan2004)
ThisisthewebsiteoftheDesign-BasedResearchCollective,asmallgroupofresearcherswhoengageindesign-basedresearch,oftenintechnologyenhancedlearningenvironments.Itcontainsreferencesofanumberofpublications,aswellasanumberoflinkstorelevantrelatedwebsites.OtherpublicationsApartfromthesourcesmentionedabove,manyarticlesandbookchaptershavebeenpublisheddealingwithconceptualand/ormethodologicalaspectsofdesignresearch,orreportingaboutdesignresearchprojects.Manyofthesereferences(plusabstracts)canbefoundonthewebsitesmentionedinthissection,butwehaveselectedanumberwhicharesummarizedinthefinalsectionofthischapter.
UGA Website ‘Design-based Research EPSS’, November 2006
TheURLoftheUGAwebsiteElectronicPerformanceSupportSystem(EPSS)is:http://projects.coe.uga.edu/dbr/index.htm.Thewebsite(November2006),createdbyInstructional Technology Ph.D. students at The University of Georgia(UGA),supervisedbyTomReeves1,hasthreeparts:1. PEERTutorial2. Webliography3. ExpertInterviews
1.PEERTutorialThisusefultutorialiscomposedoffourprimarysections:(i) tutorialsurvey,(ii) explanation,(iii)enactment,and(iv)reflection.Parts(ii)and(iii)areusefulandinformativetobecomefamiliarwithdesignresearchandhowtogetstarted.(ii)Explanation:Thepurposeoftheexplanationpartofthetutorialistoprovidetheuserwithfundamentalknowledgeandinsightaboutdesignresearchcomposedoffivesections:• WhatisDesign-basedResearch(DBR)?• HowdoesDBRdifferfromotherapproaches?• HowdidDBRgetstarted?• WhatarethebenefitsofDBR?
1) TheauthorsofthiswebsiteusethetermDesignBasedResearch(DBR),whilstweusethroughoutthisbookletthe term‘DesignResearch’:bothtermsshouldbeseenassynonymous.
an introduction to educational design research110
• Whataresomecriticalperspective?Eachsectionisconciseandprovidescoreinformationwithamplereferencestoliterature.(iii)Enactment:Thispartconsistsofthreesections:- HowdoIgetstartedwithDesign-BasedResearch(DBR)?- SomeexamplesofDBR- WhatarethechallengesofdoingDBR?EachsectiondiscussesanumberofrelevanttopicsforthosewhowanttoconductDesignResearch(DR)orDesign-BasedResearch.Asstated,thisisausefultutorial.Butkeepinmindthatauthorsmaydifferinemphasis,approachand/oruseoftermsandconcepts.Butwhenyouareanopen-minded,criticalreaderyouwillfindmanyusefulideasandsuggestionsinthistutorial.
2.WebliographyThepurposeofthispartofthewebsiteis–accordingtoitscreators–toprovidevarioustypesofresourcesthatmayprovideinterestedresearchersabeginningpointforinvestigatingandpursuingthetopicofdesign-basedresearch.Warning:Asthewebsitehasbeenpreparedin2006,itmaybepossiblethatsomeoftheURLsreferredtoarenolongeraccessibleoractive.Thewebliographyhasthefollowingsections:(i) GlossaryAlimitednumberofkeyconceptsaredefined,themostimportantbeingdesign-basedresearch.(ii)PrintedresourcesOnebookislisted(VandenAkker,etal.,2006)andquiteanumberofjournalarticles(andtheirabstracts),dividedintomethodologicalarticlesandresearchexamples.Thisisausefullistofpublicationsup‘tillNovember2006,andespeciallythosepublishedinNorthAmerica.(iii)OnlineresourcesThissectionhasanumberofsub-sections:anumberofusefulwebsites,twoonlinejournals(toolimitedtobereallyuseful),andtheURLsofanumberofusefularticles.(iv)OrganizationsTwoorganizationsarementioned,viz.Design-basedResearchCollective(DBRC)andtheDesign-basedResearchSIGoftheCanadianInstituteofDistanceEducationResearch,butthewebsitesofbothorganizationsseemnottobeuptodate.Nevertheless,thewebsiteofDBRC(http://www.designbasedresearch.org/index.html)givesusefulreferencestotwospecialissuesofjournals:
an introduction to educational design research 111
• Kelly,E.A.(Ed.).(2003).Theroleofdesignineducationalresearch[specialissue].Educational Researcher, 32 (1).• Sandoval,W.&Bell,P.(Eds.).(2004).Design-basedresearchmethodsforstudying
learningincontext[SpecialIssue].Educational Psychologist, 39(4)2.
(v)Expertsindesign-basedresearchContainsshortbiographicalnotesandapictureofanumberofexpertsindesignresearch(amongstthemallexceptthefirstauthorofthisbooklet).(vi)FrequentlyAskedQuestions(FAQ):Threequestionsareaddressed,vizwhatisdesign-basedresearch(DBR),howtobeginwithDBR,andhowdoesDBRdifferfromotherresearchmethodologies.
3.ExpertinterviewsThispartofthewebsitecontainsanumberofvideotapedinterviewswithanumberofexpertsindesignresearch.
Selected journal articles and book chapters on concept and methodology
Therearesomanypublicationsoneducationaldesignresearchthatitisimpossibletodrawupacomprehensivebibliography.Anumberofpublicationshavealreadybeenlisted(withabstracts)ontheUGAwebsite(see6.2).Howeverwewanttopointthereadertoanumberofarticlesandchapterthathavehelpedustogetinvolvedindesignresearchandtounderstandthemainissuesinourfield.Giventhisrationaleforselectingthesetitles,thereaderwillfindthatsomeofthetitlesarealsoreferredtoontheUGAwebsite.
Akker,J.vanden(1999).Principlesandmethodsofdevelopmentresearch3.InJ.vandenAkkeretal.(Eds.),Design approaches and tools in education and training(pp.1-14).Dordrecht:KluwerAcademicPublishers. Abstract:Thischapterdiscussestheroleofresearchinrelationtoeducationaldesign
anddevelopmentactivities.Thefirstpartofthechapterfocusesontherationaleandbasicprinciplesofdevelopmentresearchbyoutliningmotivesforconductingformativeresearch,analyzingdefinitionsandaimsofvarioustypesofdevelopmentresearch,anddiscussingseveralofitskeycharacteristics.Thesecondpartofthechapterdealswith
2) Thewebsiteofthisspecialissueofferstheopportunitytopurchasearticlesforb22.00plusVAT.3) Theconceptdevelopmentresearch,usedinsometitles,issynonymoustodesignresearch.
an introduction to educational design research112
methodsofdevelopmentresearch,exploringsomeofitstypicalproblemsanddilemmas,anddiscussingseveralchallengesforfurtheractionandreflection.
Akker,J.vanden&Plomp,Tj.(1993).Developmentresearchincurriculum:propositionsandexperiences.PaperpresentedattheannualconferenceoftheAmericanEducationalResearchAssociation,April1993,Atlanta(GA,USA). Abstract:Reasontoincludethispaperisthatitisthefirstpaperfromthegroupatthe
UniversityofTwenteonwhattheycalledatthattimedevelopmentresearch.Basedontheassertionthatbothcurriculumdevelopmentandcurriculumresearchhavemuchrelevancetogainfromacloseliaison,theauthorssuggestthatboundariesbetweenthetwoshouldfade,whichcanbedoneinanewresearchstrategycalleddevelopmentresearch.Thepaperpresentsthepurpose,aconceptualframeworkandsomecharacteristicsofdevelopmentresearchincurriculum
See:www.leerplanevaluatie.slo.nl/taakhulp/lezen
Bannan-Ritland,B.(2003).Theroleofdesigninresearch:Theintegrativelearningdesignframework. Educational Researcher, 32(1),21-24. Abstract(fromUGAwebsite):Inthisarticle,ageneralmodelisproposedfordesign
researchineducationthatgrowsoutoftheauthor’sresearchandworkinrelateddesignfields.Themodelemphasizesthestagesensitivityof(a)researchquestions,(b)dataandmethods,and(c)theneedforresearcherstodesignartifacts,processes,andanalysesatearlierstagesintheirresearchthatcanthenbeprofitablyused(perhapsbydifferentresearchers)inlaterstages.
Barab,S.A.,andSquire,K.D.(2004).Design-BasedResearch:PuttingaStakeintheGround.Journal of the Learning Sciences, 13(1),1-14. Abstract(fromUGAwebsite):Thearticlehighlightsandproblematizessomechallenges
thatarefacedincarryingoutdesign-basedresearch.Itstatesthattheemergingfieldoflearningsciencesisonethatisinterdisciplinary,drawingonmultipletheoreticalperspectivesandresearchparadigmssoastobuildunderstandingsofthenatureandconditionsoflearning,cognitionanddevelopment.Afundamentalassumptionofmanylearningscientistsisthatcognitionisnotathinglocatedwithintheindividualthinkerbutisaprocessthatisdistributedacrosstheknower,theenvironmentinwhichknowingoccursandtheactivityinwhichthelearnerparticipates.Inotherwords,learning,cognition,knowingandcontextareirreduciblyco-constitutedandcannotbetreatedasisolatedentitiesorprocesses.
an introduction to educational design research 113
Brown,A.L.(1992).Designexperiments:Theoreticalandmethodologicalchallengesincreatingcomplexinterventionsinclassroomsettings.Journal of the Learning Sciences, 2,141-178. Abstract(fromUGAwebsite):Thisistheseminalarticleondesignresearch.Discusses
theoreticalandmethodologicalchallengesincreatingcomplexinterventionsinclassroomsettings.Movementfromtheclassicalpsychologicalpositionofconcentratingonatheoreticalstudyofthelearningprocessesofindividualstudentstoaconcentrationonconceptualchangeinteachersandstudents;Classroomrestructuring;Designexperiments;Experiencesonlearningtheory.
Design-BasedResearchCollective.(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher, 32(1),5-8. Abstract(fromUGAwebsite):Theauthorsarguethatdesign-basedresearch,which
blendsempiricaleducationalresearchwiththetheory-drivendesignoflearningenvironments,isanimportantmethodologyforunderstandinghow,when,andwhyeducationalinnovationsworkinpractice.Designbasedresearchers’innovationsembodyspecifictheoreticalclaimsaboutteachingandlearning,andhelpusunderstandtherelationshipsamongeducationaltheory,designedartifact,andpractice.Designiscentralineffortstofosterlearning,createusableknowledge,andadvancetheoriesoflearningandteachingincomplexsettings.Designbasedresearchalsomaycontributetothegrowthofhumancapacityforsubsequenteducationalreform.
Kelly,A.E.(2006).Qualitycriteriafordesignresearch.In:J.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge. Abstract:thischapterdiscussesforeachofthreedifferentusesfordesignresearchin
educationanumberofcharacteristicsandexemplaryexamples.Itintroducesthenotionofthecommissivespaceofdesignresearch,meaningthat(amongstothercharacteristics)designresearchdoesnotstriveforcontext-freeclaimsbutseescontextsascentraltoitsconceptualdomain,thatdesignresearchisexperimentalbutnotanexperiment,andthatdesignresearcherschoosetoworkinthe“contextofdiscovery”,ratherthaninthe“contextofverification”utilizingrandomizedtrials.
Reeves,T.(2000).Enhancingtheworthofinstructionaltechnologyresearchthrough“designexperiments”andotherdevelopmentalstrategiesPaperpresentedattheAnnualMeetingoftheAmericanEducationalResearchAssociation,April2000,NewOrleans(LA,USA).RetrievedOct.20,2006fromhttp://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf
an introduction to educational design research114
Abstract:Theauthorarguesthatingeneralresearchintheareaofinstructionaltechnologyispoor,notprovidingpractitionerswithsufficientguidance.Hediscussesvarioustypesofinstructionaltechnologyresearchgoalsandmethodsandsuggeststhat‘use-inspiredbasicresearch’isneededinthedomainofinstructionaltechnologyreferringtoapproacheslikedevelopmentresearchanddesignexperiments.Hepresentsaframeworkandcharacteristicsfordevelopmentresearchintheareaofinstructionaltechnology.
Selected journal articles and book chapters on design research in domains
Overthelastfewyears,increasinglyexamplesofdesignresearchhavebeenpublished.Thissectioncontainsjustafewexemplaryreferencestoarticlesandchaptersinbooksofdesignresearchinvariousdomains,ofwhichafewaretakenfromtheUGAwebsiteWehaveaddedinthenextsectionreferencestosomePhDdissertationsreportingondesignresearchconductedatDutchuniversitiesthatcaneasilybeaccessedthroughtheWorldWideWeb.
DomainofmathematicseducationAseminalchapteris:Gravemeijer,K.&Cobb,P.(2006).Designresearchfromthelearningdesignperpective,invandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.)EducationalDesignresearch:Thedesign,developmentandevaluationofprograms,processesandproducts.London:Routledge,17-51. Abstract:thischapterpresentsanapproachtodesignresearchthathasbeenusedand
refinedinaseriesofdesignresearchprojectsinwhichthetwoauthorscollaboratedoveraten-yearperiod.Theirapproachisfallingwithinthebroadercategoryofdesignresearchthataimsatcreatinginnovativelearningecologiesinordertodeveloplocalinstructiontheoriesontheonehand,andtostudytheformsoflearningthatthoselearningecologiesareintendedtosupportontheotherhandinthedomainofmathematicseducation(includingstatisticseducation).
Theapproachtodesignresearchhasitsrootsinthehistoryofthetwoauthorswhichistheworkonrealisticmathematicseducation(RME)thatiscarriedoutintheNetherlands(firstauthor)andthatofsocio-constructivistanalysisofinstruction(secondauthor).
Somereferences4fordesignresearchcasesinthedomainofmathematicseducationconductedintheUSAare:
4) WewanttoexpressourthankstoPaulCobbforprovidingthesereferences.
an introduction to educational design research 115
Bowers,J.S.,Cobb,P.,&McClain,K.(1999).Theevolutionofmathematicalpractices:Acasestudy.Cognition and Instruction, 17,25-64.Cobb,P.(1999).Individualandcollectivemathematicallearning:Thecaseofstatisticaldataanalysis.Mathematical Thinking and Learning, 1,5-44.Cobb,P.,McClain,K.,&Gravemeijer,K.(2003).Learningaboutstatisticalcovariation.Cognition and Instruction, 21,1-78.Confrey,J.,&Smith,E.(1995).Splitting,covariation,andtheirroleinthedevelopmentofexponentialfunctions.Journal for Research in Mathematics Education, 26,66-86.Lehrer,R.,&Schauble,L.(2004).Modelingnaturalvariationthroughdistribution.American Educational Research Journal, 41,635-679.Lobato,J.(2003).Howdesignexperimentscaninformarethinkingoftransferandviceversa.Educational Researcher, 32(1),17-20.Simon,M.A.(1995).Reconstructingmathematicspedagogyfromaconstructivistperspective.Journal for Research in Mathematics Education, 26,114-145.Stephan,M.,Bowers,J.,&Cobb,P.(Eds.).(2003).Supporting students’ development of measuring conceptions: Analyzing students’ learning in social context. Journal for Research in Mathematics Education Monograph No. 12. Reston,VA:NationalCouncilofTeachersofMathematics.
Anillustrativeexampleofdesignresearchinthecontextofadevelopingcountryis:Vos,P.,Devesse,T.G.,andPinto,A.A.R.(2007).DesigningMathematicsLessonsInMozambique:StartingFromAuthenticResources.African Journal of Research in SMT Education, 11(2),pp.51-66 Abstract:Thisarticledescribesresearchonthedesignofstudent-centredinstructionin
Mozambique.Thestartingpointwastheuseofreal-liferesources,suchastraditionalartcraftobjectsandauthenticnewspaperclippings.Theresearchwasbasedonaninstructionaldesignmodel,whichattemptstoaligntheorywithpracticeandwhichisgearedtowardsimprovingpractice.Intwoparallelstudies,oneongeometryandoneonstatistics,student-centredinstructionwasfacilitatedthroughtheuseofworksheetswithopen-endedquestionstailoredforgroupwork.Inacyclicprocess,theprototypematerialsandtheassociatedinstructionalmethodwereformativelyevaluated.Theevaluationsshowedthatthedesignswereusefuleveninclassroomspackedwithmorethansixtystudents.
DomainofscienceeducationHoadley,C.M.,&Linn,M.C.(2000)Teachingsciencethroughonline,peerdiscussions:SpeakEasyintheknowledgeintegrationenvironment.International Journal of Science Education,22(8),839-857.
an introduction to educational design research116
Abstract:Thisarticlediscusseswhetherstudentscanlearnsciencefromcarefullydesignedonlinepeerdiscussions.Contraststwoformatsofcontributedcomments--historicaldebateandnarrativetext--andassessestheimpactofanasynchronousdiscussiononstudentunderstandingofthenatureoflight.Italsoreportsthatstudentsgainintegratedunderstandingofthenatureofcolorfrombothdiscussionformats.
Kafai,Y.B.,&Ching,C.C.(2001).Affordancesofcollaborativesoftwaredesignplanningforelementarystudents’sciencetalk.Journal of the Learning Sciences, 10(3),323-363. Abstract:Thisarticleinvestigateswhethersciencepermeatesthedesignenvironment
andisthuscontextedwithintheotheractivitiesofcollaborativemanagementandtechnology.Focusesonwhichcontextsgaverisetosciencetalk.Studiesaclassroomwith(n=33)studentsdividedintoseventeams
Knippels,M.C.P.J.,Waarlo,A.J.,andBoersma,K.Th.(2005).Designcriteriaforlearningandteachinggenetics.Journal of Biological Education,39(3),108-112. Abstract:Whilelearningandteachingdifficultiesingeneticshavebeenabundantly
exploredanddescribed,therehasbeenlessfocusonthedevelopmentandfield-testingofstrategiestoaddressthem.Toinformthedesignofsuchastrategyareviewstudy,focusgroupinterviewswithteachers,acasestudyofatraditionalseriesofgeneticslessons,studentinterviews,andcontentanalysisofschoolgeneticsteachingwerecarriedout.SpecificdifficultiesreportedintheliteraturewerecomparabletothoseperceivedbyDutchteachersandfoundinthecasestudyandthestudentinterviews.Theproblemsassociatedwiththeabstractandcomplexnatureofgeneticswerestudiedinmoredetail.Theseparationofinheritance,reproductionandmeiosisinthecurriculumaccountsfortheabstractnatureofgenetics,whilethedifferentlevelsofbiologicalorganisationcontributetoitscomplexnature.Finally,fourdesigncriteriaaredefinedforalearningandteachingstrategytoaddresstheseproblems:linkingthelevelsoforganism,cellandmolecule;explicitlyconnectingmeiosisandinheritance;distinguishingthesomaticandgermcelllineinthecontextofthelifecycle;andanactiveexplorationoftherelationsbetweenthelevelsoforganisationbythestudents.
Key words:Biologyeducation;Genetics;Learningandteachingdifficulties;Designcriteria
Lijnse,P.L.(1995).“DevelopmentalResearch”asawaytoanempiricallybased“DidacticalStructure”ofScience.Science Education, 29(2),189-199. Abstract:Theauthorarguesthatdevelopmentalresearch(inthisbookcalled´design
research’)isneededinwhichsmall-scalecurriculumdevelopmentiscyclicallycoupledtoin-depthclassroomresearchofteaching-learningprocesses.Suchresearchshouldresultinworkedoutexamplesofsuccessfulwaysofteaching,accordingtonew
an introduction to educational design research 117
conceptualcurriculumstructures.Designingsuch`didactical´structuresconstitutesalongertermresearchprogram,whichasksforinternationalexchangeandcooperatition.
Domainofreading-writingAbbott,S.P.,Reed,E.,Abbott,R.D.,&Berninger,V.W.(1997).Year-longbalancedreading/writingtutorial:Adesignexperimentusedfordynamicassessment.Learning Disability Quarterly,20(3),249-263. Abstract:Sixteenchildrenwithseverereadingproblemsinfirstgradereceivedayear-
longindividualtutorialintervention.Growthcurveanalysesfoundsignificantgainsonmeasuresoforthographicandphonologicalcoding,wordidentification,wordattackskills,readingcomprehension,letterautomaticity,andspellingandmarginallysignificantgainsinwritingcomposition.
DeCorte,E.,Verschaffel,L.,&vandeVen,A.(2001).Improvingtextcomprehensionstrategiesinupperprimaryschoolchildren:Adesignexperiment.The British Journal of Educational Psychology,71,531-559. Abstract:Withrespecttotheacquisitionofcompetenceinreading,newstandardsfor
primaryeducationstressmorethanbeforetheimportanceoflearningandteachingcognitiveandmetacognitivestrategiesthatfacilitatetextcomprehension.Therefore,thereisaneedtodesignaresearch-basedinstructionalapproachtostrategicreadingcomprehension.Thedesignexperimentaimedatdeveloping,implementingandevaluatingaresearch-based,butalsopracticallyapplicablelearningenvironmentforenhancingskilledstrategyuseinupperprimaryschoolchildrenwhenreadingatext.Thisdesignexperimentshowsthatitispossibletofosterpupils’useandtransferofstrategicreadingcomprehensionskillsinregularclassroomsbyimmersingtheminapowerfullearningenvironment.Butthisinterventiondoesnotautomaticallyresultinimprovementofperformanceonastandardizedreadingcomprehensiontest.
Neuman,S.B.(1999).Booksmakeadifference:Astudyofaccesstoliteracy.Reading Research Quarterly,34(3),286-311. Abstract:Thisarticleexaminestheimpactofaninterventiontargetingeconomically
disadvantagedchildrenthatfloodedover330child-carecenterswithhigh-qualitychildren’sbooksandprovided10hoursoftrainingtochild-carestaff.Itexaminestheproject’simpactandgivessupportforthephysicalproximityofbooksandthepsychologicalsupporttochild-carestaffonchildren’searly-literacydevelopment.
DomainofinstructionaltechnologyBannan-Ritland,B.(2003).Theroleofdesigninresearch:Theintegrativelearningdesignframework.Educational Researcher,32(1),21-24.
an introduction to educational design research118
Abstract:(fromUGAwebsite):Inthisarticle,ageneralmodelisproposedfordesignresearchineducationthatgrowsoutoftheauthor’sresearchandworkinrelateddesignfields.Themodelemphasizesthestagesensitivityof(a)researchquestions,(b)dataandmethods,and(c)theneedforresearcherstodesignartifacts,processes,andanalysesatearlierstagesintheirresearchthatcanthenbeprofitablyused(perhapsbydifferentresearchers)inlaterstages.
Herrington,J.,&Oliver,R.(1997).Multimedia,magicandthewaystudentsrespondtoasituatedlearningenvironment.Australian Journal of Educational Technology,13(2),127-143.Availableat:http://www.ascilite.org.au/ajet/ajet13/herrington.html Abstract:Thisarticlepresentsadesignofaninteractivemultimedialearning
environmententitledInvestigatingassessmentstrategiesinmathematicsclassrooms,whichrepresentstheoperationalizedcharacteristicsofsituatedlearning.Theauthorsalsosuggestthecriticalguidelinesforthedesignofthemultimediasoftwaretoenableittosupportasituatedlearningenvironment.Theythenreportastudythatinvestigatespatternsofbehaviorofstudentsimmersedinthismultimediasituatedlearningenvironment.Thefindingssuggestthattheuseofthesituatedlearningmodelissuccessfulinprovidingguidelinesforthedevelopmentofaninteractivemultimediaprogram.Theyalsorevealthatininstanceswherelearnersareempoweredandareenabledtoassumehigherdegreesofresponsibilityfortheiractivityandconductinalearningsetting,theresearchersneedtobecognizantofthevariousdesignfactorswhichcanimpedeorenhancelearning.Inmultimediaenvironments,theseincludesuchelementsasthemotivationalaspectsoftheenvironment,theinterfacedesign,andthenavigationelementsemployed.Inconclusion,theauthorssuggestthatitisalsoimportanttopracticeresearchwhichexplorestheimpactofthemoretangibleaspectsofmultimediadesignsuchasthoseexploredinthisstudy.
Herrington,J.,&Oliver,R.(2000).Aninstructionaldesignframeworkforauthenticlearningenvironments.Educational Technology Research and Development,48(3),23-48.Availableat:http://edserver2.uow.edu.au/~janh/Assessment/Authentic%20Assessment_files/ETR%26D.pdf Abstract:Theinstructionaltechnologycommunityisinthemidstofaphilosophical
shiftfromabehaviouristtoaconstructivistframework,amovethatmaybegintoaddressthegrowingriftbetweenformalschoollearningandreal-lifelearning.Onetheoryoflearningthathasthecapacitytopromoteauthenticlearningisthatofsituatedlearning.
Thepurposeofthisthreepartstudywasfirstly,toidentifycriticalcharacteristicsofasituatedlearningenvironmentfromtheextensiveliteraturebaseonthesubject;secondly,tooperationalisethecriticalcharacteristicsofasituatedlearningenvironmentbydesigningamultimediaprogramwhichincorporatedtheidentifiedcharacteristics;
an introduction to educational design research 119
andthirdly,toinvestigatestudents’perceptionsoftheirexperiencesusinganmultimediapackagebasedonasituatedlearningframework.
Thelearningenvironmentcomprisedamultimediaprogramforpreserviceteachersonassessmentinmathematics,togetherwithrecommendedimplementationconditionsintheclassroom.Eightstudentswereobservedandinterviewedtoexploretheirperceptionsofthesituatedlearningenvironment.Findingssuggestthattheuseofthesituatedlearningframeworkappearedtoprovideeffectiveinstructionaldesignguidelinesforthedesignofanenvironmentfortheacquisitionofadvancedknowledge.
Reeves,T.(2006).Designresearchfromatechnologyperspective.In:J.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge. Abstract:Theeffectivenessofthefieldknownaseducationaltechnologyin
fundamentallyenhancingteachingandlearninghasincreasinglybeencalledintoquestion,ashastheefficacyofeducationalresearchingeneral.Doubtsabouteducationaltechnologyresearchprimarilystemfromdecadesofanarguablyflawedresearchagendathathasbeenbothpseudoscientificandsociallyirresposible.Itisproposedthatprogressinimprovingteachingandlearningthroughtechnologymaybeaccomplishedusingdesignresearchasanalternativemodelofinquiry.Designresearchprotocolsrequireintensiveandlong-termcollaborationinvolvingresearchersandpractitioners.Itintegratesthedevelopmentofsolutionstopracticalproblemsinlearningenvironmentswiththeidentificationofreusabledesignprinciples.Examplesofdesignresearchendeavorsineducationaltechnologyaredescribedhere.Thechapterendswithacallfortheeducationaltechnologyresearchcommunitytoadoptdesignresearchmethodsmorewidely.
Reeves,T.C.,Herrington,J.,&Oliver,R.(2004).Adevelopmentresearchagendaforonlinecollaborativelearning.Educational Technology Research and Development, 52(4),53-65. Abstract:Althoughimportant,traditionalbasic-to-appliedresearchmethodshave
providedaninsufficientbasisforadvancingthedesignandimplementationofinnovativecollaborativelearningenvironments.Itisproposedthatmoreprogressmaybeaccomplishedthroughdevelopmentresearchordesignresearch.Developmentresearchprotocolsrequireintensiveandlong-termcollaborationamongresearchersandpractitioners.Inthisarticle,weproposeguidelinesforimplementingdevelopmentresearchmodelsmorewidely,andconcludewithaprescriptionforanonlinecollaborativelearningresearchagendaforthenextfivetotenyears.
Reinking,D.,&Watkins,J.(2000).Aformativeexperimentinvestigatingtheuseofmultimediabookreviewstoincreaseelementarystudents’independentreading.Reading Research Quarterly, 35(3),384-419.
an introduction to educational design research120
Abstract:Thisstudyinvestigateshowacomputer-basedinstructionalintervention(creatingmultimediareviewsofbooks)mightincreasefourthandfifthgraders’independentreading.Thestudyfindsthatthesuccessoftheinterventionwasrelatedtothemediatingeffectsofusingtechnology,changesintheinteractionsamongstudentsandteachers,andstudents’engagementinrelationtotheirreadingability.Italsonotesseveralotherfactors.
DomainofcurriculumMcKenney,S.&vandenAkker,J.(2005).Computer-basedsupportforcurriculumdesigners:Acaseofdevelopmentalresearch.Educational Technology Research & Development, 53(2)41-66. Abstract:Inthisarticle,weexplorethepotentialofthecomputertosupportcurriculum
materialsdevelopmentwithinthecontextofsecondarylevelscienceandmathematicseducationinsouthernAfrica.Duringthefour-yearcourseofthestudy,acomputerprogramwasdevelopednamedCASCADE-SEA,whichstandsforComputerAssistedCurriculumAnalysis,DesignandEvaluationforScience(andmathematics)EducationinAfrica.Bycarefullydocumentingtheiterativeprocessofanalysis,prototypedesign,evaluation,andrevision,wesoughtinsightintothecharacteristicsofavalidandpracticalcomputer-basedtoolthatpossessesthepotentialtoaffecttheperformanceofitsusers.TheresultsofthisstudyincludetheCASCADE-SEAprogramitself,whichassistsusersinproducingbetterqualitymaterialsthantheyotherwisemight,whiletheyalsolearnfromthedevelopmentprocess.Further,thisresearchhascontributedtothearticulationofdesignprinciplesandrelateddevelopmentalresearchmethods.Thisarticlehighlightstheresearchanddevelopmentthattookplace,andonlybrieflyaddressesthetoolitself.
McKenney,S.,Nieveen,N,&vanderAkker,J.(2002).Computersupportforcurriculumdevelopers:CASCADE.Educational Technology Research and Development,50(4),25-35. Abstract:Thisarticleexaminesresearchonacomputer-basedtool,CASCADE(Computer
AssistedCurriculumAnalysis,DesignandEvaluation),thatwasdevelopedattheUniversityofTwente(Netherlands)toassistincurriculumdevelopment.Thearticlediscusseselectronicperformancesupportsystemsandtheneedforincreasedattentiontoimplementationandimpactstudies.
Nieveen,N.M.(1999).Prototypingtoreachproductquality.In:J.J.H.vandenAkker,R.Branch,K.Gustafson,N.M.Nieveen,&Tj.Plomp(Eds.),Design approaches and tools in education and training(pp.125-136).Dordrecht:Kluwer. Abstract: Thischapterprovidesaframeworkforproductqualityconsistingofthe
followingthreecriteria:validity,practicalityandeffectiveness,andprovidesinsightinto
an introduction to educational design research 121
theapplicabilityoftheframeworkinvariousdomainsofeducationalproductdevelopment.Inordertoreachproductquality,theprototypingapproachisseenandunderstoodasasuitableapproach.Thischapterdiscussesthreesignificantcharacteristicsofaprototypingapproach:extensiveuseofprototypes,highdegreeofiterationandtheroleofformativeevaluation,andtheparamountimportanceofuserinvolvement.Thechapterillustratesthewaytheprototypingapproachhasbeeninstrumentalindevelopingacomputersupportsystemforinstructionaldevelopers.Duringtheprototypingprocess,theframeworkassistedindecidingthefocusofeachprototypeandenhancedthetransparencyoftheentireprocess.
Nieveen,N.M.&Akker,J.J.H.,vanden(1999).Exploringthepotentialofacomputertoolforinstructionaldevelopers. Educational Technology Research & Development,47(3),77-98. Abstract:Informationandcommunicationtechnologytoolscurrentlypermeatealmost
everyprofessionaldomain.Thosegearedtowardthefieldofinstructionaldevelopmenthaveemergedinrecentyears.Thisarticleexploresthepotentialforlinkingthedomainsofcomputersupportandinstructionaldevelopment.ThisarticlereportsonthedesignandevaluationofCASCADE(ComputerAssistedCurriculumAnalysis,DesignandEvaluation),acomputersystemthatsupportsinstructionaldevelopersduringformativeevaluationefforts.Fiveprototypesofthesystemwerecreatedandevaluatedonthebasisoftheirvalidity(reflectionofstate-of-the-artknowledgeandinternalconsistency);practicality(abilitytomeettheneeds,wishesandcontextualconstraintsofthetargetgroup);andeffectiveness(improvedusertaskperformance).TheresultsofthisstudysuggestthattheuseofCASCADEcould:(a)improvetheconsistencyofformativeevaluationplansandactivities;(b)motivatedevelopersbyelevatingtheirconfidenceinusingformativeevaluationactivities;(c)savetime;and(d)helptoprovidejustificationsfordecisionsmade.
Some PhD theses utilizing design research as a research approach
Overtheyears,variousPhDthesehavebeenwritteninwhichdesignresearchhasbeenappliedasthemainresearchapproach.InthissectionwejustmentionafewthathavebeendefendedinTheNetherlandsattheUniversityofTwenteandtheUniversityofUtrecht.Undoubtly,manymoredissertationscanbefoundviasearchmachinesontheWWWeb,butwejustwanttopointthereadertoafewexamplesofdesignresearchwearefamiliarwith.
DomainofcurriculumdevelopmentNieveen,N.(1997).Computer support for curriculum developers.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente,.
an introduction to educational design research122
Availablefrom:http://projects.edte.utwente.nl/cascade/original/McKenney,S.(2001).Computer-based support for science education materials developers in Africa: exploring potentials. Doctoralthesis.Enschede(TheNetherlands):UniversityofTwente.Availablefrom:http://projects.edte.utwente.nl/cascade/seastudy/
Kouwenhoven,W.(2003).Designing for competence in Mozambique: towards a competence-based curriculum for the Faculty of Education of the Eduardo Mondlane University. Doctoralthesis.Enschede(TheNetherlands):UniversityofTwente.Availablefrom:http://doc.utwente.nl/41442/1/thesis_Kouwenhoven.pdfNote:thisisanexampleofdesignresearchinwhichtheresearcherwasnotactivelyinvolvedinallphasesofthedesignprocess.
DomainofprofessionaldevelopmentofteachersTeclaiTecle,Andemariam(2006). The potential of a professional development scenario for supporting biology teachers in Eritrea.Doctoralthesis.Enschede(TheNetherlands):UniversityofTwente.Availablefrom:http://purl.org/utwente/55985
DomainofmathematicseducationArmanto,Dian(2002).Teaching multiplication and division realistically in Indonesian primary schools : a prototype of local instructional theory.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/58710
Bakker,A.(2004).Design research in statistics education : on symbolizing and computer tools.Doctoralthesis.Utrecht(TheNetherlands:UniversityofUtrecht.Availablefrom:http://igitur-archive.library.uu.nl/dissertations/2004-0513-153943/inhoud.htm
Fauzan,Ahmad(2002).Applying realistic mathematics education (RME) in teaching geometry in Indonesian primary schools. Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/58707
DomainofscienceeducationKnippels,M.C.P.J.(2002).Copingwiththeabstractandcomplexnatureofgeneticsinbiologyeducation:Theyo-yolearningandteachingstrategy.Doctoralthesis.Utrecht(TheNetherlands:UniversityofUtrecht.Availablefrom:http://igitur-archive.library.uu.nl/dissertations/2002-0930-094820/inhoud.htm
an introduction to educational design research 123
Mafumiko,FideliceSimbagungileMbaruku(2006).Micro-scale experimentation as a catalyst for improving the chemistry curriculum in Tanzania.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/55448Ottevanger,W.(2001).MaterialsdevelopmentasacatalystforsciencecurriculumimplementationinNamibia.Doctoralthesis.Enschede:UniversityofTwente.
Tilya,FrankNicodem(2003).Teacher support for the use of MBL in activity-based physics teaching in Tanzania. Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/41462
Westbroek,H.B.(2005).Characteristics of meaningful chemistry education - The case of water quality.Doctoralthesis.Utrecht(TheNetherlands:UniversityofUtrecht.Availablefrom:http://igitur-archive.library.uu.nl/dissertations/2005-0922-200121/index.htm
an introduction to educational design research 125
AuthorbiographiesJanvandenAkkerJanvandenAkkerisdirectorgeneralofNetherlandsInstituteforCurriculumDevelopment[SLO].Besides,heispart-timeprofessor(chaironCurriculumDesign)attheUniversityofTwente.Inhiswideteaching,research,supervision,andconsultancyexperiences(bothintheNetherlandsandabroad)hetendstoapproachcurriculumdesignchallengesfromabroadereducationalinnovationperspective.Overtheyearshispreferencefordesignresearchhasincreasedbecauseofitsstrongcombinationofpracticalrelevanceandknowledgegrowth.Somebooksoverthelastdecade(editedwithsomecolleagues)thatrepresenthisorientation:Design Approaches and Tools in Education and Training (1999), Curriculum Landscapes and Trends (2003),and Educational Design Research (2006).
BrendaBannanBrendaBannanisanassociateprofessorintheinstructionaltechnologyprogramatGeorgeMasonUniversityinFairfax,Virginia,USA.Herresearchinterestsprimarilyinvolvetheintegrationofdesignandresearchprocessesrelatedtoeducationaltechnologydesignanddevelopment.ShehasauthoredseveralarticlesandchaptersondesignresearchintheHandbookforDesignResearchMethodsinEducation(Eds.Kelly,Lesh&Baek,2008)aswellastheEducational Researcher,vol32,2003.Dr.BannanwasawardedanNSFCAREERgrantawardin2003toconductafive-yearcycleofdesignresearchandwasinvitedasavisitingscholaratStanfordUniversityfortheacademicyearof2005-2006.
AnthonyE.KellyAnthonyE.KellyisaprofessorofeducationalpsychologyatGeorgeMasonUniversityinVirginia,USA.Hisresearchinterestsextendtoresearchmethodologydesign,andresearchattheintersectionofcognitiveneuroscienceandeducation.HecurrentlyhastwoNationalScienceFoundationgrants,oneonmodelinglearningincyberinfrastructurenetworksinearthsciences(withBrendaBannan-Ritland),andoneonlearninginmegacities.HisNSFgrantondesignresearchmethodswithRichardLesh,producedtheHandbookofDesignResearchMethods(Kelly,Lesh&Baek,2008).Kellyeditedthespecialissueondesignresearch,EducationalResearcher,vol32,2003.HeistheeditorofthespecialissueontheUSNationalMathematicsAdvisoryPanelReport,Educational Researcher,forthcoming.HeservedasaprogrammanagerattheNationalScienceFoundationfrom1997-2000,and2006.
NienkeNieveen(authorandeditor)NienkeNieveenworksattheresearchdepartmentofNetherlandsInstituteforCurriculumDevelopment[SLO].HerworkcentersaroundcoordinatingandsupportingtheInstitute’sdesignresearchactivitiesbytakingseriesofconcertedactionstoassistcurriculum
an introduction to educational design research126
developersininterweavingdesignandformativeevaluationactivities.Indoingso,SLOaimsatknowledgegrowthandprovenqualityforallofitsproducts,i.e.curriculumframeworksandteachingandlearningmaterials.Herdissertation,in1997,wasbasedonafouryeardesignresearchprojectand,alsoafterthisperiod,shecontinuedworkingondesignresearchprojectsattheUniversityofTwente.Shehasauthoredseveralarticlesandchaptersondesignresearchandco-editedthebookEducational design research.
TjeerdPlomp(authorandeditor)TjeerdPlompisemeritusprofessorofcurriculumoftheUniversityofTwenteinEnschede,TheNetherlands.Hehasbeeninchargeofteachingeducationaldesignmethodologyinthe(atthattime)FacultyofEducationalScienceandTechnology.HewaschairofIEA,theInternationalAssociationfortheEvaluationofEducationalAchievement,from1989-1999.IntheIEAheservedaschairforthe‘ComputersinEducation’study(Comped),theThirdInternationalMathematicsandScienceStudy(TIMSS)andtheIEASecondInternationalTechnologyinEducationStudy(SITES).Hisresearchinterestsareeducationaldesignanddesignresearch,internationalcomparativeresearch,andinformationtechnologyinthecurriculumandteachereducation.Hehasbeenrecentlyinvolvedasadvisorinvariousresearchprojectsandprogramsutilizingdesignresearch,bothinTheNetherlandsandinternationally.
An Introduction to Educational Design Research
SLO • Netherlands institute for curriculum developmentSLO
PO box 20417500 CA EnschedeThe Netherlands
T +31(0)53 484 08 40F +31(0)53 430 76 92E [email protected]
www.slo.nl
SLO is the Netherlands institute for curriculum development. We are bridging the contexts of policy, research, and practice. Our expertise focuses on the development of curricular goals and content for various educational levels, from national policy to classroom practices. We closely collaborate with many different stakeholders from policy circles, schools (boards, principals, teachers), research, civic organizations, and the society at large. This allows us to design and validate relevant curriculum frameworks, to elaborate exemplary materials and to evaluate these in school practices. Our products and services support both policy makers and schools and teachers in making substantive curricular decisions and in elaborating these into relevant, inspiring and effective education.
ISBN: 978 90 329 2329 7
An In
trod
uctIo
n to
Edu
cAtIon
Al dEsIg
n rEsEArch
Editors: tjeerd Plomp &
nienke n
ieveen
Editors:Tjeerd Plomp & Nienke Nieveen