A study of teachers’ use of online learning resources to design classroom activities

This article was downloaded by: [University of California Santa Cruz]On: 10 October 2014, At: 18:23Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

New Review of Hypermedia andMultimediaPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/tham20

A study of teachers’ use of onlinelearning resources to design classroomactivitiesM. Recker a , A. Walker a , S. Giersch b , X. Mao a , S. Halioris a , B.Palmer a , D. Johnson a , H. Leary a & M. B. Robertshaw aa Department of Instructional Technology , Utah State University ,Logan, UT, 84322, USAb National Science Digital Library , PO Box 3000, Boulder, CO,80307, USAPublished online: 07 Dec 2007.

To cite this article: M. Recker , A. Walker , S. Giersch , X. Mao , S. Halioris , B. Palmer , D.Johnson , H. Leary & M. B. Robertshaw (2007) A study of teachers’ use of online learning resourcesto design classroom activities, New Review of Hypermedia and Multimedia, 13:2, 117-134, DOI:10.1080/13614560701709846

To link to this article: http://dx.doi.org/10.1080/13614560701709846

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

http://www.tandfonline.com/loi/tham20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/13614560701709846

http://dx.doi.org/10.1080/13614560701709846

Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

A study of teachers’ use of onlinelearning resources to design classroom

activities

M. RECKER*$, A. WALKER$, S. GIERSCH%, X. MAO$,S. HALIORIS$, B. PALMER$, D. JOHNSON$, H. LEARY$ and

M. B. ROBERTSHAW$$Department of Instructional Technology, Utah State University, Logan, UT 84322, USA

%National Science Digital Library, PO Box 3000, Boulder, CO 80307, USA

While much progress has been made on the technical design and development of digitallibraries, much less is known about how and why education digital library content andassociated tools can support and enhance the activities of educators in their professionalwork. This article elaborates a conceptual framework that characterizes teachers’practices when using online learning resources (called ‘teaching as design’), and aprofessional development model aimed at increasing teachers’ capacity for designinglearning activities in the context of authentic practice. Findings from two workshopimplementations showed positive impacts on teachers’ knowledge, attitudes, andsubsequent behaviours using online learning resources. An analysis of teacher createdactivities indicates a relationship between the form of design (offload, adaptation, orimprovisation) and the granularity of the learning objects utilized in the activity.

1. Introduction

The past decade in the United States has seen several efforts to capitalize onthe possibilities afforded by the Internet, growing bandwidth capacity andcomputing power, and myriad other technology innovations to supporteducation goals in every type of learning environment. Within the broadspectrum of education technology, education digital libraries, such as theNational Science Digital Library (NSDL.org), are but a subset. At one level,they provide access to abundant high-quality, free online resources forlearning. Coupled with increased classroom access to the Internet (NCES2006), they enable much greater use of online resources by teachers. Atanother level, digital libraries present a complex environment for study, withtechnology supporting a collaborative network that allows users to contributeknowledge, either actively through annotations or reviews, or passivelythrough patterns of resource use. This contextualization expands the Webof inter-relationships and layers of knowledge that extend among selectedprimary resources and users (Lagoze et al. 2005).

*Corresponding author. Email: [email protected]

New Review of Hypermedia and Multimedia,Vol. 13, No. 2, December 2007, 117�134

New Review of Hypermedia and Multimedia

ISSN 1361-4568 print/ISSN 1740-7842 online # 2007 Taylor & Francis

http://www.tandf.co.uk/journals

DOI: 10.1080/13614560701709846

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

The development of technology used in education, from expectations andrequirements for use, through technical design, to implementation andevaluation, has been influenced by several fields: computer science, cognitivepsychology and artificial intelligence, library and information science,instructional design, and, broadly, education. During formative stages ofdevelopment for education digital libraries, which often occurs in thecomputer sciences, the primary question to address is does it (an interface,a piece of code, a program) work? A body of research from the past 5 yearsindicates steady progress on the technical design and development ofeducation digital libraries. However, much less is known about in situ usageof online learning resources by their primary audience: teachers and learners.In particular, little is known about how and why education digital librarycontent and associated tools can support and enhance the activities ofeducators in their professional practices, e.g. teaching.

To address this knowledge gap, our article first presents a conceptualframework from Brown and Edelson (2003) that characterizes teachers’practices when using curricular resources as a design activity. Building on thisframework, the article then describes a professional development modelaimed at increasing teachers’ capacity for designing learning activities bycustomizing and combining online learning resources to fit their local needsand context. These teacher-designed activities can then be shared withstudents and other teachers. The article concludes with a presentation offindings from two workshop implementations of this model with 16 teachersthrough the ‘Digital Libraries go to School’ project (Recker and Howe 2006).

These findings begin to shed light on the various ways teachers use digitallibrary resources in teaching and learning contexts. Much recent research oflate has called for an increased focus on how teachers use particulareducational technologies in the classroom (Wallace 2004, Lawless andPellegrino 2005, Mishra and Koehler 2006). Indeed, it is only by studyingtool use in everyday activities that the technology’s practical functionalitiescan be understood (O’Day and Nardi 2003). Such findings also serve acritical role in informing the next design cycle.

Moreover, by focusing on teacher practice, this work is advancing the roleof digital libraries in supporting teacher collaboration and context-buildingactivities. In particular, involving teachers as designers of learning activitiesthat are relevant to their classroom needs helps them participate in theemerging education cyberinfrastructure: first they become users of onlineresources and eventually close the loop by contributing newly learningactivities using online resources back to education digital libraries (Sumnerand Marlino 2004, Lagoze et al. 2005).

2. Theoretical framework: Teaching as design

The perspective informing our approach is one in which all teachers can beviewed as designers of learning activities for their students, a positiongenerally aligned with a constructivist learning philosophy. As an outcome of

118 M. Recker et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

studying teachers’ adaptation of an innovative curriculum, Brown andEdelson (2003) distilled the Design Capacity for Enactment (DCE) frame-work within a ‘teaching as design’ paradigm. As part of the framework, theydefined a continuum of teachers’ curriculum use, ranging from offloads toadaptations to improvisations.

They explained, ‘‘these three types of use describe the differential degrees inwhich responsibility for guiding instructional activity is distributable betweenthe teacher and available instructional resources’’ (Brown and Edelson 2003,p. 4). For example, in offloads, the curriculum is implemented essentiallyunchanged, and the bulk of instructional decisions is contained in theresources. In improvisation, the teacher flexibly borrows and customizespieces while playing the major role in the decision-making process. Theadaptation category represents the midpoint of the continuum.

Brown and Edelson (2003) surmised that curricular materials afford andconstrain design, interacting with teachers’ unique knowledge, skills, andexperience. But they noted that the continuum is neutral with regards toquality or effectiveness of the teacher or their designed activity. For example,in designing a class activity, an offload strategy may be planned by a teacherwith low pedagogical or subject-matter knowledge who cannot perceive aneed to adapt a resource. The same offload strategy might be employed by ateacher with high pedagogical or subject-matter knowledge who plans towander around the room giving individual help to students.

Informed by this framework, we argue that the kinds of learning activitiesteachers can design are both supported and constrained by the wideavailability and diversity of education digital library resources, and that thedesign of these activities also interacts with teachers’ unique backgrounds andneeds. Brown and Edelson’s continuum of teacher curriculum use provides astarting point, then, from which to examine how and why teachers use onlinelearning resources. In the next section, we describe the education digitallibrary environment framing our study, followed by a description of how weadapted some aspects of the framework to better analyse our unique context.

3. Adapting the framework

The teacher-designed learning activities studied using Brown and Edelson’scontinuum are situated within the context of the National Science DigitalLibrary and the Instructional Architect, described next. This sectiondescribes how we modified the framework to incorporate the context ofeducation digital libraries, specifically the granular nature of online learningobjects.

3.1 National Science Digital Library

Funded by the National Science Foundation (NSF) since 2000, the NationalScience Digital Library (NSDL.org) is an education digital library that offersaccess to over 1.5 million online learning resources from over 100 partner

Online learning resources to design classroom activities 119

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

education digital libraries (Lagoze et al. 2006). The NSDL cataloguesinnovative online learning resources that are targeted at science, technology,engineering, and mathematics (STEM) learners and teachers of all ages.Teachers and learners can access a wide variety of online resources, in manyformats and at various levels of granularity. These can include large, self-contained courses to small, easily adapted resources such as a simplegraphing calculator applet (Zia 2001). Additionally, the NSF has fundeddevelopment of infrastructure and services, such as the InstructionalArchitect, that support collaborative relationships between users and thatallow users to develop context around resources.

3.2 Instructional architect

Funded by the National Science Foundation in 2000, a team of researchers atUtah State University has developed a simple, free, web-based, end-userauthoring service, called the Instructional Architect (IA, http://ia.usu.edu),which supports teachers’ designing online learning activities. With the IA,teachers can find and gather NSDL and Web resources, design IA projectscontaining learning activities with gathered resources, and share these IAprojects with students and peers (Recker et al. 2005).

3.3 Modifying the continuum

Note that Brown and Edelson were examining instructional planning andclassroom implementation together. We separated these temporal events byexamining the activities designed by teachers, stopping short of observingclassroom implementation. The differentiation between planning and enact-ment time has also been proposed by others (Remillard 2005).

In particular, aspects of the Brown and Edelson continuum wereoperationalized to classify teacher design of learning activities, as follows:

(1) Improvisation: teachers link to resources as a starting point or referencebut have clearly designed their own elements such as learning goals, addedinstructional content or activities, description of context of resource use,and assessment items;

(2) Adaptation: a midpoint, with only some of the elements listed above; and(3) Offload: teachers provide links to resources with little additional teacher-

created instructional guidance (e.g. explanations or instructions). Usetends toward lists of links (perhaps with added navigational information).

3.4 Defining granularity

Furthermore, the DCE framework was developed within the context of acomplex science curriculum, which included a complete sequence of tasks. Weexamine teacher design with online resources from the NSDL and wider Web,


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

both of which consist of essentially unlimited sets of resources that varywidely in terms of instructional scope, sequence, and granularity.

Thus, we also posit that the granularity (or size) of a resource has animpact on the extent to which teacher-designed activities are offloads,adaptations or improvisations. Our hypothesis is that because of manyinternal dependencies, large resources are best used with little modification oradditional effort, and the number of contexts in which they can be applied issmall. Conversely, small self-contained resources afford greater teacherimprovisation and adaptation in a wider range of situations (Wiley et al.2000). Put differently, large granularity resources typically come with theirown context for use, whereas small resources typically need to have contextadded.

Within the learning objects literature, there have been several attempts todefine taxonomies for resource granularity, as these deeply affect catalogue-ing, metadata, and instructional design decisions. For example, SCORM(Advanced Distributed Learning 2005 2006) defines three levels of granularity(from small to large):

(1) an Asset is any small media or data object deliverable by a browser;(2) a Shareable Content Object (SCO) is a collection of one or more assets

comprising a self-contained unit of instruction (and possibly delineatingthe relationship between and set of activities); and

(3) a Content Aggregation provides structure in which SCOs are aggregatedinto a cohesive sequence of instruction.

Note that while the standard does not impose any particular constraints onthe exact size of SCOs, it assumes they will be reused across multiple learningcontexts.

CANCORE (Cancore 2006), on the other hand, defines four levels ofcontent aggregation, from small to large:

(1) raw media data (equivalent to a SCORM asset);(2) packages that can be accessed as individual files, where those files form a

single aggregate resource (similar but not limited to SCORM SCO);(3) resources that can be decomposed into two or more objects that are

collections of raw data (similar but not limited to SCORM ContentAggregation); and

(4) resources that incorporate more than two levels of combination oraggregation.

In practice, these definitions proved unwieldy because they emphasizetechnical levels of composition or aggregation rather than aggregation toachieve learning outcomes. Also, the definitions are vague, leading to highlysubjective interpretations when attempting to categorize resources. As aresult, we distilled the following, more instructionally relevant definitions forour purposes, though as discussed later, they still contain ambiguouselements:


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

(1) Small (Asset): Text, images, sound, quiz, etc, which can be created bycombining several assets and easy to use across several different contexts(e.g. changes in pedagogy, changes in instructional level, etc.).

(2) Medium (Learning Activity): collection of several Assets, with a providedcontext. Although it could be repurposed, it would require some level oftechnical knowledge or some sort of added context for students abouthow the resource should be used in a way other than originally intended.

(3) Large (Course): collection of several Learning Activities, and may includea sequence or suggested sequence. These can be repurposed, but it wouldpresent technical challenges, and likely not be cost-effective.

4. Professional development model

The design of the Digital Libraries go to School project provided anopportunity to introduce teachers to the ‘teaching as design’ theory andexamine the artefacts teachers created as they enacted the theory.

Following best practices from the teacher professional developmentliterature (e.g. Sparks and Loucks-Horsley 1989, Marx et al. 1997, Putnamand Borko 2000, Sorge and Russell 2000, Borko 2004), we designed aprofessional development model consisting of two hands-on workshopsessions with between-workshop activities that include teaching, sharing,and communicating.

The characteristics of the professional development model are summarizedin table 1. The model follows a modified problem-based learning (PBL)approach (Barrows 1996, Gijbels et al. 2005), which was selected in part

Table 1. Professional development model.

Phase PD goals and activities Data collected

Workshop 1 Learn about digital libraries and toolsLearn search techniquesEngage in modified PBL. Participants:

. Identify authentic instructional problem

(Putnam and Borko 2000)

. Design IA project(s) to address need

Pre-survey Observations

Betweenworkshopactivities

Continue modified PBL Activities:. Implement IA project(s) in classroom

. Review peers’ designed activities (Hoffman

and Thompson 2000)

. Write reflection paper noting barriers and

successes

Email discussions andfollow-up Reflection papers

Workshop 2 Increase design capacity with online resourceFinish modified PBL activities:

. Reflect on and discuss designed activities

and classroom implementation stories

. Reflect on and discuss pedagogical and

design strategies (Pianfetti 2001)

Group interviewsObservations Post-surveyWebmetrics Artifact analysis


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

because it has proven to be effective in professional development settings withadult learners (Walker and Leary 2007).

During the first session, teachers are each asked to identify an instructio-nal problem and then to design learning activities using the InstructionalArchitect and using online resources from the National Science DigitalLibrary to meet identified classroom needs. Although PBL intends studentsto be self-directed, typically problems solved by students are carefullydesigned and sequenced to both promote student learning and mimic real-world problems (Savery 2006). In our case, in an environment where teachersneeded to see immediate relevance, we opted to err on the side of authenticityand have teachers select their own problems. This has the disadvantage of notbeing able to design problems that bring out the features of the professionaldevelopment that we identify as most critical to learn, but the benefit ofallowing teachers to see immediate benefit from their efforts.

Between the two workshops, participants design further activities, some ofwhich are implemented in their classroom. They also engage in small groupinteractions. Note that in the full version of PBL (Barrows 1996), these typesof interactions have two essential parts, the first serving as a discovery phase,in which group members discuss the problem, brainstorm about their existingknowledge gaps in relation to finding a problem resolution, then split up thetask of addressing those knowledge gaps. This phase was not part of ourworkshop model predominantly because teachers were not asked to solvethe same problem, rather one that was unique to their own classroom. Thesecond phase consists of reflection, a critical analysis of solutions and ofthe knowledge used as part of that solution. This phase was a part ofthe workshop, both in between the workshop activities, and supported by asimple e-mail listerv, as well as within the second workshop. In addition toonline interaction, participants were encouraged and did engage in face-to-face interactions where possible.

At the second session, participants present, discuss, and reflect upon theirclassroom implementation activities and experiences. Table 1 also shows datathat are collected at each stage.

The following detailed description of the process of creating learningactivities in the Instructional Architect will help ground the explanation ofour analysis and findings presented below.

4.1 Two Examples

We begin the description of the Instructional Architect with two examplescreated by teachers using our tool (figures 1 and 2). The foreground of eachfigure shows one of the teacher’s selected online resources. The backgroundshows the output of using IA: a Web page containing the content created bythe teacher, consisting of activities and annotations for online resources(referred to by links).

As is apparent from the figures, teacher-created IA projects are fairlysimple. Typical teachers are not professional Web designers; nor should we


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

expect them to be. Instead, they are teaching professionals attempting toefficiently and effectively address classroom and learning issues.

4.1.1 System description. From the home page of the Instructional Architect,users can (1) browse IA projects, (2) register as a new user, or (3) login as aregistered user (or guest, reduced functionality).

(1) Browse. Currently, the IA has over 1175 IA projects made publiclyavailable by registered users. Users can access IA projects by performingkeyword searches or by browsing these IA projects by subject area, gradelevel, author’s last name, or title (figure 3).

(2) Register. Users can create a free account, which provides them secureaccess to their saved resources and IA projects.

(3) Login. After the user logs in, the IA offers three major usage modes. First,with the ‘My Resources’ tool, users can search for resources in the NSDL.Queries are sent to the NSDL REST-based search interface (Lagoze2002). Metadata records for matching resources are displayed to users inan abbreviated form (including title, author, brand, description, and date).After browsing these results and viewing resources, users can selectdesired resources for further use. Users can also add any Web resource by

Figure 1. Instructional Architect project. To protect their privacy, the names of the project

authors have been removed.


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

entering its URL. Users can also organize their selected resources intofolders (figure 4).

Second, with the ‘My Projects’ tool, users can create Web pages in whichthey sequence and annotate their selected resources in order to create learningactivities (called ‘IA projects’).

Finally, users can share their IA projects by ‘Publishing’ them and settingpermissions on them, such as (a) user-only view, (b) users and their students(student view), or (c) public view (anyone browsing the IA site). Users can alsoadd basic metadata about their IA projects, including subject area, gradelevel, and core curriculum standard. These are then used to support browseand search of existing IA projects, as described above.

With our theoretical framework and definitions specific to educationdigital libraries in place, the next sections describe the methods for andfindings from two implementations of the professional development model.In particular, we are interested in gauging the impact of the workshops onteachers’ knowledge, attitudes and beliefs with regards to the use of anddesign activities with online resources. In addition, we examined teachers’resulting designed learning activities, with a particular focus on designstrategy and the role of resource granularity.

Figure 2. Instructional Architect project.


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

5. Methods

5.1 Participants

Findings from two workshops are reported, with eight schoolteachers fromUtah participating in the first and eight teachers from New York in thesecond (total N�16). Participation was voluntary, and recruiting was donecooperatively with the regional organizations and the school districtadministration of two rural school districts in Utah and New York. Schooladministrators provided logistical support and recruitment, and were presentat both workshops, which provided a supportive environment (Sorge andRussell 2000). Upon completion, participants received a small stipend fortheir time, in addition to continuing education credits.

Complete survey results were collected from 13 of the 16 participants.Table 2 summarizes demographic information for survey respondents fromthe two groups. Eleven of 13 teachers (85%) reported more than 6 years ofexperience, and 10 of 13 (77%) taught at the secondary level.

5.2 Design and procedures

The procedures used and data collected are summarized in table 1. The researchdesign consisted of mixed methods, in a one-group pre-test, post-test design

Figure 3. User browsing other teachers’ public projects.


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

(Campbell and Stanley 1966). Although two sites were involved, they were bothexposed to the same workshop, and the goal was not to make comparisonsbetween them. At the start and end of the workshops, participants completedonline surveys, consisting of both open-ended and Likert scale items. Somemeasures were unique to the pre- or post-surveys and two different groups.

Figure 4. User organizing selected online resources into folders for future use.

Table 2. Utah and New York participant demographic information.

Teacher demographicsTeaching years

Workshop 5 or less 6�10 11�15 16 or more

Utah 1 (14%) 1 (14%) 2 (29%) 3 (43%)NY 1 (16.5%) 1 (16.5%) 0 4 (67%)Total 2 (15%) 2 (15%) 2 (15%) 7 (55%)

Grade levels

Pre-school Elementary Secondary

Utah 1 (14%) 2 (29%) 4 (57%)NY 0 0 6 (100%)Total 1 (8%) 2 (15%) 10 (77%)


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

6. Findings

Analyses involved a mixed method approach (Johnson and Onwuegbuzie2005), drawing from analyses of pre- and post-workshop online surveys,workshop observations, group interviews, Web server logs (webmetrics), andanalysis of the IA projects created by workshop participants. Table 1 showsthe stages of data collection during the study. Unfortunately, the reflectionpapers, peer reviews, and e-mail discussions suffered from low response andparticipation rates. This may in part be due to the inherently individualnature of the problem-solving activity. It may also be because we did not setconcrete deadlines for participation (and many teachers saw each other atschool anyway). Future workshops should set clearer deadlines and clarifythe benefits of participating.

We first report results for the pre and post-surveys to document teacherattitudes toward their experiences. We then present more in-depth analyses ofteacher design activities.

6.1 Survey results

Analyses of the pre-survey responses indicated that participants’ attitudeswere favourable to begin with (Questions 1 and 2 in table 3). The data indicatea potential ceiling effect as evidenced by the standard deviation for the onlineresource questions placing the mean beyond the top of the scale. Despite this,comparisons with the post-survey results (as indicated by the effect size (d)comparisons) show that the workshop still managed to have a positive impact(all effect sizes were calculated using the unbiased estimate of the populationstandard deviation as the denominator, classifying effect sizes in the followingrange as: d�0.8 large, d�0.5 medium and d�0.2 small; Cohen 1988). Inaddition, intended behaviour changes (as self-reported on the surveys)reported high means (see the last two questions in table 3).

In sum, survey results show that participants had very positive attitudestoward the potential of online resources and introduced tools. Such findingsare echoed in our previous work (Recker 2006, Recker et al. 2006), as well asother studies of teacher use of online resources (McCormick and Li 2006).

Anecdotally, we also noticed much enthusiasm by teachers beyond thescope of the workshops. For example, one teacher is teaching colleagues how

Table 3. Participant attitudes towards online resources and tools.

Pre-survey Post-survey

Likert scale question (0�very low; 4�very high) N M SD M SD d

Online resources important for the quality of education 13 3.3 0.8 3.6 0.6 0.3Online resources make the teacher’s job easier 13 3.1 0.8 3.3 0.6 0.2I will use the Instructional Architect in my class 13 � � 3.4 0.6 �I will tell teachers about the Instructional Architect 13 � � 3.5 0.6 �


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

to use the tool. Another wants to present about her experiences at a nationalteachers’ conference.

6.2 IA project analysis

However, increased positive attitudes do not necessarily translate intochanged behaviours. Because teacher use of IA is situated in an onlineenvironment, webmetrics are one way to analyse usage patterns. In addition,the learning activities created by IA users offer a unique and convenientwindow into understanding how teachers organize and design learningactivities with online learning resources. In this way, these data move beyondidentifying whether teachers are simply using online resources, to character-izing how and what teachers do with those resources in educational contextsas evidenced by the learning activities created.

The 16 participants created a total of 89 IA projects. Of these, we identifiedpublic IA projects that had been viewed more than 10 times, under theassumption that they were likely to have been accessed by students and thusrepresented completed as opposed to draft IA projects. These 35 IA projectsform the basis for the subsequent analyses.

These IA projects were viewed a mean of 78.4 times, and each referenced amean of 7.4 online resources (table 4). As the Utah workshop occurredapproximately 6 months prior to the New York workshop, it is not surprisingthat the number of created and resulting accesses of Utah projects is higher.Note that at the time of the analysis, IA projects made use of 11 resources(6%) that could not be accessed on the Web. Thus, persistent access to contentremains a problem for teachers.

Authors of this article each coded a randomly selected subset of the IAprojects in the categories previously described along the following twodimensions: (1) its design strategy on the continuum (offload, adaptation, orimprovisation) and (2) the granularity of resources (small, medium, or large)included in those activities. This exercise also helped to further refine thedefinitions of the design continuum and resource granularity rubrics.

6.2.1 Inter-rater reliability. Intra-class correlations (ICC) for IA project andresource categorizations were calculated as a measure inter-rater reliability(ICC range from �1 to 1, with an acceptable reliability measure set at �0.8

Table 4. IA project usage and characteristics.

IA projects viewed �10 times

No. ofparticipants No.

Mean accesses(max)

Mean no. resources used perIA project (SD)

Utah 8 24 112 (432) 3.3 (2.2)New York 8 11 44.4 (80) 6.1 (4.4)Total 16 35 78.4 (48.0) 4.2 (3.3)


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

for established measures and �0.7 for new or pilot measures). Projects (n�3)and resources (n�19) with less than three codings were excluded from theanalysis.

ICC yielded an initial value of .57 for project categorizations and .33 forresource categorizations, both below the acceptable reliability threshold. As aresult, after clarifying of the rubric, authors recoded all IA projects andresources, yielding an acceptable ICC value of .88 for projects and .84 forresources. Both of these levels showed acceptable reliability.

6.2.2 Coding results. Categorizations were tabulated in order to display therelationship between IA project design strategy and resource granularity.Only those categorizations where at least two coders agreed were included inthe analysis. This criterion resulted in the exclusion of three IA projects and31 resources.

Table 5 tabulates, for each IA project created, its design strategy and thegranularity of all of the resources it used. First, note that offloading (see figure2 for an example) followed by improvising appear to be the most populardesign strategies. Second, approximately 50% of the resources used were ofmedium granularity. Interestingly, no project with an improvising strategyused large resources (as expected). Offloading projects, however, appeared tolargely rely on medium granularity resources. Naturally, it is hard to know ifthis pattern reflects teachers’ selection strategies or the nature of the resourcesthey were working with, or if it is an artefact of our coding rubric.

In addition, this type of artefact analysis fails to reveal any additionalcontext that may have been provided by a teacher at classroom implementa-tion time. For example, a teacher may have provided verbal explanations andguidance before or during the students’ learning activities. In this case, it canbe argued that the teacher is following an improvisation strategy atimplementation time, even though the related artefact looks like an offloadwhen viewed at design time.

6.3 Revisiting the coding scheme

As a result of applying the coding scheme, several suggestions were made toimproving the definitions used. We describe these suggestions, while notingthat precise definitions of the granularity of online learning resources remainelusive.

Table 5. IA project design strategy and its resources’ granularity.

Design strategy

Resource granularity Offload Adapt Improvise Total%

Small 8 6 7 30Medium 13 7 16 51Large 7 6 0 19Total % 40 27 33


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

In an effort to clarify granularity, coders observed that a teacher might linkto a large or medium granularity resource but provide accompanyingannotations for students to access a specific part of the resource. Thus, theresource should not really be considered ‘‘large.’’ This observation also pointsto teachers’ desires to ‘‘modify’’ resources.

Second, some IA projects looked more like lesson plan outlines, as theycomprised a collection of links and little added annotations. The teacher mayhave intended to use these to help stimulate classroom discussions, or asresources for student work. However, without accompanying classroomobservations, it is hard to know the teachers’ real intent.

Finally, the resources used by teachers varied widely in terms of theirinstructional characteristics. Some were informational (e.g. a site listingpopulations for various countries), while others were very clearly instruc-tional in nature (e.g. containing objectives and assessment). Existinggranularity schemes do not adequately capture important differences betweencontent and instructional intent. Further work should focus on capturingthese instructional characteristics, and clarifying the interaction betweenresource affordances and activity design (Brown and Edelson 2003, Wallace2004)

7. Limitations

This research has several limitations, including (a) participant self-selection,(b) the use of self-report survey data, (c) some low response rates, and (d) thelack of follow-up. In this section, we briefly address these problems.

First, participants in the workshop chose to engage in professionaldevelopment opportunities. As such, it is not possible to generalize findingsto all teachers. A more rigorous study would involve a randomly selectedsample of teachers who varied in terms of comfort with informationtechnology, the Internet, and the design of activities using learning resources.

Second, some conclusions were based on self-report survey data, whichmay be subject to recall bias and hence under-reported or over-reported. Inparticular, due to the halo effect, participants possibly overstated the valueand potential of learning resources and tools.

Third, as noted above, the discussion aspects of the workshop sufferedfrom low participation rates, perhaps because they were not sufficientlyemphasized, and teachers were working on individually (rather thancollaboratively) selected problems. These sources of data could have helpedfurther clarify teachers’ design and resource selection strategies. However, itremains an empirical question as to whether increasing the level of interactionamong teachers will benefit workshop participants.

Finally, little contact occurred with participants after the workshop. Whileour Web server log files reported subsequent use of the IA, little is knownabout how teachers subsequently designed projects and how they wereimplemented (if at all) in classrooms. And, of course, we were unable to trackfurther use of Web resources and the NSDL.


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

8. Conclusion

By creating a professional development model that supports teachers asdesigners of learning activities, is relevant in situ, and includes stages forqualitative and quantitative data collection throughout, there are nowmultiple forms of data that begin to help characterize usage. Moreover, thedata move beyond identifying whether teachers are using online resources, tohow and what they do with those resources.

Findings suggest that the workshop implementations had positive impactson teachers’ knowledge, attitudes, and subsequent behaviours, all of whichindicate a quality professional development model. Moreover, analyses revealthe different strategies teachers used to design learning activities composed ofonline learning resources. These suggest that design strategies are in partrelated to the granularity of the online resources available in education digitallibraries. In particular, 40% of the time, teachers appeared to engage in anoffload design strategy, and most commonly used medium granularityresources. The relationship is perhaps most evident with large granularityresources, which were never utilized in improvisation activities, and the mostin design-time offload activities.

However, more follow-up is needed to examine how IA projects and onlineresources are used during classroom implementation time*recall this was theoriginal context for Brown and Edelson’s (2003) work. Although this researchtends to show that the framework holds up and can be reliability assessed atdesign time, the relationship between design time and classroom implementa-tion time activities needs to be examined. Informal data collection from oneworkshop participant suggests that they may be quite different. Ongoingwork includes classroom observations as teachers use online resources afterattending one of our workshops. Other researchers have echoed this call forfurther long-term impact studies of professional development workshops(Lawless and Pellegrino 2005).

Finally, application of our rubric by several independent coders led torefinements of definitions of design. It also served to identify where furtherclarity is needed. In particular, we found that established definitions in themetadata standards literature regarding resource granularity (i.e., CAN-CORE and SCORM) were ambiguous, and that our coding scheme based onthese definitions is in need of further refinement. We expect that as the rubricis refined, it will contribute to the improvement of metadata vocabularies.

In sum, we anticipate that these empirical findings will be of interest todesigners of tools that help users find, select, and add annotations to digitallibrary resources. We also expect them to be of interest to an emerging fielddevoted to the development of online learning resources. In particular, one ofthe implications of this teacher�curriculum interaction is that learning-resource developers can build in affordances and constraints to furthersupport teacher design.

Finally, we believe that involving teachers as designers of learning activitiesthat are relevant to their classroom needs helps them become participants inthe emerging education cyberinfrastructure. In the first case, they become


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

users of online resources; eventually they close the loop by contributinglearning activities back to education digital libraries.

Acknowledgements

This work was partly supported by grants from the National Science Foundation (ESI-0554440

and DUE-0333818). Any opinions, findings, and conclusions or recommendations expressed in

this material are those of the authors and do not necessarily reflect the views of the National

Science Foundation. A portion of this paper was presented at the 2007 Joint Conference on

Digital Libraries. We thank the participating teachers and members of the IA research group.

References

Advanced Distributed Learning, ‘‘Sharable content reference model’’, 2005. Available online at: http://

www.adlnet.org/ (accessed 23 May 2005).

Advanced Distributed Learning, SCORM 2004 (3rd ed.), 2006. Available online at: http://www.adlnet.org/

(accessed 23 April 2007).

H.S. Barrows, ‘‘Problem-based learning in medicine and beyond: A brief overview’’, New Directions Teach.

Learn., 68, pp. 3�12, 1996.

H. Borko, ‘‘Professional development and teacher learning: Mapping the terrain’’. Educ. Res., 33, pp.

3�15., 2004

M. Brown and D. Edelson, Teaching As Design: Can We Better Understand the Ways in Which Teachers

Use Materials So We Can Better Design Materials to Support Their Change in Practice? (Design Brief).

Evanston, IL: Center for Learning Technologies in Urban Schools, 2003.

D.T. Campbell and J.C. Stanley, ‘‘Experimental and quasi-experimental designs for research’’. Chicago:

Rand McNally, 1966.

Cancore, ‘‘Cancore Guidelines [Electronic Version]’’, 2006. Available online at: http://www.cancore.ca/

guidelines/CanCore_Guidelines_General_2.0.pdf (accessed 5 May 2007).

J. Cohen, Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Erlbaum, 1988.

D. Gijbels, F. Dochy, P. Van den Bossche and M. Segers, ‘‘Effects of problem-based learning: A meta-

analysis from the angle of assessment’’, Rev. Educ. Res., 75, pp. 27�61, 2005.

E. Hoffman and G. Thompson, ‘‘Putting the research to work: Professional development models from

Michigan’’, Tech. Trends, 44, pp. 20�23, 2000.

R. Johnson and A. Onwuegbuzie, ‘‘‘‘Mixed methods research: a research paradigm whose time has

come’’’’, Educ. Res., 33, pp. 14�26, 2005.

C. Lagoze, ‘‘Core Services in the Architecture of the National Digital Library for Science Education

(NSDL)’’, in Proceedings of Joint Conference of Digital Libraries. New York: ACM, 2002, pp. 201�209.

C. Lagoze, D. Krafft, T. Cornwell, N. Dushay, D. Eckstrom and J. Saylor, ‘‘Metadata aggregation and

‘automated digital libraries’: A retrospective on the NSDL experience’’, in Joint Conference on Digital

Libraries (JCDL). New York: ACM, pp. 230�239, 2006.

C. Lagoze, D. Krafft, S. Payette and S. Jesuroga, ‘‘What is a digital library anymore, anyway?’’, D-Lib

Mag., 11, 2005. doi: 10.1045/november2005-lagoze.

K. Lawless and J. Pellegrino, ‘‘A prospectus for design of evaluation studies of the USDOE EETT program:

professional development of teachers in the integration of technology into teaching and learning’’.

Unpublished manuscript, Chicago (2005).

R. Marx, P. Blumenfeld, J. Krajcik and E. Soloway, ‘‘Enacting project based science’’, Element. School J.,

97, pp. 341�358, 1997.

R. McCormick and N. Li, ‘‘An Evaluation of European Learning Objects in Use’’, Learn. Media Technol.,

31, pp. 213�231, 2006.

P. Mishra and M. Koehler, ‘‘Technological Pedagogical Content Knowledge: A Framework for Teacher

Knowledge’’, Teach. College Rec., 108, pp. 1017�1054, 2006.

NCES, Internet Access in U.S. Public Schools and Classrooms: 1994�2005. Washington, DC: US

Department of Education, 2006.


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

V. O’Day and B. Nardi, ‘‘An ecological perspective on digital libraries’’, in Digital Library Use: Social

Practice in Design and Evaluation, A. Bishop, N. Van House and B. Buttenfield, Eds, Cambridge: MA: The

MIT Press, pp. 65�84, 2003.

E. Pianfetti, ‘‘Teachers and technology: digital literacy through professional development’’, Lang. Arts, 78,

pp. 255�262, 2001.

R. Putnam and H. Borko, ‘‘What do new views of knowledge and thinking have to say about research on

teacher learning?’’, Educ. Res., 29, pp. 4�15, 2000.

M. Recker, ‘‘Perspectives on teachers as digital library users: consumers, contributors, and designers’’,

D-Lib Mag., 12, 2006. doi: 10.1045/september2006-recker.

M. Recker, J. Dorward, D. Dawson, Y. Liu, X. Mao, B. Palmer, S. Halioris and J. Park, ‘‘You can lead a

horse to water: teacher development and use of digital library resources’’, in Proceedings of the Joint

Conference on Digital Libraries, New York: ACM, 2005, pp. 1�9.

M. Recker, J. Dorward, D. Dawson, Y. Liu, X. Mao, B. Palmer, S. Halions and J. Park, ‘‘Teaching, creating,

sharing: a context for learning objects’’, Interdisciplinary Journal of Knowledge and Learning Objects, 1,

pp. 197�216, 2006.

M. Recker and K. Howe, Digital Libraries go to School (DUE-0333818). Washington, DC: National

Science Foundation, 2006.

J. Remillard, ‘‘Examining key concepts in research on teachers’ use of mathematics curricula’’, Rev.Educ.

Res., 75, pp. 211�246, 2005.

J.R. Savery, ‘‘Overview of problem-based learning: definitions and distinctions’’, Interdisc. J. Problem-

Based Learn., 1, pp. 9�20, 2006.

D. Sorge and J. Russell, ‘‘Properly designed workshops distributed over an academic year’’, Educ. Technol.,

40, pp. 46�48, 2000.

D. Sparks and Loucks-S. Horsley, ‘‘Five models of professional development for teachers’’, J. Staff Dev.,

10, pp. 40�57, 1989.

T. Sumner and M. Marlino, ‘‘Digital libraries and educational practice: a case for new models’’, in

Proceedings of the Joint Conference on Digital Libraries, 2004, pp. 170�178.

R. Wallace, ‘‘A framework for understanding teaching with the Internet’’, Am. Educ. Res. J., 41,

pp. 447�488, 2004.

A. Walker and H. Leary, ‘‘Problem based learning: a meta analytic review of problem and implementation

types across disciplines and educational levels’’, in Association of Educational Communications and

Technology, Anaheim, CA, 2007, October.

D. Wiley, M. Recker and A. Gibbons, ‘‘In defense of the by-hand assembly of learning objects’’, 2000.

Available online at: from http://wiley.ed.usu.edu/writings/ (accessed 23 May 2005).

L. Zia, ‘‘Growing a national learning environments and resources network for science, mathematics,

engineering, and technology education: current issues and opportunities for the NSDL program’’, 7(11),

2001. Available online at: from http://www.dlib.org/dlib/march01/zia/03zia.html (accessed 17 July 2003).


Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 18:

23 1

0 O

ctob

er 2

014

Documents

A study of teachers’ use of online learning resources to design classroom activities