PROFESSIONAL LEARNING IN INFORMAL ONLINE ENVIRONMENTS: EXAMINING

ELEMENTARY TEACHERS’ COGNITIVE PROCESSES AND LEARNING EXPERIENCES

AS THEY NAVIGATE A PROFESSIONAL DEVELOPMENT WEBSITE

By

Pamela-Jane Kathleen Beach

A thesis submitted in conformity with the requirements

for the degree of Doctor of Philosophy

Graduate Department of Applied Psychology and Human Development

Ontario Institute for Studies in Education of the

University of Toronto

© Copyright by Pamela-Jane Kathleen Beach 2015

ii

PROFESSIONAL LEARNING IN INFORMAL ONLINE ENVIRONMENTS:

EXAMINING ELEMENTARY TEACHERS’ COGNITIVE PROCESSES AND

LEARNING EXPERIENCES AS THEY NAVIGATE A PROFESSIONAL

DEVELOPMENT WEBSITE

Doctor of Philosophy, 2015

Pamela-Jane Kathleen Beach

Department of Applied Psychology and Human Development

University of Toronto

Abstract

This thesis is comprised of two studies. The first is a mixed methods study comparing three

types of think aloud methods for examining elementary teachers’ cognitive processes as they

use a multimedia professional development website. Forty-five elementary teachers were

assigned to one of three think aloud conditions—concurrent, retrospective, and virtual revisit.

Participants in the concurrent condition verbalized their thoughts while simultaneously

navigating the website for 20 minutes; participants in the retrospective condition verbalized their

thoughts immediately following their 20-mintue navigation without any aids; and participants in

the virtual revisit condition verbalized their thoughts while viewing a screen recording of their

20-minute navigation. Results revealed differences (p < .05) in the complexity of verbalizations

produced by participants in each condition, in task completion, and in reported comfort levels

with thinking aloud.

The findings of study 1 revealed that participants from the virtual revisit condition

produced extensive complex verbalizations reflecting higher cognitive thought processes.

Therefore, a second study was conducted using qualitative methods. Specifically, a grounded

theory approach to analysis provided an in-depth understanding of the learning experiences of

participants from the virtual revisit condition as they navigated the given website. A theoretical

model was developed describing: (1) causal conditions that affect elementary teachers’

navigation of a professional development website, (2) the central phenomenon, navigating a

iii

professional development website, (3) navigational strategies, and (4) potential learning

outcomes.

Overall, the findings of this thesis provide insight into the benefits and limitations of

each think aloud method and of elementary teachers’ cognitive processes and learning

experiences during informal online professional learning. The results also highlight the need for

further investigations of teacher cognition and online learning using the virtual revisit think

aloud method. Educational and research implications are discussed for both studies.

iv

Acknowledgements

I would like to express my sincere gratitude to my thesis supervisor, Dr. Dale Willows

for the endless encouragement, guidance, and advice. She contributed immensely to this

thesis—by conceiving significant aspects of the virtual revisit think aloud, as well as providing

insightful comments and questions to refine my thoughts throughout the entire process. It has

been an honour to work under Dr. Willows’ guidance. Her dedication to teaching, learning, and

knowledge mobilization is inspiring.

I would like to thank my committee members, Dr. Rhonda Martinussen and Dr. Joan

Peskin, for their commitment, support, and valuable feedback during the entire process. I would

also like to express my gratitude to Dr. Vera Woloshyn for her willingness to undertake the role

of the external examiner, and for her insightful comments and questions. As well, I would like

to thank Dr. Janette Pelletier for her time and thoughtfulness.

I am indebted to Ronna Kluger and Robin Bennett for their ongoing support,

encouragement, and assistance during data collection. I would like to thank Jayme Herman for

her role as the second coder during the initial stages of coding, and for her insights and advice

during analysis. I am thankful to Dr. Yiola Cleovoulou for her continued encouragement,

guidance, and for sharing her knowledge of qualitative research with me. I would also like to

thank Dr. Eunice Jang and Dr. Olesya Falenchuk—the knowledge that I gained from their

research methods courses was instrumental in carrying out my analyses.

I would like to acknowledge the incredible Balanced Literacy Diet website team for their

dedication to the development of an amazing online teacher resource. Your support and

friendship are invaluable. Thank you to Madison Aitken, Kate Bryant, Lesley Dookie, Julia

Forgie, Kasia Kania, Nathalie Rothschild, and Taj Uppal.

v

Thank you to all of my friends and family for their positive energy, interest, and ongoing

support. I am especially thankful to my parents, Ross and Jane Beach for their enthusiasm,

endless encouragement, and occasional editing. And to Dani Oore—thank you for listening to

me think aloud about my research, for your thoughtfulness, support, and kind heart.

Finally, I would like to thank the teachers who participated in this research. Their

willingness to share their thoughts and experiences contributed to the understanding of teacher

learning.

vi

Table of Contents

Abstract ............................................................................................................................................ ii

Acknowledgements......................................................................................................................... iv

Table of Contents ............................................................................................................................ vi

List of Tables ................................................................................................................................... x

List of Figures ................................................................................................................................. xi

List of Appendices ......................................................................................................................... xii

CHAPTER 1—General Introduction .......................................................................................... 1

Introduction to the Studies ..................................................................................................... 1

Thesis Overview ..................................................................................................................... 6

CHAPTER 2—General Methodology.......................................................................................... 8

Context ................................................................................................................................... 8

Participant Recruitment ........................................................................................................ 11

Data Sources ......................................................................................................................... 12

Demographic Questionnaire ............................................................................................ 12

Think Aloud ..................................................................................................................... 15

Screen Capture Technology ............................................................................................. 16

Semi-structured Interview ............................................................................................... 16

Procedure .............................................................................................................................. 16

One-on-One Meeting ....................................................................................................... 16

CHAPTER 3—Study 1: Comparing Three Think Aloud Methods for Examining

Teachers’ Cognitive Processes as They Use a Multimedia Professional Development

Website.......................................................................................................................................... 19

Literature Review ................................................................................................................. 19

Think Aloud Methodology .............................................................................................. 19

Teacher Cognition ........................................................................................................... 23

Research Questions .............................................................................................................. 25

Methods ................................................................................................................................ 25

Research Design .............................................................................................................. 25

Data Analysis ................................................................................................................... 28

Data Review Process ....................................................................................................... 38

Results and Discussion ......................................................................................................... 38

vii

Research Question 1: To What Extent Do Participants’ Verbalizations Differ Across the

Three Think Aloud Methods? .......................................................................................... 39

Summary of Results and Discussion for Research Question 1........................................ 52

Research Question 2: How Do Think Aloud Tasks Differ Across the Three Think Aloud

Conditions? ...................................................................................................................... 52

Summary of Results and Discussion for Research Question 2........................................ 60

Research Question 3: To What Extent Does Comfort Level with Thinking Aloud Differ

Across the Three Think Aloud Methods? ........................................................................ 61

Summary of Results and Discussion for Research Question 3........................................ 66

General Discussion of Study 1 ............................................................................................. 67

A Participant’s Experience with the Concurrent Think Aloud Method .......................... 68

A Participant’s Experience with the Retrospective and Virtual Revisit Website Task ... 70

A Participant’s Experience with the Retrospective Think Aloud .................................... 71

A Participant’s Experience with the Virtual Revisit Think Aloud .................................. 71

Benefits and Limitations of Employing Each Type of Think Aloud Method ................. 72

Study 1 Limitations ......................................................................................................... 75

Significance and Educational Implications ..................................................................... 76

Conclusion ............................................................................................................................ 78

CHAPTER 4—Understanding Teachers’ Learning Experiences as They Use a Professional

Development Website: A Qualitative Study .............................................................................. 79

Theoretical Models for Understanding Teacher Learning ................................................... 81

Knowles’ Theory of Adult Learning ............................................................................... 81

Self-Directed Learning Theory ........................................................................................ 84

Shulman and Shulman’s Model of Teacher Learning ..................................................... 86

Formal and Informal Professional Development ................................................................. 88

Study Purpose and Research Questions ............................................................................... 91

Methods ................................................................................................................................ 92

Participants ...................................................................................................................... 92

Research Design .............................................................................................................. 94

Data Analysis ................................................................................................................... 96

Establishing Trustworthiness ......................................................................................... 101

Findings and Discussion ..................................................................................................... 105

Conditions Affecting Elementary Teachers’ Website Navigation ..................................... 108

Theme 1: Perception of Professional Learning .................................................................. 108

viii

1. Quality of Professional Resources ............................................................................. 108

2. Beliefs about Literacy Instruction ............................................................................. 112

Theme 2: Focusing on Student Needs and Instructional Goals.......................................... 114

1. Student Needs ............................................................................................................ 115

2. Classroom Context .................................................................................................... 115

3. Current and Future Instructional Goals ..................................................................... 116

Theme 3: Individual Differences ........................................................................................ 116

1. Motivating Factors to Use Online Resources ............................................................ 117

2. Online User Behaviour .............................................................................................. 119

3. Prior Experience and Professional Knowledge ......................................................... 122

The Central Phenomenon Resulting from the Conditions.................................................. 124

Theme 4: Navigating a professional development website ................................................ 124

1. Gaining a General Sense of the Website ................................................................... 125

2. Exploring In-Depth .................................................................................................... 125

3. Understanding the Website ........................................................................................ 126

Strategies for Navigating a Professional Development Website........................................ 127

Theme 5: Evaluating Information. ..................................................................................... 128

1. Content Richness ....................................................................................................... 129

2. Functionality of the Technology ................................................................................ 129

3. Extent of Technology Integration .............................................................................. 130

4. Novelty of the Technology and Uniqueness of the Learning Environment .............. 131

5. Potential for Learning ................................................................................................ 131

Theme 6: Encoding Information for Future Retrieval........................................................ 132

1. Note-Taking ............................................................................................................... 132

2. Saving Ideas for Future Use ...................................................................................... 133

Potential Outcomes Resulting from Elementary Teachers’ Website Navigation .............. 134

Theme 7: Reflecting on Learning ....................................................................................... 134

1. General Feelings about the Website .......................................................................... 135

2. Future Use of the Website ......................................................................................... 135

Theme 8: Continued Professional Learning ....................................................................... 136

1. Advantages of Online Professional Learning ............................................................ 137

2. Increased Self-Efficacy .............................................................................................. 137

3. Feeling Inspired ......................................................................................................... 139

ix

4. Knowledge Construction ........................................................................................... 140

Theme 9: Intentions for Practice. ....................................................................................... 140

1. Adapting Information ................................................................................................ 141

2. Professional Goals ..................................................................................................... 142

General Discussion ............................................................................................................. 143

Overview of Major Findings .............................................................................................. 144

Key Conditions .............................................................................................................. 144

Key Navigational Strategies .......................................................................................... 145

Key Potential Outcomes ................................................................................................ 145

Study 2 Limitations ............................................................................................................ 146

Significance and Future Directions .................................................................................... 148

CHAPTER 5—General Conclusions ....................................................................................... 151

References.................................................................................................................................... 153

Appendices .................................................................................................................................. 169

x

List of Tables

Table 2.1 Demographic Characteristics………………………………..………………….…….14

Table 3.1 Average Word Counts across the Three Conditions………………………………….29

Table 3.2 Coding Scheme-Categories Used to Code the Transcripts…………………….……..30

Table 3.3 Average Number of Thought Units across the Three Conditions……………………33

Table 3.4 Frequency of Thought Units across Conditions…………………….…….………….34

Table 3.5 Time Sampling Analysis-Variables………………………….……………………….36

Table 3.6 ANOVA Summary Statistics for Thought Units………………….……….………....40

Table 3.7 ANOVA Summary Statistics for Word Count and Number of Thought Units……....48

Table 3.8 Pearson Correlations…………………….…………………………………………....51

Table 3.9 ANOVA Summary Statistics for Web-Based Action Variables……………………..54

Table 3.10 Themes Related to Reported Comfort Level………………………………………..63

Table 3.11 Benefits and Limitations of Employing Each Type of Think Aloud Method……....74

Table 4.1 Demographic Characteristics for Study 2…………………………………………….93

Table 4.2 Study 2 Themes……………………………………………………………………..105

Table 4.3 General Agreement of What Constitutes Best Practices in Literacy Instruction……114

Table 4.4 Song and Lee’s Criteria for Evaluating Informal Online Environments……………128

xi

List of Figures

Figure 2.1 Homepage of The Balanced Literacy Diet website

Figure 3.1 Convergence mixed methods research design

Figure 3.2 Topics pertaining to research question 1 results and discussion

Figure 3.3 Topics pertaining to research question 2 results and discussion

Figure 3.4 Topics pertaining to research question 3 results and discussion

Figure 3.5 Topics pertaining to the general discussion of study 1

Figure 4.1 Outline of overall process for study 2

Figure 4.2 Topics pertaining to establishing trustworthiness

Figure 4.3 Theoretical model for elementary teachers’ learning experiences as they navigated a

professional development literacy website

Figure 4.4 Topics pertaining to theme 1: Perception of professional learning

Figure 4.5 Topics pertaining to theme 2: Focusing on student needs and instructional goals

Figure 4.6 Topics pertaining to theme 3: Individual differences

Figure 4.7 Topics pertaining to theme 4: Navigating a professional development website

Figure 4.8 Topics pertaining to theme 5: Evaluating information

Figure 4.9 Topics pertaining to theme 6: Encoding information for future retrieval

Figure 4.10 Topics pertaining to theme 7: Reflecting on learning

Figure 4.11 Topics pertaining to theme 8: Continued professional learning

Figure 4.12 Topics pertaining to theme 9: Intentions for practice

xii

List of Appendices

Appendix A-1 Sample Pages of The Balanced Literacy Diet Website

Appendix A-2 Invitation to Participate Script

Appendix A-3 Information Letter and Consent Form

Appendix A-4 Recruitment Poster

Appendix A-5 Demographic Characteristics across the Three Different Modes of Recruitment

Appendix A-6 Demographic Questionnaire

Appendix A-7 Think Aloud and Interview Consent Form

Appendix A-8 Screen Shot of Camtasia Studio During a Participant’s Navigation

Appendix A-9 Interview Questions

Appendix B-1 Initial List of Potential Categories for Coding Think Aloud Transcripts

Appendix B-2 Krathwohl’s Revised Taxonomy

Appendix B-3 Frequency Counts of Web-based Actions

Appendix B-4 Sample Observations of Web-based Actions with Think Aloud Recordings

Appendix B-5 Data Screening Results and Descriptives

Appendix C-1 Initial List of Categories for Study 2

Appendix C-2 Sample of Memos Written During Analysis

Appendix C-3 Themes and Descriptions Sent to Participants Involved in Member Check

Appendix C-4 Participants’ ‘Member Check’ Comments

Appendix C-5 Frequency of Themes across Data Sources and across Participants

Appendix C-6 Samples from Audit Trail

1

CHAPTER 1

General Introduction

This thesis presents two studies that investigate the cognitive processes and learning

experiences of elementary teachers as they use a multimedia professional development website.

The first investigation reports on the results of a mixed methods study that compares the relative

utility of three think aloud methods for examining elementary teachers’ cognitive processes as

they use a professional development website. Based on the findings of study 1, the second study

involves an in-depth investigation of elementary teachers’ online professional learning

experiences. Below is a brief introduction to the studies.

Introduction to the Studies

Elementary teachers are a necessary foundation for building successful programs in the

classroom (Gambrell & Anders Mazzoni, 1999; Pressley, Mohan, Raphael, & Fingeret, 2007).

Successful programs begin with a repertoire of pedagogical knowledge, pedagogical content

knowledge, and research-based instructional practices. This repertoire of information can be

delivered to practicing elementary teachers through various professional development

opportunities (Cervetti, Kulikowich, Drummond, & Billman, 2012; Desimone, 2009; Kao, Wu,

& Tsai, 2011).

One facet of teacher professional development is online learning, which occurs when

professional knowledge is constructed from multiple modes of digital information—

photographs, videos, and interactive tools, to name a few (Mayer, 2002). Online learning is a

favoured approach to professional development because it creates accessible opportunities;

online learning takes place within platforms that deliver information in a means that removes

time, place, and situational barriers (Kanuka & Nocente, 2003). As elementary teachers

increasingly turn to the Internet for their professional learning (Charalambousa & Ioannou,

2

2011; Kao, Wu, & Tsai, 2011), it is essential to examine how they use and learn from online

resources and professional development websites.

Teacher learning is a complex process that involves a range of cognitive activities,

decision-making strategies, and learning experiences. Studying teachers’ cognitive processes

and learning experiences in the context of online professional development has several

educational and research implications. The study of how teachers use online learning

environments can provide educational website developers with guidance in the design and

framework of professional development tools and technologies. Study outcomes can inform

educational website developers and administrators of professional development programs about

elementary teachers’ motivations to use professional development websites, their beliefs and

attitudes towards these learning platforms, and potential learning outcomes. In addition,

studying teacher online learning can lead to greater insight into teachers’ experiences within

these online environments. Greater insight into teachers’ cognitive processes and online

learning experiences can contribute to a holistic and multifaceted view of teacher professional

development.

Most studies that have examined online teacher learning have gathered data through

surveys, questionnaires, and interviews (Duncan-Howell, 2010; Hur & Brush, 2009; Kao et al.,

2011). These methods offer information about teachers’ attitudes towards online professional

learning; however, data generated from these methods is limited to participants’ recollection of

past events. A method that tracks teachers’ cognitive processes as they make online choices is

necessary to provide further insight into how teachers use and learn from online environments.

The think aloud methodology is an approach that can track teachers’ cognitive processes during

decision-making activities.

3

The think aloud methodology is based on the techniques of protocol analysis by Ericsson

and Simon (1984, 1993) and has been shown to provide direct data about the reasoning process

(van Someren, Barnard, & Sandberg, 1994). Verbal reports provide information about

experiences and knowledge that cannot easily be accessed by more traditional observations and

interviews (Ericsson & Simon, 1993). Two variations of the think aloud most often employed

by researchers are the concurrent think aloud and the retrospective report. During the

concurrent think aloud participants complete a task while they simultaneously verbalize their

thoughts. The retrospective report requires participants to verbalize their thoughts after they

complete a given task. Studies that have successfully employed the concurrent think aloud

method in the context of online learning have focused on how undergraduate university students

and elementary school students use and evaluate educational websites (Armstrong, 2011;

Barzilai & Zohar, 2012; Damico & Balidon, 2007); however, few studies that employ the

concurrent think aloud method focus on elementary teacher participants. The retrospective

report on its own has received little attention in the field of online learning, and research

comparing the concurrent and retrospective think aloud methods as techniques for assessing

online learning is limited (van den Haak, de Jong, & Schellens, 2003). Moreover, several

studies show that both the concurrent think aloud and retrospective report have weaknesses—the

concurrent think aloud has the disadvantage of completing two tasks simultaneously while the

retrospective think aloud has the disadvantage of thinking aloud after the task has been

completed (Branch, 2006; Kuusela & Paul, 2000; McDonald, Edwards, & Zhao, 2012; van Gog,

Kester, Nievelstein, Giesbers, & Paas, 2009). Findings of such studies suggest that alternative

think aloud methods warrant attention.

To address these gaps in the literature, study 1 compares three think aloud methods—

concurrent, retrospective, and virtual revisit—for examining elementary teachers’ cognitive

4

processes as they use a multimedia professional development website. The virtual revisit is a

variation of the retrospective think aloud method and allows participants to review and comment

on a visual recording of how they interacted with a particular website. The goal of the virtual

revisit is to aid recall of original events and thought processes by using a screen-capture

recording of participants’ navigational experiences. Similar to cued retrospective reporting

where participants are given instructions to report retrospectively on the basis of a record of

observations (van Gog, Paas, van Marrienboer, & Witte, 2005), the virtual revisit think aloud

combines a retrospective think aloud with screen capture technology to aid recall of original

events and thought processes.

Study 1 involves 45 elementary teachers from the Greater Toronto Area. Participants

were assigned to one of three think aloud conditions—concurrent, retrospective, and virtual

revisit. They were all asked to use the website as they would normally do when seeking

information related to their teaching practices but the three groups were asked to think aloud

about their site navigation under different conditions. Participants in the concurrent condition

verbalized their thoughts while simultaneously navigating the website for 20 minutes;

participants in the retrospective condition verbalized their thoughts immediately following their

20-minute navigation without any aids; and participants in the virtual revisit condition

verbalized their thoughts while viewing a screen recording of their 20-minute navigation.

Comparisons were made between concurrent, retrospective, and virtual revisit think aloud

procedures. Think aloud protocols, screen recordings, and interview data were analyzed using

quantitative and qualitative methods. Study 1 results show differences in the types and

complexity of verbalizations produced by participants in each condition (p < .05). Furthermore,

results reveal differences in website task completion and participants’ reported comfort levels

with thinking aloud. Overall findings of study 1 show benefits and limitations to each type of

5

think aloud method in the context of online teacher professional development. Findings also

suggest that the virtual revisit think aloud method provides a comprehensive picture of

elementary teachers’ cognitive processes as they use a professional development website.

Following findings from study 1, study 2 examines data from the virtual revisit condition

(n = 15). The findings of study 1 showed that participants assigned to the virtual revisit think

aloud produced rich and thorough verbalizations related to higher cognitive processes (e.g.,

planning, reasoning, reflecting). Therefore, a second study was conducted to analyze the think

aloud transcripts, interview data, and questionnaire data from the virtual revisit condition.

While research that examines teacher professional development is extensive and includes

multiple contexts—course instruction (Podhajski, Mather, Nathan, & Sammons, 2009), in-

school professional communities (Porche, Pallante, & Snow 2012), and web-based learning

environments (Downer et al., 2012)—research that examines the learning processes and

experiences of elementary teachers as they use informal professional development websites is

limited (Bakkenes, Vermunt, & Wubbels, 2010; Downer et al., 2012; Hoekstra, Brekelmans,

Beijaard, & Korthagen, 2009; Vermunt & Endedijk, 2011). Therefore, study 2 utilizes the

virtual revisit think aloud method and two other data sources (interview and questionnaire data)

to examine elementary teachers’ learning experiences as they navigate the given website. This

topic warrants attention for two main reasons: elementary teachers are increasingly using the

Internet for their professional learning, and “to understand teacher learning we must study it

within multiple contexts” (Borko, 2004, p. 4).

Study 2 utilizes qualitative methods to analyze the think aloud transcripts, interview

data, and questionnaire data from the virtual revisit condition. Specifically, study 2 follows a

grounded theory approach to gain an in-depth understanding of 15 participants’ learning

experiences as they navigate The Balanced Literacy Diet website. Three phases of coding—

6

open coding, axial coding and selective coding—led to a theoretical model describing: (1)

conditions that affect how elementary teachers use and learn from a professional development

website, (2) the central phenomenon, navigating a professional development website, as

influenced by the conditions, (3) navigational strategies, and (4) potential learning outcomes.

Thesis Overview

The remainder of this thesis includes four chapters. Chapter 2 provides the general

methodology relevant to both study 1 and study 2. The context, participant recruitment,

demographic characteristics, data sources, and procedure are similar for study 1 and study 2;

therefore, to reduce redundancy as much as possible, the second chapter presents the general

methodology for this thesis.

Chapter 3 presents study 1, an empirical study that compares the relative utility of three

think aloud methods: concurrent, retrospective, and virtual revisit. First, Chapter 3 discusses

two main topics relevant to study 1: (1) the think aloud methodology, and (2) teacher cognition

and cognitive processes. Chapter 3 continues with a description of the study purpose and

research questions, the relevant methods, and the results and discussion. Chapter 3 concludes

with a general discussion, including study 1 limitations, significance and implications, and

future directions.

Based on the findings of study 1, Chapter 4 presents study 2, an in-depth qualitative

study that examines the online learning experiences of participants from the virtual revisit think

aloud condition. First, Chapter 4 describes three theoretical models for understanding teacher

learning: adult learning theory, self-directed learning, and Shulman and Shulman’s theory of

teacher learning (2004). Components of each learning theory are discussed with respect to

online teacher professional development. Chapter 4 continues with a description of the study

purpose and research questions for study 2, the relevant methods for study 2, and the findings

7

and discussion. Chapter 4 concludes with a general discussion, including study limitations,

significance and implications, and future directions.

Chapter 5, the final chapter of the thesis, presents a discussion of the general conclusions

derived from the two studies. The final chapter also reflects on the research and educational

implications of the thesis as a whole.

8

CHAPTER 2

General Methodology

This chapter describes the general methodology for study 1 and study 2. As previously

mentioned, the context, participant demographic characteristics, data sources, and procedure are

similar for both studies; thus the purpose of including a general methodology chapter is to

describe the common components of the study 1 and study 2 method sections. Chapter 3 and

chapter 4 will describe details of the methods pertaining to each study, including the research

design and data analysis. This chapter begins with a description of the context of the current

dissertation, The Balanced Literacy Diet website, and highlights the website’s main features and

various web pages. Then, the chapter describes participant recruitment. The next section

presents a description of the four data sources: demographic questionnaire, think aloud, screen

capture technology, and semi-structured interview. This chapter ends with an outline of the

procedure.

Context

This dissertation research was undertaken within the context of the development and

refinement of a professional development website, The Balanced Literacy Diet: Putting

research into practice in the classroom (www.LitDiet.org). This website is a multimedia

evidence-informed literacy website that provides free professional learning resources such as

virtual tours of exemplary classrooms (PreK-6), video clips of expert teachers explaining and

demonstrating effective educational practices, detailed lesson plans, photos of teaching

materials, and exemplars of student work. In addition, The Balanced Literacy Diet website uses

a healthy diet framework as a metaphor for the various components of literacy at different stages

of development. These literacy components are consistent with The National Reading Panel’s

(2000) report on evidence-based literacy instruction. The report concluded that high quality

effective components of literacy instruction include phonological awareness, phonics, fluency,

9

vocabulary, and comprehension strategies. Reading researchers generally agree on these

components of literacy practices (Cunningham, Zibulsky, Stanovich, & Stanovich, 2009) and

several studies have shown a positive relationship between these components and student

growth in reading and writing (Downer et al., 2011; Porche et al., 2012; Pressley et al., 2007).

Taken together, The Balanced Literacy Diet website is content rich, interactive, and research-

informed; thus it offers an ideal context in which to examine elementary teachers’ use of

professional development websites while, at the same time, generating data which can serve to

further develop the effectiveness of the site as an online professional resource.

Figure 2.1 shows the homepage of The Balanced Literacy Diet website. The homepage

provides visitors of the site with five tabs from which to choose at the top of the page:

Homepage, Food Groups, Recipe Finder, Virtual Tours, and How To Videos. The homepage

also provides visitors with an overview of the site, sample virtual tours and lesson plans, and

relevant external links.

Figure 2.1

Homepage of The Balanced Literacy Diet website

10

The Food Groups tab brings site users to an introductory page on the different

components of literacy that contribute to a balanced literacy program. Site visitors can access

each literacy component, (e.g., phonemic awareness, reading comprehension, text structures and

genres) from the drop down Food Groups menu. The food group pages provide a combination

of theoretical and practical information on each component of literacy. The site visitor can

access this information visually, auditorily and textually.

From the Recipe Finder tab, site visitors can choose lessons and activities that range

from pre-kindergarten through sixth grade. Following the healthy diet theme, the lessons and

activities are referred to as recipes. Recipes include photos of instructional material and student

work, videos of the teacher explaining or demonstrating the lesson, and the activity objective

and implementation. For further reading, site visitors can also access additional resources from

the recipes, including journal articles and teacher guide books.

Site visitors can also visit classrooms virtually, through the Virtual Tour tab. Within

each virtual tour, users can navigate around a virtual classroom—they can zoom in to view

close-ups of teacher material and student work, and click on dots or “hot spots” to hear from the

classroom teacher about a specific lesson or area of the classroom.

The How To Videos tab brings users to the website’s YouTube channel. The Balanced

Literacy Diet YouTube channel provides over 500 videos of teachers explaining and

demonstrating literacy activities.

In addition, site visitors can access content pages relevant to their role (i.e., teachers,

administrators, parents, and teacher educators). These content pages include pedagogical

information and relevant external links. Appendix A-1 contains sample screen shots of each

type of webpage.

11

Participant Recruitment

Forty-five Toronto area practicing elementary teachers participated in this research over

an eight-month period. Participants were recruited through visits to Additional Qualifications

(AQ) courses, recruitment posters, and by word of mouth. First, elementary teachers were

recruited through AQ courses at the Ontario Institute for Studies in Education (OISE). A

request to contact AQ course instructors was sent in an email to the OISE AQ office. The email

described the study purpose and participant criteria (e.g., grades taught ranging from

kindergarten to grade six, practicing teachers). The AQ office replied with a contact list of 15

AQ course instructors. Course instructors were contacted by email and asked for permission to

attend one of their classes to invite teachers to participate in the research. Appendix A-2

presents the invitation to participate script. Ten instructors replied to the email. Seven agreed to

a course visit and three agreed to email the information to the teachers in their course. The

course visits involved a verbal explanation of the research purpose and participant involvement,

and a request for teachers to take part in the study. Interested teachers were given an envelope

with an information letter and consent form explaining the research in greater detail. Appendix

A-3 contains the information letter and consent form. Interested teachers were asked to include

their email address on the consent form so that the questionnaire link could be emailed to them.

Envelopes were collected at the time of the visit and at the end of the class. Instructors who

were willing to forward the information to the teachers enrolled in their course were sent the

explanation of the research purpose and the description of participant involvement. The total

number of teachers contacted through AQ courses was approximately 200 (with approximately

20 teachers per class). Many teachers initially showed interest in participating during the course

visits. However, given the criteria for participation and the reality that practicing teachers are

12

very busy, the response rate was low; only four teachers became study participants despite a $20

gift card incentive.

Due to the difficulty recruiting practicing primary and junior level teachers through AQ

course visits, recruitment also occurred through study posters and by word of mouth. Study

posters were displayed at OISE and sent via email by colleagues interested in the research.

Appendix A-4 presents the recruitment poster. Potential participants were also contacted by

teachers who completed the study. Of the 45 teachers who agreed to participate 28 were

recruited through the recruitment posters, four were recruited from an AQ course, and 13

teachers were recruited by word of mouth. Demographic characteristics are described below.

Due to the challenges of participant recruitment, a non-random sample was used in this

research. A distribution chart shows the demographic characteristics across the three different

modes of recruitment (see Appendix A-5).

Data Sources

This research uses four different data sources: a demographic questionnaire, think aloud,

screen capture technology, and a semi-structured interview.

Demographic Questionnaire

A demographic questionnaire (see Appendix A-6) was administered to participants to

obtain data on a range of relevant factors based on the literature related to teacher development

and online learning (e.g., age, gender, years of teaching experience, education, extent of

involvement with various web-based technologies). Participants were asked to complete the

questionnaire online prior to a one-on-one meeting. The questionnaire was administered

through Survey Planet (https://www.surveyplanet.com/), an online survey program. After

participants completed the questionnaire, they were placed in one of three think aloud conditions

(concurrent, retrospective, and virtual revisit). The conditions are described below. Stratified

13

random assignment was employed to ensure that certain demographic features were represented

within each group as equally as possible. Stratification variables were selected based on the

literature on web navigation. Research has found that gender (Page, Robson, & Uncles, 2012; J.

Pearson, A. Pearson, & Green, 2007), age (Laberge & Scialfa, 2005), subject matter knowledge

and experience (Laberge & Scialfa, 2005), and computer self-efficacy (Page et al., 2012)

influence the perception and navigation of websites; therefore, comparable aspects including

gender, age, years of teaching experience, current teaching grade, comfort with technology, and

frequency of Internet use for professional purposes, were selected as the key variables used for

the group assignment. A one-way ANOVA was conducted to compare the three conditions to

determine if the conditions had any significant differences with respect to these key variables.

The conditions did not differ significantly on any of these key variables.

The results of the demographic questionnaire revealed that, in the sample overall, male

participants (n = 9, 20%) represented a smaller proportion of the sample than female participants

(n = 36, 80%). This finding is consistent with the proportion of few male elementary teachers

in primary and junior positions (Jamieson, 2007). The participants represented a range of age

and years of experience with the majority of teachers being 30-34 years of age (n = 34, 27%)

and most teachers having 5-9 years of experience (n = 17, 38%). Grades taught ranged from

kindergarten through sixth grade. Two teachers (4%) taught kindergarten, 17 teachers (38%)

taught at the primary level, 19 teachers (42%) taught at the junior level, and seven teachers

(16%) taught across divisions, at the primary and junior levels.

Results from the demographic questionnaire revealed similarities across participants in

terms of comfort using the Internet and frequency of Internet use for professional purposes. The

majority of teachers reported feeling very comfortable using the Internet (n = 41, 91%).

14

Furthermore, the majority of teachers reported using the Internet for professional purposes at

least once a day (n = 21, 47%).

Table 2.1 summarizes the demographic characteristics for the participants across the

conditions. As can be seen from the table, participants in the three groups were quite similar

with respect to gender, age, years of teaching, current grade, comfort with technology, and

frequency of Internet use. In other words, the stratified random assignment was successful. As

mentioned above, statistical tests found no significant differences between groups on these

variables.

Table 2.1

Demographic Characteristics

Characteristic Concurrent

Condition

(n = 15)

Retrospective

Condition

(n = 15)

Virtual

Revisit

Condition

(n = 15)

Total

(N = 45)

n (%) n (%) n (%) n (%)

Gender

Male 3 (20%) 3 (20%) 3 (20%) 9 (20%)

Female 12 (80%) 12 (80%) 12 (80%) 36 (80%)

Age

25-29 3 (20%) 3 (20%) 3 (20%) 9 (20%)

30-34 3 (20%) 5 (33%) 4 (27%) 12 (27%)

35-39 4 (27%) 3 (20%) 4 (27%) 11 (24%)

40-44 2 (13%) 2 (13%) 2 (13%) 6 (13%)

45-49 1 (6%) 1 (6%) 1 (6%) 3 (7%)

50-54 2 (13%) 0 0 2 (4%)

55+ 0 1 (6%) 1 (6%) 2 (4%)

Years

Teaching

2-4 years 3 (20%) 3 (20%) 4 (27%) 10 (22%)

5-9 years 6 (40%) 6 (40%) 5 (33%) 17 (38%)

10-14 years 3 (20%) 5 (33%) 4 (27%) 12 (27%)

15-19 years 1 (6%) 0 1 (6%) 2 (4%)

20-24 years 1 (6%) 0 1 (6%) 2 (4%)

25+ years 1 (6%) 1 (6%) 0 2 (4%)

Current

Grade

Kindergarten 1 (6%) 1 (6%) 0 2 (4%)

Primary 5 (33%) 5 (33%) 7 (47%) 17 (38%)

Junior 7 (47%) 6 (40%) 6 (40%) 19 (42%)

15

Primary/Junior 2 (13%) 3 (20%) 2 (13%) 7 (16%)

Comfort

Using

Internet

Very 15 (100%) 13 (87%) 13 (87%) 41 (91%)

Somewhat 0 2 (13%) 2 (13%) 4 (9%)

Not very 0 0 0 0

Frequency of

Internet Use

for

Professional

Purposes

> once/day 5 (33%) 3 (20%) 6 (40%) 14 (31%)

Once/day 6 (40%) 8 (53%) 7 (47%) 21 (47%)

Once/week 4 (27%) 4 (27%) 1 (6%) 9 (20%)

Once/month 0 0 1 (6%) 1 (2%)

Think Aloud

Participants completed a think aloud during a one-on-one meeting with the primary

researcher. A digital recorder was used to audio record participants’ verbalizations.

Participants were asked to complete a second consent form giving permission to record the think

aloud and interview (see Appendix A-7).

Concurrent think aloud. Participants in the concurrent condition verbalized their

thoughts for 20 minutes while simultaneously completing a website task.

Retrospective think aloud. Immediately following a 20-minute website task,

participants in the retrospective condition recalled and verbalized their thought processes

without any aids.

Virtual revisit think aloud. Immediately following a 20-minute website task,

participants in the virtual revisit condition reviewed their online choices virtually and verbalized

their thoughts while viewing the 20-minute screen recording of their explorations.

16

Screen Capture Technology

During participants’ navigation of The Balanced Literacy Diet website, each visual step

was captured with Camtasia Studio, a screen-recording computer software program developed

by TechSmith (Uppal, 2011). Appendix A-8 displays a screen shot of Camtasia Studio that

includes an image of a screen recording of one participant’s navigation. The purpose of using

screen capture technology was to gain greater insight into participants’ web-based actions and

behaviours. Web tracking through screen-recordings offers a deeper understanding of

information-seeking phenomena on the Internet (Fourie & Bothma, 2007).

Semi-structured Interview

A semi-structured interview (see Appendix A-9) followed the participants’ website

navigations. A digital recorder was used to record the interviews. Questions concerned

participants’ general feelings about their navigational experiences as well as their comfort levels

while thinking aloud, whether verbalizing their thoughts concurrently, retrospectively or via the

virtual revisit.

Procedure

One-on-One Meeting

Upon completion of the questionnaire, participants met with the primary research for a

one-on-one meeting. Meetings occurred at a time and location that was most convenient to the

participants (e.g., participant’s school or home, OISE). An open location led to an increase in

teacher interest and willingness to participate in the study. The one-on-one meetings followed a

sequence of events and lasted approximately 45 minutes. First, the primary research restated the

purpose of the research. Next, participants completed the website task and think aloud. Finally,

the semi-structured interview was conducted.

17

Website task and think aloud. The following website task instructions were presented

to all participants:

Your task is to use The Balanced Literacy Diet website as you normally would when

seeking information online for your teaching practices.

While the website task instructions were consistent across the conditions, the think aloud

instructions varied for each condition. Participants in the concurrent condition were given the

think aloud instructions before they completed the website task, whereas participants in the

retrospective and virtual revisit groups were given the think aloud instructions after they

completed the website task. The purpose of informing participants in the retrospective and

virtual revisit conditions of the think aloud instructions after their navigation was to reduce

reactivity—“influences of the verbalizations on the decision process” (Ranyard & Svenson,

2010, p. 119)—as much as possible. The following passage outlines the think aloud

instructions. The underlined portions state the different think aloud instructions given for each

condition.

In this study we are interested in what you think about when you explore a professional

development website. In order to do this I am going to ask you to think aloud

(concurrent condition: as you explore The Balanced Literacy Diet website; retrospective

condition: about your exploration of The Balanced Literacy Diet website; virtual revisit

condition: while you view a recording of your exploration of The Balanced Literacy Diet

website). What I mean by think aloud is that I want you to tell me everything that you

are/were thinking from the time you begin/began exploring the website until the end of

your exploration. I would like you to talk aloud constantly. I don’t want you to try to

plan out what you say or try to explain to me what you are saying. Just act as if you are

alone in the room speaking to yourself. It is most important that you keep talking.

18

While think aloud studies most often employ a specific task, an open-ended task was

used in this study to reflect as naturally as possible how teachers use and learn from professional

development websites. To reduce disruption to the participants’ cognitive processes, prompts,

redirections, and interventions were kept to a minimum during the process of verbalizing

(Jaspers, 2009). Participants who were silent for a period of 30 seconds were only told to “keep

talking.” This prompt was given to one participant in the concurrent condition.

Prior to their navigation of The Balanced Literacy Diet website, participants in the

concurrent condition were given a short warm-up exercise in thinking aloud. This exercise was

completed on another educational website, Reading Rockets (http://www.readingrockets.org/),

and lasted approximately 30 seconds to 1 minute. The warm-up exercise included instructions

that were consistent with the concurrent think aloud procedure. The practice task provided the

participants in the concurrent condition with an opportunity to become familiar with thinking

aloud while simultaneously navigating the website. Feedback was provided to ensure that the

participants continuously verbalized their thoughts while navigating the website (van Someren

et al., 1994). Participants in the retrospective and virtual revisit conditions were not given a

warm up task for two reasons: (1) to reduce reactivity as much as possible and, (2) to reduce the

time lapse between participants’ navigation and their think aloud. Feedback was provided if

participants in either of these two conditions requested more information or clarity.

Semi-structured interview. The semi-structured interview was conducted following the

completion of participants’ website task and think aloud. At the end of the interview,

participants were thanked for participating in this research and given a $20 gift card for their

contribution.

19

CHAPTER 3

Study 1: Comparing Three Think Aloud Methods for Examining Teachers’ Cognitive

Processes as They Use a Multimedia Professional Development Website

Chapter 3 presents a complete empirical study in seven major sections: (1) a literature

review, (2) the study purpose and research questions, (3) methods, (4) results and discussion, (5)

general discussion and limitations, (6) significance and implications, and (7) conclusion. First,

the literature review discusses the think aloud methodology, teacher cognition, and cognitive

processes. Next, this chapter presents the study purpose and research questions. The third

major section of this chapter is the methods section, which begins with an overview of the

research design. The methods section continues with the data analysis and highlights three

types of analyses: the think aloud analysis, time sampling analysis, and interview analysis. The

methods section concludes with a description of the data review process. The fourth major

section of this chapter is the results and discussion. A general discussion including the

limitations of study 1 follows the results and discussion section. Finally, this chapter ends with

the study significance, research implications, and a conclusion.

The following review of the literature includes two main topics: think aloud

methodology, and teacher cognition. The purpose of including a literature review in these areas

is twofold: (1) this study compares three think aloud methodologies, and (2) this study examines

teachers’ cognitive processes as they use a professional development website; therefore, the

following review sets a foundation for study 1.

Literature Review

Think Aloud Methodology

Thinking aloud has historical roots in introspection analysis, a form of data collection

aimed at investigating psychological claims and theories of mind during the eighteenth century

(Boren & Ramey, 2000; Ericsson, 2002). The cognitive revolution of the 50s and 60s produced

20

alternative types of verbal reports of thinking to gather information about cognitive structures

and processes (Ericsson, 2003). Today, the think aloud method most widely employed is based

on the techniques of protocol analysis by Ericsson and Simon (1984, 1993). As described by

Ericsson and Simon (1984), thinking aloud captures cognitive processes in real time and verbal

reports “provide the most informative data available on thinking during cognitive tasks”

(Ericsson, 2003). Cognitive processes underlying decisions and behaviours are usually “hidden

from direct observation” (Gaissmaier, Fifc, & Rieskany, 2010, p. 141). However, the think

aloud method makes monitoring cognitive processes possible—the think aloud generates direct

data about the ongoing cognitive processes that occur during task performance (Jaspers, Steen,

van den Bos, & Geenen, 2004).

Ericsson and Simon (1984) describe three levels of verbalizations that can occur during

the think aloud method. The first two levels require information processing in the participant’s

short term memory and the third level requires additional cognitive resources and retrieval of

information from long term memory (Olmsted-Hawala, Murphy, & Hawala, 2010). While

Ericsson and Simon (1984) state that Level 3 verbalizations or higher cognitive processes are

less reliable because they involve access to long-term memory, usability researchers suggest that

this type of data provides useful information about online learning, website user goals, and

online behaviours (Boren & Ramey, 2000; Guan, Lee, Cuddihy, & Ramey, 2006; Olmsted-

Hawala et al., 2010).

Usability researchers most often employ the concurrent and retrospective think aloud

methods to gain insight into web seeking behaviours and to evaluate a website’s content and

ease of use (Aranyi, Schaik, & Barker, 2012; Barzilai & Zohar, 2012; Branch, 2006; Kuusela &

Paul, 2000). As previously mentioned, during the concurrent procedure participants verbalize

their thoughts aloud while they simultaneously complete a task. Verbal reports that result from

21

the concurrent procedure generate data about the website user’s navigational experience. For

instance, Aranyi and his colleagues (2012) conducted an exploratory study of interaction

experience with a news website. The concurrent think aloud yielded five categories of

experience based on the participants’ evaluative statements: impression, content, layout,

information, architecture, and diversion (Aranyi et al., 2012). Similarly, Barzilai and Zohar

(2012) utilized the concurrent procedure to examine epistemic thinking in action. Data was

collected to shed light on the relationship between sixth grade students’ knowledge construction

and their online practices. Analysis revealed a positive relation between students’ online

strategies and their epistemic cognition (Barzilai & Zohar, 2012). Damico and Balidon (2007)

also employed the concurrent procedure to examine how elementary students engage with an

educational website. Findings from their study highlight how elementary students evaluated

claims and evidence of online educational resources (Damico & Balidon, 2007).

The retrospective procedure is also referred to as post-task testing, retrospective report,

and think after. Retrospective think alouds alone are used less often in the fields of online

learning and website usability since they require participants to think aloud after a task has been

completed. An international survey found that just 5% of think aloud studies (not limited to

website usability) employed the retrospective technique, whereas 89% used the concurrent think

aloud, and 6% used an alternative think aloud (McDonald et al., 2012).

One of the main reasons why retrospective think alouds are used less often is due to the

fact that the procedure relies on the ability to recall decisions after a task has been completed.

As participants recall their decisions, information may be incomplete and include errors,

omissions, and substitutions (Branch, 2006). For instance, a comparative study that examined

retrospective and concurrent verbal protocol analysis in the context of a decision-making task

22

found retrospective reports more prone to errors of omission whereas concurrent reports

contained more relevant information about the decision making process (Kuusela & Paul, 2000).

While retrospective procedures are limited by the fact that they may be incomplete and

include errors, omissions, and substitutions (Branch, 2006), they have the advantage of freeing

cognitive resources by thinking aloud after the task has been completed—retrospective think

alouds do not interfere with task performance (McDonald et al., 2012). Concurrent think

alouds, on the other hand, can interfere with task performance since participants verbalize their

thoughts while they simultaneously complete a given task—participants engage in two different

processes at the same time. When two processes occur simultaneously there is an increase in

cognitive load—“the level of mental energy required to process a given amount of information”

(Ping Lim, 2004, p. 17). As a result of a higher cognitive load, task completion may be

compromised during the concurrent procedure and resulting think aloud reports are often

procedural in nature (McDonald et al., 2012; van Gog, Kester, Nievelstein, Giesbers, & Paas,

2009).

Despite the limitations of the concurrent procedure, it has been widely used in usability

research, mostly as a means to evaluate a given website—participants verbalize their thoughts

about the ease of use and accessibility of information. While evaluative data contributes to the

refinement of websites, user experience is a complex process and usability research should go

beyond evaluating websites to include a range of cognitive processes and learning strategies

(Dillon, 2001). Moreover, few studies have been undertaken to compare the relative utility of

different think aloud procedures; and no research, to the investigator’s awareness, has compared

the relative utility of the concurrent, retrospective, and virtual revisit think aloud procedures

with practicing elementary teacher participants. Taken together, the current study addresses

these gaps in the literature by examining the utility of three think aloud methods for

23

understanding elementary teachers’ cognitive processes as they use a multimedia professional

development website. Given that this study examines elementary teachers’ cognitive processes,

the next section provides an overview of teacher cognition and cognitive processes.

Teacher Cognition

Teacher cognition can be described simply as how teachers process information. In

greater complexity, teacher cognition integrates theoretical and practical knowledge and is

guided by a set of organized beliefs, professional goals, and expectations (Bakkenes et al., 2010;

Borg, 2006). Teacher cognition research has examined several areas, including teacher

planning, differences between expert and novice teachers, and teacher decision-making during

instruction (Borko, Livingston, & Shavelson, 1990; Cercone, 2008; Krathwohl, 2002; Mercier,

Girard, Brodeur, & Laplante, 2010). With respect to teacher planning, research has identified

the foci of teachers as they plan for specialty topics and core subjects (Borko et al., 1990; Wing-

mui So, 1997). Research that reports on how teachers plan has also described factors that

influence the planning process: for example, information about students, the nature of the

instructional task, and the context of instruction (Borko et al., 1990; Warren, 2000).

Research on teacher cognition has also identified differences between expert and novice

teachers during lesson planning and implementation (Westerman, 1991). For instance, one

major difference is how expert and novice teachers perceive their students: expert teachers

usually consider individual learners during planning, whereas novice teachers are more likely to

consider the class as a whole during planning (Hogan, Rabinowitz, & Craven, 2003). In

addition, distinct differences in pedagogical knowledge, pedagogical content knowledge,

instructional approaches, and communication have been found to exist between expert and

novice teachers (Hogan et al., 2003; Livingston & Borko, 1989).

24

Teacher cognition research has also identified underlying reasons for instructional

decisions (Rich & Hannafin, 2008). For instance, in a study that examined preservice teacher

decision-making, participants identified several teaching strategies they employed during

instruction (e.g., asking questions, guiding, and modeling). These decisions were accompanied

by a specific reason. For example, pedagogical reasons (underlying beliefs about the nature of

learning) were reported as reasons for employing modeling as an instructional strategy (Rich &

Hannafin, 2008).

Cognitive processes underlie the study of teacher decision-making during their

professional learning experiences. Cognitive processes can be categorized as higher order

processes and lower order processes. Higher order processes, such as reasoning, involve access

to “thematically related information in long-term memory” (Horz & Schnotz, 2010, p. 238).

Information stored in memory “is interrelated and rearranged, and extended to achieve a specific

purpose” (Lewis & Smith, 1993, p. 136). Lower order processes, such as procedural

knowledge, are normally executed in an automated way and are “only marginally influenced by

intentional processes” (Horz & Schnotz, 2010, p. 238). Lower order processes provide a

foundation for higher levels of thinking. In the current study, higher order cognitive processes

refer to planning, connecting, reasoning, reflecting, and evaluating, whereas lower order

cognitive processes refer to diversion, understanding, describing, and reading. These cognitive

processes are described in detail in the methods section of this chapter.

As previously described, online learning is a favored approach to teacher professional

development. Given that elementary teachers are increasingly accessing professional

development websites, it is essential to examine teachers’ cognitive processes as they use and

learn from these online environments. Moreover, methods for understanding how teachers think

and learn within online environments need to be examined and compared so that researchers can

25

gain a comprehensive picture of teachers’ cognitive processes and learning experiences within

online learning environments.

Research Questions

This study is exploratory in nature and is not aimed at verifying hypotheses. In addition,

this study is context-specific and uses one professional development website. The following

research questions guided the study:

1. To what extent do participants’ verbalizations differ across the three think aloud

methods?

2. How do think aloud tasks differ across the three think aloud conditions?

3. To what extent does comfort level with thinking aloud differ across the three think

aloud methods?

Methods

The methods section of this chapter begins with a description of the research design and

overall process for study 1. Next, the data analysis highlights three analyses and their

procedures: the think aloud analysis, time sampling observation analysis, and interview analysis.

The methods section of study 1 concludes with a description of the data review process—a

process of inspecting the data for normality and equal variance prior to performing any

statistical tests.

Research Design

Study 1 employs a convergence mixed methods research design. The purpose of the

convergence mixed methods design is to “end up with valid and well-substantiated conclusions

about a single phenomenon” (Creswell & Piano Clark, 2007, p. 65). This study utilizes the

approach to combine statistical results with qualitative findings to better understand the relative

usefulness of three think aloud procedures for examining elementary teachers’ cognitive

26

processes as they explore a multimedia professional development website. Furthermore, the

qualitative analyses are used to corroborate and expand upon the quantitative results,

specifically in relation to research question two: How do think aloud tasks differ across the three

think aloud conditions?; and research question three: To what extent does comfort level with

thinking aloud differ across the three think aloud methods?

The research process for study 1 begins with four different data sources: demographic

questionnaire, think aloud, screen capture technology, and semi-structured interview. The main

purpose of the demographic questionnaire for study 1 was to ensure that certain demographic

features were represented within each group as equally as possible; therefore, the data set for

study 1 includes only three components: think aloud transcripts, 20-minute screen recordings,

and interview transcripts. These components were analyzed separately in an attempt to answer

the three research questions as follows:

(1) Question 1: To what extent do participants’ verbalizations differ across the three

think aloud methods?—Descriptive statistics, analysis of variance (ANOVA), and

Pearson correlations were performed on the think aloud transcripts. These

procedures are described in the data analysis section under the heading, Think Aloud

Analysis.

(2) Question 2: How do think aloud tasks differ across the three think aloud

conditions?—Descriptive statistics, ANOVA, Pearson correlations, and qualitative

observations were performed on the screen recording data. These procedures are

described in the data analysis section under the heading, Time Sampling Observation

Analysis.

(3) Question 3: To what extent does comfort level with thinking aloud differ across the

three think aloud methods?—Descriptive statistics, ANOVA, Pearson correlations,

27

and a general inductive approach to analysis were performed on portions of the

interview transcripts. These procedures are described in the data analysis section

under the heading, Interview Analysis.

Data Collection

Date sources:

demographic questionnaire

think aloud

screen capture technology

semi-structured interview

Study One Data Set

Three components:

45 think aloud transcripts

45 screen recordings

45 interview transcripts

Research Question 1: To what extent do

participants’ verbalizations

differ across the three think

aloud methods?

Think aloud analysis:

1. Descriptive statistics,

ANOVA, and Pearson

correlations were

performed on think aloud

transcripts

Research Question 2:

How do think aloud tasks

differ across the three think

aloud conditions?

Time sampling

observation analysis: 1. Descriptive statistics,

ANOVA, and Pearson

correlations were

performed on screen

recording data

2. Qualitative observations

were performed on

screen recordings

Research Question 3:

To what extent does

comfort level with

thinking aloud differ

across the three think

aloud methods?

Interview Analysis

1. Descriptive statistics,

ANOVA, and Pearson

correlations were

performed on

interview transcripts

2. A qualitative approach

to analysis was

performed on

interview transcripts

Interpretative Phase

-Results merged to produce a more

complete understanding of the single

phenomenon

Figure 3.1. Convergence mixed methods research design

28

Results of the analyses converged during the interpretive phase—a phase of analysis that

produces a more complete understanding of the main phenomenon (Creswell & Piano Clark,

2007). This chapter presents the interpretative phase in the general discussion section. Figure

3.1 outlines the overall process for study 1. The following data analysis section describes the

three different analyses and their procedures.

Data Analysis

This section presents the analysis procedures. First, the think aloud analysis describes

the procedures performed on the think aloud transcripts. Secondly, the time sampling

observation analysis describes two procedures performed on the 20-minute screen recordings.

Finally, the interview analysis describes two procedures performed on portions of the interview

transcripts.

Think aloud analysis. The think aloud analysis involves six events: (1) transcription

and word counts, (2) the development of a coding scheme, (3) unitizing the transcripts, (4) inter-

rater agreement, (5) coding the think aloud transcripts, and (6) performing statistical tests.

1. Transcription and word counts. Audio recordings were transcribed verbatim

resulting in 45 think aloud transcripts. The primary researcher transcribed all 45 recordings to

ensure greater transcription quality—errors, such as omissions and substitutions, are less likely

to occur when the transcriber is familiar with the research (Poland, 1995; Tilley, 2003).

Word counts were calculated for each of the three conditions (concurrent, retrospective,

and virtual revisit). As shown in Table 3.1, there are clear differences in the average number of

words participants generated in the three think aloud conditions. These word counts were

analyzed and results from this analysis are presented in the results and discussion section. On

average, transcripts that resulted from the concurrent think aloud contained 1676.13 words (SD

= 394.89), transcripts that resulted from the retrospective think aloud contained 658.80 words

29

(SD = 229.00), and transcripts that resulted from the virtual revisit think aloud contained

2637.87 words (SD = 359.87). Table 3.1 presents the average number of words across the three

conditions.

Table 3.1

Average Word Counts across the Three Conditions

Concurrent

Condition

Retrospective

Condition

Virtual

Revisit

Condition

M

(SD)

M

(SD)

M

(SD)

Word Count

1676.13

(394.89)

658.80

(229.00)

2637.87

(359.94)

2. Development of a coding scheme. The coding scheme used to code all 45 transcripts

was developed through a series of steps. First, an initial coding scheme was generated based on

several studies relating to website usability (Aranyi et al., 2012; Cooke, 2010; Damico &

Balidon, 2007; van Gog et al., 2005; Tan & Wei, 2006; Zhao & McDonald, 2010), teacher

planning and decision making (Kansanen et al., 2000; Moos, 2014), and Ericsson and Simon’s

levels of verbalizations and suggested statements (1984). Fifty-three potential categories

comprised the initial list (see Appendix B-1). Categories were refined, interconnected, and

eliminated resulting in a coding scheme of nine categories. The following three examples show

how categories were refined, interconnected, and eliminated:

(1) the category, Identifying and making sense of new information, was refined to

Understanding;

(2) the three categories, Procedure: describing current or future activity, Observation:

making an observation about the website or participant’s own behaviour, and Cognitive

mapping: taking in sensory information were found to be similar and therefore grouped

together as Describing; and

30

(3) the category, Verbal fillers, was eliminated as a result of re-examining the research

questions in relation to the initial list (verbal fillers, such as “um”, were deemed

uninformative).

To test this coding scheme, two transcripts were coded from each condition. As a result,

two categories, Evaluating and Describing, were separated into sub-categories. Evaluating was

separated into two groups: (1) Evaluating website content, and (2) Evaluating user experience.

This decision resulted from a clear distinction in the participants’ evaluative statements.

Participants either evaluated the website content (i.e., “These anchor charts are really useful,

they’re very practical”) or their user experience (i.e., “It was nice also that you could sort by

grade and then re-filter by the food groups”). Describing was also separated into two groups:

(1) Describing procedural behaviour, and (2) Describing website content. This decision resulted

from a clear distinction in the participants’ descriptive statements. Participants either described

what they viewed on the screen (i.e., “It’s showing me words and how to sound them out and

how many sounds are in each one”) or they described their web-based actions (i.e., “I’m going

to go into the food groups section”). The final coding scheme used to code all 45 transcripts

includes 11 categories. Table 3.2 summarizes the coding scheme and offers a description of

each category and an example from the think aloud transcripts.

Table 3.2

Coding Scheme-Categories Used to Code the Transcripts

Verbalization

Category

Description Examples from the Transcripts

Planning Referring to program planning,

reorganizing information to form or

develop new ideas; constructing and

creating

“…that is a lesson I can just take and

tweak for my class right away.”

Connecting Recalling information; activating

prior knowledge in relation to

information presented on the site;

finding a past example or recalling a

concept

“This reminds me of the mini lessons that

I like to do at the beginning of the year.”

31

Reasoning Providing a rationale for making a

navigational decision; explaining why

“I was considering looking at text

structures and I ended up choosing

reading comprehension strategies because

I’m trying to make that one of the main

focuses of our reading program.”

Reflecting Making meta-comments in reference

to awareness of their own thinking

and learning style

“I like to go over everything first and

then go back and look at something more

in-depth.”

Evaluating-

Website

Content

Making judgments or expressing

opinions about an aspect of the

website or information presented on

the website

“This kind of photo tour really informs

me in terms of good practice for

classroom management and good

classroom environments.”

Evaluating-

User

Experience

Expressing a positive or negative

feeling towards the usability and

accessibility of the website and its

features

“It’s nice that the link is already there for

me, that I don’t have to type in a separate

search button or go onto Google, I can

just click onto the link.”

Diversion

Verbalizing difficulties, including

utterances where participants indicate

uncertainty and confusion

“Hmm, assessment, how do I go, so what

do I do? Enter? Search?”

“How do I go back? This is…am I doing

something wrong?”

Understanding

Identifying and making sense of new

information and web-based tools

“At this point I’m just looking at the

main page and I’m understanding how

the website is organized.”

Describing-

Procedural

Behaviour

Describing what they are doing or

going to do or just did; statements

about participants’ actions during

their navigation

“I’m just looking at the videos right

now.”

Describing-

Website

Features

Describing the spatial characteristics,

website features and images; what

participants notice

“It’s showing various pictures and

monthly virtual tours.”

Reading Reading words, phrases or sentences

off the screen; reading out texts,

headings, links, and other on-screen

text

“Motivation for literacy, oral language,

knowledge building, concepts of print,

writing conventions.”

The categories listed in Table 3.2 are types of cognitive processes. As previously

discussed, cognitive processes can be categorized into two levels: higher order processes and

32

lower order processes. The higher order and more complex cognitive processes “involve the

manipulation of information” (McLoughlin & Mynard, 2009, p. 148), whereas the lower order

cognitive processes demand only “mechanical application of previously acquired information”

(Lewis & Smith, 1993, p.133). As previously stated, this study categorizes higher order

cognitive processes as planning, connecting, reasoning, reflecting, and evaluating. These

cognitive processes go beyond what is written in the text and relate a given concept to the

learner’s experience. In addition, this study categorizes lower order cognitive processes as

diversion, understanding, describing, and reading. The execution of these lower order cognitive

processes occurs in a more automated way.

The organization of the categories into higher and lower order cognitive processes is

consistent with Krathwohl’s revised taxonomy of educational objectives (2002). Krathwohl

(2002) organizes six major categories and 19 sub-categories of the cognitive domain

hierarchically and discusses how they differ in complexity (see Appendix B-2 for Krathwohl’s

Revised Taxonomy). Krathwohl (2002) distinguishes between higher and lower order cognitive

processes: the more complex categories (e.g., create) are higher on the scale, whereas the less

complex categories (e.g., recalling) are lower on the scale. This distinction was taken into

consideration during the development of the coding scheme for the current study.

3. Unitizing. Prior to coding, the think aloud transcripts were first segmented or

“unitized” into thought units—each utterance was deemed a separate segment or thought unit if

it conveyed relevant information and was preceded and followed by a pause and a change of

ideas (Lincoln & Guba, 1985). Unitizing involves subjective interpretation and

contextualization (Lomard, Snyder-Duch, & Campanella Bracken, 2004). Therefore, the

transcripts were unitized by the primary researcher, who had a thorough understanding of the

research topic.

33

Once all 45 transcripts were unitized, the total thought units were calculated. Similar to

the differences in word counts, there are clear differences in the average number of thought units

generated by participants in each think aloud condition. As with the word counts, the thought

units were analyzed, and results are presented in the results and discussion section. On average,

transcripts that resulted from the concurrent think aloud contained 147.40 thought units (SD =

35.83), transcripts that resulted from the retrospective think aloud contained 30.73 thought units

(SD = 13.15), and transcripts that resulted from the virtual revisit think aloud contained 123.27

thought units (SD = 20.58). Table 3.3 presents the average number of thought units across the

three conditions.

Table 3.3

Average Number of Thought Units across the Three Conditions

Concurrent

Condition

Retrospective

Condition

Virtual

Revisit

Condition

M

(SD)

M

(SD)

M

(SD)

Number of Thought

Units

147.40

(35.83)

30.73

(13.15)

123.27

(20.58)

4. Inter-rater agreement. Once the transcripts were unitized, a second coder was trained

on the coding scheme to establish inter-rater agreement. Training involved a review of the

research study questions, a review of the definitions used to code the transcripts, and a

demonstration of the coding process. Ten percent of the transcripts were coded and

disagreements were discussed and resolved until an inter-rater agreement of 97% was reached.

Cohen’s Kappa was performed to determine consistency among raters and was found to be 0.98,

CI (0.978-0.996).

5. Coding the transcripts. Due to the high level of inter-rater agreement, all 45

transcripts were coded by the primary researcher. NVivo (QSR International, 2012), a

34

qualitative data analysis software program, was used to code the transcripts. NVivo was chosen

for this study because it aids in the organization and management of large data sets. Thought

units were tallied to provide frequency counts for each category. These frequency counts were

then transformed into percentages based on the total number of thought units across categories.

Coding resulted in a total of 4,521 thought units. Among these, 102 (2.3%) thought

units were coded as planning, 354 (7.8%) as connecting, 939 (20.8%) as reflecting, 453 (10.0%)

as reasoning, 363 (8.0%) as evaluating user experience, 580 (12.8%) as evaluating website

content, 215 (4.8%) as understanding, 195 (4.3%) as diversion, 495 (11.0%) as describing

procedural behaviour, 318 (7.0%) as describing website features, and 507 (11.2%) as reading.

Table 3.4 displays the distribution of the frequency of thought units across the three conditions.

Table 3.4

Frequency of Thought Units across Conditions

Category Concurrent

Condition

Retrospective

Condition

Virtual Revisit

Condition

Total

Total Total Total Total %

Planning

9 20 73 102 2.26

Connecting

127 57 170 354 7.83

Reasoning

47 47 359 453 10.02

Reflecting

354 121 464 939 20.77

Evaluating Website

Content

229 66 285 580 12.83

Evaluating User

Experience

166 59 138 363 8.03

Diversion

136 3 56 195 4.31

Understanding

141 14 60 215 4.76

Describing

Procedural

Behaviour

265 57 173 495 10.95

35

Describing Website

Features

235 17 66 318 7.03

Reading 502 0.00 5 507 11.21

6. Performing statistical tests. First, the frequencies of each thought unit were

converted to percentages for each participant (for example, a participant in the concurrent

condition verbalized 2 planning thought units out of a total of 228 thought units; therefore, 0.9%

of the total thought units from this participant were planning thought units). These percentages

were entered into SPSS for analysis. The percentage of thought units in each condition were

then analyzed quantitatively. Analyzing the percentage of thought units, as opposed to the

frequency of thought units, produces a more accurate representation of the cognitive processes

(Rosenzweig, Krawec, & Montague, 2011). This allows for a more accurate comparison of

thought units across conditions. The word counts and total number of thought units were also

analyzed quantitatively. Descriptive statistics, ANOVA, and Pearson correlations were

performed on the proportion of thought units, total word counts, and total number of thought

units across the three conditions (concurrent, retrospective, and virtual revisit). The results and

discussion section presents these test results.

Time sampling observation analysis. Screen-capture recordings of participants’

navigations were analyzed using two different time sampling observations: (1) observations of

the participants’ web-based actions, and (2) observations of participants’ web-based actions with

the audio recordings of participants’ think alouds.

1. Observations of web-based actions. Participants’ web-based actions were analyzed in

10-second intervals. This analysis had two purposes: (1) to track participants’ online

behaviours, and (2) to examine the differences in think aloud tasks across the three conditions.

Prior to this analysis, a list of web-based actions was created based on the website used in this

36

study (e.g., types of web pages, features, and web-based tools). Table 3.5 presents the list of 18

actions used for the first time sampling observation analysis.

Table 3.5

Time Sampling Analysis-Variables

Web-Based Action Description

1. Enters a search term Text is entered in search box

2. Maneuvers around virtual

tour

Moves around the virtual tour by clicking and dragging the

mouse

3. Opens content page Selects a content page from the tabs at top of the page (e.g.,

Teacher Information)

4. Opens food groups page Selects a food groups page from the tabs (e.g., Motivation for

Literacy)

5. Opens homepage Selects the homepage from the tabs at the top of the page

6. Opens how to videos Selects the how to video page from the tabs at the top of the page

7. Opens outside link Clicks on a hyperlink and views an outside website

8. Opens recipe Selects a recipe (a lesson)

9. Opens recipe finder page Selects the recipe finder page from the tabs at the top of the page

10. Opens virtual tour Selects and views a virtual tour

11. Selects filter option Topic choice and grade is narrowed down using the filter option

12. Starts a video Selects a video by clicking on the video image

13. Stops video before the end The video is stopped before the end

14. Takes a note Text is entered in a Word document

15. Uses virtual tour arrow Within the virtual tour, the arrows are selected to move around

the classroom

16. Uses virtual tour zoom Within the virtual tour, the zoom function is selected to view a

close up of student work and teacher materials

17. Views a photograph Within a recipe, selects the photo gallery to view photos

18. Views a video Time viewing videos is tracked in seconds

Participants’ navigations were observed and counts were made in 10-second intervals. A

highlighted cursor helped track participants’ actions during the screen recording observations.

37

Actions were tallied for each participant and frequency counts were used to compare the

conditions. Appendix B-3 presents a chart showing these frequency counts for each condition.

2. Observations of web-based actions with think aloud recordings. A second time

sampling analysis was performed to determine the correspondence between web-based actions

and verbalizations. Participants’ web-based actions were observed simultaneously with the

think aloud audio recordings. In 10-second intervals agreements and discrepancies between the

actions observed on the screen recordings and the think aloud verbalizations were recorded on a

chart. Appendix B-4 presents a sample chart from each condition. The results and discussion

section includes the findings of both time sampling observation analyses.

Interview analysis. The interview analysis involves three parts: (1) transcription, (2)

statistical analysis, and (3) a general inductive approach to analysis.

1. Transcription. Interview audio recordings were transcribed verbatim resulting in 45

interview transcripts. Once again, the primary researcher transcribed all 45 interviews. The

transcripts were entered into NVivo.

2. Statistical analysis. Numerical responses to interview question two, which asked

participants to rate their comfort level of thinking aloud on a 7-point scale, were entered into

SPSS for statistical analysis. ANOVA was performed on the participants’ comfort level ratings.

ANOVA results, including descriptive statistics, are discussed in the results and discussion

section.

3. General inductive approach to analysis. A qualitative analysis followed the ANOVA

procedure. The purpose of analyzing the interview responses qualitatively was to provide

possible reasons why participants were comfortable or uncomfortable with their experience of

thinking aloud. During data collection, participants were asked to provide a rationale for their

comfort level rating. These rationales were analyzed using a general inductive approach to

38

analysis, which allows themes to emerge from the data (Thomas, 2006). The descriptive

responses were first read reflectively. Phrases were then categorized and interconnected. This

resulted in six themes related to participants’ comfort with thinking aloud. The results and

discussion section presents the findings of the interview analysis.

Data Review Process

Variables used in the quantitative analyses were examined to determine if the

assumptions of normality and equal variances were met. Appendix B-5 summarizes the data

screening results including the descriptive statistics, normality tests, and homogeneity of

variance tests. The data examination revealed that variables were both normal and skewed

(positively or negatively). Skewed variables indicate an uneven distribution of scores above and

below the mean; therefore, both parametric and non-parametric tests were performed. The non-

parametric test used in this study was the Kruskal-Wallis test which is less restrictive than

ANOVA, its parametric counterpart (Field, 2009). Any variables that included extreme outliers

were also examined and reanalyzed with extreme outliers removed. According to the

questionnaire data, the extreme outliers did not demonstrate any unique characteristics that

required further examination. A comparison of ANOVA (with and without outliers) and

Kruskal-Wallis tests indicated similar results; therefore, the parametric test results are presented

for this study.

Results and Discussion

Given the complexity of study 1, this chapter organizes the results and discussion

together according to the three research questions. This section begins with a description of the

results from the think aloud analysis and discusses key findings that resulted from this analysis.

Next, this section presents the results from the time sampling analysis and discusses key

39

findings from the time sampling analysis. This section concludes with a description of the

interview analysis results and a discussion of the key findings from the interview analysis.

A general discussion of the findings follows the results and discussion section. The

general discussion of study 1 begins with a description of participants’ typical experience of

completing the website task and think aloud from each of the three conditions. These

descriptions offer a representation of participants’ experience with the three types of think aloud

methods. The general discussion also presents the benefits and limitations of each think aloud

method based on this study’s findings.

Research Question 1: To What Extent Do Participants’ Verbalizations Differ Across the

Three Think Aloud Methods?

This section is divided into three parts (see Figure 3.2). The first two parts present the

results and discussion of the think aloud analysis: the first will involve the thought unit variables

and the second will involve the word count and total number of thought unit variables. Both

will highlight the main effects, post hoc results, and significant findings. The third part of this

section presents Pearson correlations of the variables from the think aloud analysis.

Research Question 1: To what extent do participants’ verbalizations

differ across the three think aloud methods?

1. Thought unit variables

a. Main effects

b. Post hoc test results and discussion—significant findings

2. Word counts and number of thought units

a. Main effects

b. Post hoc test results and discussion—significant findings

3. Pearson correlations of think aloud variables

Figure 3.2. Topics pertaining to research question 1 results and discussion.

40

Thought unit variables. A one-way between subjects ANOVA was performed to

compare the percentage of each category verbalized by participants in each think aloud

condition. The analyses revealed a significant main effect for planning, F(2, 42) = 7.05, p =

.002, η2

= .251; connecting, F(2, 42) = 8.35, p = .002, η2

= .251; reasoning, F(2, 42) = 22.01, p

= .000, η2

= .512; and reflecting, F(2, 42) = 5.36, p = .008, η2

= .203. ANOVA results also

indicated significant main effects for describing website features, F(2, 42) =11.05, p = .000, η2

=

.345; and for reading, F(2, 42) =43.81, p = .000, η2

= .676. There were no significant main

effects for the other five categories. Table 3.6 displays the summary statistics for the main

effects and the means and standard deviations for each thought unit variable. What follows is a

discussion of each significant main effect and possible explanations for each finding. Direct

quotes from the think aloud transcripts are included to support the discussions.

Table 3.6

ANOVA Summary Statistics for Thought Units

Concurrent

Condition

Retrospective

Condition

Virtual Revisit

Condition

M

(SD)

M

(SD)

M

(SD)

F p η2

Planning

0.407 (0.70) 3.96 (3.83) 4.17 (3.66) 7.05 .002* .251

Connecting

6.04 (3.59) 13.23 (6.53) 9.08 (3.81) 8.35 .001* .285

Reasoning

2.23 (2.46) 10.99 (7.65) 19.27 (9.16) 22.01 .000** .512

Reflecting

16.71 (7.70) 26.86 (11.45) 25.08 (7.48) 5.36 .008* .203

Evaluating

Website

Content

10.87 (4.96)

12.18 (8.71)

15.36 (5.27)

1.87

.166

.082

Evaluating

User

Experience

7.96 (6.98)

12.13 (9.03)

7.47 (3.30)

2.09

.136

.091

Diversion

5.97 (3.01) 0.79 (2.08) 2.99 (3.42) 13.65 .067 .394

Understanding

5.99 (3.50) 2.97 (4.46) 3.36 (3.90) 2.88 .067 .121

41

Describing

Procedural

Behaviour

11.64 (6.21)

11.71 (9.12)

9.37 (4.91)

0.55

.582

.025

Describing

Website

Features

10.15 (4.87) 3.45 (4.83) 3.56 (3.53) 11.05 .000** .345

Reading 22.03 (12.79) 0.00 0.29 (0.64) 43.81 .000** .676

*p < .05, **p < .01

Note: df = (2, 42) for all variables

Planning. Post hoc comparisons using Tukey HSD revealed that participants in the

retrospective and virtual revisit conditions verbalized a significantly greater proportion of

planning thought units than participants in the concurrent condition (p < .05). One possible

reason for this finding relates to cognitive load. Cognitive load refers to the amount of mental

effort in working memory: cognitive load “represents the load that performing a particular task

imposes on the cognitive system” (Pass & Van Merrienboer, 1994, p. 122). Participants in the

concurrent condition were required to use a higher level of mental energy to process the given

information as they simultaneously completed the website task (Ping Lim, 2004); thus, the

cognitive load on their working memory may have diminished the quality of their

verbalizations. This is consistent with Ericsson and Simon (1993) who state that concurrent

reporting may become difficult to maintain under high cognitive load conditions. Cognitive

load research also indicates that during multimedia tasks, concurrent reporting will interfere

with information processing and limit the extent of the thought units (Nielsen, Clemmensen, &

Yssing, 2002; van Gog et al., 2009). As the participants in the concurrent condition verbalized

their thoughts while simultaneously completing the website task, the cognitive demand on their

working memory increased: their brains prioritized information processing over verbalizations.

Therefore, participants in the concurrent condition verbalized fewer planning thought units

(Cooke, 2010).

42

In contrast, participants in the retrospective and virtual revisit conditions had the

advantage of thinking aloud after the website task had been completed; thus, the cognitive load

on their working memory was lighter during their think aloud than those in the concurrent

condition. As a result of a lighter cognitive load, participants in the retrospective and virtual

revisit conditions had more cognitive resources available during the think aloud; therefore, they

could focus their mental energy on constructing and verbalizing complex ideas.

Planning is a complex cognitive activity and involves visualizing the future, producing

and generating new information, and “putting elements together to form an original product”

(Krathwohl, 2002, p. 215). Planning is also an indicator of teachers’ intentions for practice

(Clark & Peterson, 1986; Krathwohl, 2002). The planning thought units verbalized by

participants in the retrospective and virtual revisit conditions contained complex information

that demonstrate intentions for practice. For instance, a video on social studies and writing

integration led a participant in the retrospective condition “to think of a different idea that [she]

could bring into [her] class which includes doing cross-curricular work with probability and

ancient civilizations and trading cards and games.” Similarly, a teacher in the virtual revisit

condition described how she could modify a lesson to include cross-curricular integration:

“Building interviewing and report writing skills, I know that there is a lot that I can do with this

in terms of reading and writing and oral and drama and themes like social justice and history

and so on.” These examples demonstrate complex verbalizations; teachers are planning as they

use the professional development website—they are visualizing the future and beginning to

generate new ideas.

Connecting. Post hoc tests using Tukey HSD revealed that participants in the

retrospective condition verbalized a significantly greater proportion of connecting thought units

than participants in the concurrent condition (p = .001). No significant differences in

43

connecting thought units were found between the retrospective and virtual revisit conditions and

between the concurrent and virtual revisit conditions. As with planning, connecting is a

complex cognitive activity—connecting involves activating prior knowledge and experiences,

and recalling an idea, a concept or an event in relation to new information. Possible reasons for

the above findings can be drawn from research on information processing and memory recall.

Information processing is enhanced when new incoming information is connected to prior

knowledge and previous experiences (Mastin, 2010; Weber, Corrigan, Fornash, & Neupauer,

2003). Information processing is also enhanced when new material is interesting to the learner

(Garner & Gillingham, 1991). The more deeply new information is processed (i.e., through

connections and interest), the more likely it will be recalled. In contrast to participants in the

concurrent condition, participants in the retrospective condition were able to process

information during the website task on a much deeper level because they did not have the same

cognitive demands of verbalizing their thoughts while simultaneously completing the website

task. They were able to verbalize significantly more connections with past experiences and

interests than participants in the concurrent condition. For instance, a participant in the

retrospective condition connected components of the website to her current reading program:

After visiting a virtual tour, I was interested in going through all the different parts of the

balanced literacy and I clicked on a couple of things that interested me and that I’m

working on, like comprehension skills and fluency and word building.

As previously mentioned, the second possible reason for the above finding relates to

memory recall. Participants in the retrospective condition did not have access to a visual cue or

memory aid. It is possible that in the absence of a visual cue, participants in the retrospective

condition verbalized more connecting thought units because they recalled meaningful memories

established during the website task. For instance, a participant in the retrospective condition

44

connected website content on hand writing to a student in her class: “I saw something about

hand writing which made me think about a student that I have in grade one who…it’s quite a

struggle for her to read what she’s writing.” The information that this participant recalled was

meaningful to her because it directly related to the needs of a student in her classroom.

Reasoning. Post hoc tests using Tukey HSD revealed that participants in the virtual

revisit condition verbalized a significantly greater proportion of reasoning thought units than

participants in the other two conditions (p < .05). Reasoning involves justifications for

decisions. Participants who verbalized reasoning thought units provided a rationale for their

web-based actions and behaviour. Two possible reasons for the above finding relate to

cognitive load and access to a visual aid. First, in contrast to participants in the concurrent

condition, participants in the virtual revisit condition had a lighter cognitive load on their

working memory. As previously discussed, this lighter cognitive load freed cognitive resources

and allowed participants in the virtual revisit condition to explain their web-based actions and

navigational choices. Secondly, in contrast to participants in the retrospective condition,

participants in the virtual revisit condition had direct access to their web-based actions via the

screen-capture recording. The screen recording captured participants’ website navigation and

acted as a visual aid during the think aloud. Participants in the virtual revisit condition utilized

the visual information as an aid to recall their navigational decisions and why they made them.

Taken together, available cognitive resources and direct access to web-based actions allowed

participants in the virtual revisit condition to extend their descriptions and clarify the reasons for

their navigational choices.

Reasoning about behaviour moves beyond simple descriptions of actions and offers rich

explanations about decisions. Three examples from the virtual revisit condition demonstrate

this perspective:

45

(1) A participant provided a descriptive rationale for her decision to remain on a

particular webpage for an extended period of time: “At this point I was trying to just read

what the student wrote to get an idea of whether they were creating their own stories or

whether they were doing more of a retell”;

(2) A participant clarifies her thinking about why she viewed just one virtual classroom

by explaining: “I only managed to look at the one teacher’s classroom because there was

actually a lot of information to be gained from her”;

(3) A participant in the virtual revisit condition extended the simple statement: “I ended

up choosing reading comprehension strategies,” by offering a reason for her choice: “I

was considering looking at text structures and I ended up choosing reading

comprehension strategies because I’m trying to make that one of the main focuses of our

reading program.”

Reasoning thought units provide thorough descriptions, clarifications, extensions and

overall greater insight into participants’ navigational choices—participants explain why they

make particular navigational decisions.

Reflecting. Post hoc tests using Tukey HSD revealed that participants in both the

retrospective and virtual revisit conditions verbalized a significantly greater proportion of

reflecting thought units than participants in the concurrent condition (p < .05). Along with the

explanation that relates to available cognitive resources, as previously described, a second

possible reason for this finding relates to the time lapse between the website task and the think

aloud. Participants in the retrospective and virtual revisit conditions completed the think aloud

immediately following the website task. Although relatively short, the time lapse between the

website task and think aloud may have allowed participants in the retrospective and virtual

revisit conditions to access meta-cognitive information from long term memory (Gaissmaier et

46

al., 2010). Participants in these two conditions reflected on their learning processes and how

they used the given website. Two examples from the retrospective and virtual revisit think

alouds demonstrate the types of reflective thoughts units verbalized by participants in these two

conditions:

(1) A participant in the retrospective condition reflected on her learning style, “I like

learning in that manner—having the visual to see what people are doing in their

classroom”;

(2) A participant in the virtual revisit condition reflected on her general use of online

professional resources: “When I look at websites, often I’m trying to look for specific

ideas, anchor charts or teaching strategies that I can take away right away and start trying

to use specifically on themes that I may not have covered.”

Describing website features and reading. Post hoc tests using Tukey HSD revealed that

participants in the concurrent condition verbalized a significantly greater proportion of

describing website features thought units and reading thought units than participants in the other

two conditions (p < .05). These findings are consistent with prior studies utilizing the

concurrent think aloud (McDonald et al., 2012; Zhao & McDonald, 2010). For instance, Zhao

and McDonald (2010) found that the categories containing the greatest number of utterances

during a concurrent think aloud were those relating to descriptive information and on-screen

text.

One possible reason for these findings is that the cognitive load diminished the quality of

verbalizations. As previously discussed, participants in the concurrent condition were required

to use a higher level of mental energy to process the given information as they simultaneously

completed the website task (Ping Lim, 2004). As participants in the concurrent condition

processed information from the given website, cognitive resources were less available for the

47

verbalization of complex thought processes. Therefore, participants in the concurrent condition

verbalized a greater proportion of thought units related to descriptions of website features and

reading text on the screen since these types of thought units are not as cognitively demanding to

verbalize.

Two quotes from participants in the concurrent condition demonstrate the simplicity of

the cognitive processes, describing website features and reading. First, a participant identifies a

pyramid image: “So here, reading pyramid.” The given website provides a reading pyramid as a

way for its viewers to understand and support reading development; however, this participant

merely describes the image as a reading pyramid and does not provide any further information.

Secondly, as a participant in the concurrent condition begins the website navigation, she reads

the main heading on the homepage: “Balanced literacy is a framework that presents literacy

concepts.” This participant moves on to read the next heading without providing further

thoughts about the main website heading.

With respect to the remaining thought units, findings indicated no significant differences

between the three conditions for evaluating, diversion, understanding, and describing procedural

behaviours. This suggests that regardless of the type of think aloud employed, participants will

verbalize a relatively equal number of thoughts related to evaluating the website, to their

confusion and understanding of the web-based tools and information, and to descriptions of their

own actions and online behaviours.

Word counts and total number of thought units. A one-way between subjects

ANOVA was performed to compare the frequency of words and thought units verbalized in

each think aloud condition. The summary statistics for the main effects and the means and

standard deviations are summarized in Table 3.7. The following section discusses the

significant main effects and possible explanations for each finding.

48

Table 3.7

ANOVA Summary Statistics for Word Count and Number of Thought Units

Concurrent

Condition

Retrospective

Condition

Virtual

Revisit

Condition

M

(SD)

M

(SD)

M

(SD)

F p η2

Word Count

1676.13

(394.89)

658.80

(229.00)

2637.87

(359.94)

130.421 .000** .861

Number of Thought

Units

147.40

(35.83)

30.73

(13.15)

123.27

(20.58)

90.756 .000** .812

**p < .01

Note: df = (2, 42) for all variables

The analysis revealed a significant main effect for both variables across the three

conditions: F(2, 42) = 130.42, p = .000, η2

= .861 for word count; and F(2, 42) = 90.76, p =

.000, η2

= .812 for number of thought units. As Table 3.7 shows, participants in the

retrospective condition verbalized the fewest number of words (M = 658.80, SD = 229.00) and

thought units (M = 30.73, SD = 13.15). Participants in the concurrent condition verbalized more

than twice the number of words as participants in the retrospective condition (M = 1676.13, SD

= 394.89) and more than four times the number of thought units than participants in the

retrospective condition (M = 147.40, SD = 35.83). Participants in the virtual revisit condition

verbalized the most number of words (M = 2637.87, SD = 359.94). However, participants in the

virtual revisit condition verbalized less thought units than teachers in the concurrent condition

(M = 123.27, SD = 20.58). This finding suggests that thought units verbalized by teachers in the

virtual revisit condition contained more words than thought units verbalized by teachers in the

concurrent condition.

Post hoc comparisons using Tukey HSD revealed significant differences in the

frequency of words (p < .01) and the frequency of thought units (p < .05) between all three

conditions. In contrast to the participants in the concurrent and virtual revisit conditions who

49

were asked to verbalize their thoughts for 20 minutes, participants in the retrospective condition

were not given a time constraint to verbalize their thoughts. Although an unlimited amount of

time was given to participants in the retrospective condition, these participants verbalized their

thoughts for an average four minutes 30 seconds. Therefore, it is not surprising that participants

in the retrospective condition verbalized significantly fewer words and thought units than the

other two conditions. A more interesting finding is the difference in word counts and number of

thought units between the concurrent and virtual revisit conditions. Participants in the virtual

revisit condition verbalized significantly more words than participants in the concurrent

condition; however, the reverse is true for the number of thought units. This finding indicates

that thought units produced by participants in the virtual revisit condition contained a greater

number of words. This suggests that the thought units verbalized by participants in the virtual

revisit condition were more complex than the thought units verbalized by participants in the

concurrent condition. For example, as a participant from the concurrent condition views the

homepage for the first time she describes her procedural behaviour: “I’m going to have a look at

the How To Videos.” A participant from the virtual revisit condition who also views the

homepage for the first time goes further to provide a reason for her behaviour: “I’m always

interested in ways to increase my students’ background knowledge of vocabulary and

comprehension so I go back and forth between comprehension and vocabulary before I

narrowed it down and selected vocabulary.” Both of these thought units were verbalized during

the participants’ initial view of the home page. However, the thought unit verbalized by the

participant in the virtual revisit condition is more complex in that it provides a reason for her

navigational choice.

Pearson correlations. Pearson correlations were computed to determine three

relationships: between the cognitive processes, between the frequency of word counts and

50

thought units, and between the cognitive processes and word counts and thought units (see

Table 3.8). According to the results, higher order cognitive processes (planning, connecting,

reasoning, and reflecting) were positively correlated to other higher order cognitive processes,

and negatively correlated to lower order cognitive processes (diversion, understanding,

describing website features, and reading). Similarly, lower order cognitive processes were more

likely to be positively correlated to other lower order cognitive processes and negatively related

to higher order cognitive processes. For instance, reading was negatively correlated to planning

(r = -.42, p < .01), connecting (r = -.39, p < .01), reasoning (r = -.50, p < .01), reflecting (r = -

.55, p < .01), evaluating website content (r = -.31, p < .05), and evaluating user experience (r = -

.37, p <.05). In contrast, reading was positively related to diversion (r = .46, p < .01),

understanding (r = .30, p < .05), and describing website features (r = .45, p < .01). These

findings corroborate the above ANOVA results and suggest that participants in the concurrent

condition, who were more likely to read text during the think aloud, were less likely to verbalize

thought units related to higher order cognitive processes.

Pearson correlations also revealed a positive correlation between the frequency of words

and the frequency of thought units (r = .70, p < .01). This finding indicates that as the number

of words increased the number of thought units increased. Results also revealed a negative

correlation between higher order cognitive processes and the number of thought units

verbalized. This suggests that teachers who verbalized thought units related to higher order

processes were more likely to verbalize fewer thought units than teachers who verbalized

thought units related to lower order processes.

51

Table 3.8 Pearson Correlations 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Cognitive

Processes

1. Planning 1

2. Connecting .31* 1

3. Reasoning .36* .06 1

4. Reflecting .19 .38* -.10 1

5. Evaluating

Website

-.23 .03 .03 .23 1

6. Evaluating

Experience

.05 -.08 -.09 .16 -.34* 1

7. Diversion -.36* -.39** -.22 -.46** -.17 -.14 1

8.Understanding -.26 -.35* -.21 -.29 -.34* .05 .38* 1

9. Describing

Behaviour

-.12 -.20 .03 -.28 -.56** -.47** -.003 .01 1

10. Describing

Website

-.43** -.33* -.45** -.31* -.13 -.23 .39** .23 .10 1

11. Reading -.42** -.39** -.50** -.55** -.31* -.37* .46** .30* .18 .45** 1

12.Word

Count

.07 -.26 .28 -.01 .18 -.19 .25 .05 -.16 .02 -.05 1

13. Thought

Units

-.31* -.53** -.15 -.38** -.04 -.30* .57** .30* .01 .44** .54** .70** 1

14. Time on

Page

.30* .47* .12 .38* -.06 .04 -.36* -.11 .02 -.22 -.41** -.07 -.43** 1

15. Number of

Pages

-.21 -.34* -.27 -.37* -.29 -.08 .46** .46** -.04 .44** .50** .11 .55** -.68** 1

16. Comfort

Level

.11 .23 .28 .13 .21 -.15 -.25 .07 .11 -.37* -.34* .14 -.13 .05 -.17 1

*p < .05, **p < .01

52

Summary of Results and Discussion for Research Question 1

This section identified and discussed significant findings in relation to research question

1: To what extent do participants’ verbalizations differ across the three think aloud methods?

First, this section identified differences in the types of verbalizations produced by participants in

each condition. Participants in the retrospective and virtual revisit conditions verbalized

significantly more thought units related to higher order cognitive processes: planning,

connecting, reasoning, and reflecting. In contrast, participants in the concurrent condition

verbalized significantly more thought units related to lower order cognitive processes:

describing website features and reading. Possible reasons for these findings relate to cognitive

load on working memory, information processing, memory recall, visual aids, and the time lapse

between the website task and the think aloud.

This section also identified differences in the frequency of words and thought units

across the three conditions. Specifically, participants in the virtual revisit condition verbalized

the highest number of words and produced complex thought units. Results of the correlation

analysis indicated correlations between the cognitive processes and frequency of word counts

and thought units. The next main section presents the results and discussion of the time

sampling observation analysis.

Research Question 2: How Do Think Aloud Tasks Differ Across the Three Think Aloud

Conditions?

This section continues the results and discussion of study 1 by presenting the results of

the time sampling observation analysis in two parts (see Figure 3.3). The first part presents the

results and discussion of the time sampling observation analysis involving the web-based action

variables, and highlights the main effects and post hoc results, followed by a discussion of the

overall findings. The second part presents the results and discussion of the time sampling

53

observation analysis involving the web-based actions with the corresponding think aloud audio

recordings.

Research Question 2: How do think aloud tasks differ across the

three think aloud conditions?

1. Web-based action variables

a. Main effects

b. Post hoc results and discussion—significant findings

2. Web-based actions with the corresponding think aloud recordings

Figure 3.3. Topics pertaining to research question 2 results and discussion.

Web-based actions. A one-way between subjects ANOVA was performed on the web-

based action variables. The analysis revealed seven significant main effects: Time spent on

pages, F(2, 42) = 7.16, p = .002, η2

= .254; total number of pages visited, F(2, 42) = 11.33, p =

.000, η2

= .351; total number of food group pages visited, F(2, 42) = 8.50, p = .001, η2 = .288;

total number of content pages visited, F(2, 42) = 3.66, p = .034, η2 = .149; number of visits to

the homepage, F(2, 42) = 9.34, p = .000, η2 = .308; time participants spent viewing videos (in

seconds), F(2, 42) = 5.13, p = .010, η2

= .196; and number of videos that participants stopped

midway, F(2, 42) = 14.64, p = .000, η2

= .417. The analysis revealed no significant main effects

for the remaining variables. Table 3.9 summarizes the means, standard deviations, and main

effects for the web-based action variables. The remaining section presents the post hoc results

for the significant findings. This is followed by a discussion of the overall results of the time

sampling observation analysis involving the web-based actions.

54

Table 3.9

ANOVA Summary Statistics for Web-Based Action Variables

Concurrent

Condition

Retrospective

Condition

Virtual Revisit

Condition

M

(SD)

M

(SD)

M

(SD)

F P η2

Time on Each

Page (in

seconds)

36.07

(9.08)

52.53

(15.11)

47.80

(11.87)

7.16 .002* .254

Number of

Pages Visited

26.93

(9.02)

16.13

(4.19)

18.60

(5.32)

11.33 .000** .351

Number of Food

Group Pages

Visited

6.40

(4.12)

2.80

(1.21)

2.93

(1.92)

8.50 .001* .288

Number of

Content Pages

Visited

1.60

(1.88)

0.87

(.834)

0.40

(.507)

3.66 .034* .149

Visits to

homepage

2.13

(1.60)

0.60

(.507)

0.67

(.90)

9.34 .000** .308

Time Viewing

Videos (in

seconds)

150.67

(125.10)

334.00

(235.00)

355.33

(200.32)

5.13 .010* .196

Number of

Videos Stopped

Midway

4.27

(2.05)

1.20

(1.26)

2.53

(2.51)

7.26 .002* .257

Enters a Search

Term

1.13

(1.96)

0.40

(.83)

0.40

(.74)

1.59 .216 .070

Maneuvers

Around Virtual

Tour

0.93

(1.71)

0.80

(2.34)

0.67

(1.50)

.08 .928 .004

Visits to How-

To Videos

1.00

(.845)

0.47

(.516)

0.53

(.743)

2.48 .096 .106

Visits to Outside

Link

2.13

(2.95)

0.80

(1.15)

1.13

(1.60)

1.73 .191 .076

Views Recipe

Page

6.80

(3.17)

6.13

(3.98)

6.27

(2.87)

0.16 .849 .008

Views Recipe

Finder Page

4.27

(2.89)

3.20

(3.84)

5.20

(2.83)

1.45 .247 .065

55

Visits Virtual

Tour

1.67

(1.45)

1.20

(.941)

1.33

(.617)

0.77 .468 .036

Selects Filter

Option

1.20

(1.66)

0.87

(2.13)

1.13

(1.64)

0.14 .870 .007

Takes a Note 0.00 0.60

(2.06)

1.53

(3.18)

1.87 .167 .082

Uses Virtual

Tour Arrow

2.00

(3.12)

1.53

(2.42)

2.40

(4.07)

0.26 .769 .012

Uses Virtual

Tour Zoom

0.07

(0.26)

0.53

(0.92)

0.20

(0.56)

2.13 .131 .092

Views a

Photograph

5.73

(5.44)

8.07

(6.20)

6.20

(8.26)

0.33 .724 .015

*p < .05, **p < .01

Note: df = (2, 42) for all variables

Time spent on pages and total number of pages visited. Post hoc comparisons using

Tukey HSD revealed that participants in the retrospective and virtual revisit conditions spent

significantly more time on web pages than participants in the concurrent condition (p < .05). In

addition, participants in the retrospective and virtual revisit conditions visited significantly

fewer pages than participants in the concurrent condition (p < .05). Pearson correlation results

support the ANOVA results and show a negative correlation between the time spent on pages

and the number of pages visited (r = -.68, p < .01). This finding confirms that participants who

spent more time on web pages visited fewer web pages during their 20-minute navigation.

Pearson correlation results also show a positive correlation between the time spent on web pages

and planning, connecting, and reflecting thought units.

Total number of food group pages, content pages, and homepage visits. Post hoc

comparisons using Tukey HSD revealed participants in the concurrent condition viewed

significantly more food groups pages than the retrospective condition (p = .002), and virtual

revisit condition (p = .003), while no significant differences were found between the

retrospective and virtual revisit conditions (p = .990). Additionally, participants in the

56

concurrent condition viewed significantly more content pages than the virtual revisit condition

(p = .027), while there were no significant differences between the retrospective and concurrent

conditions (p = .240) and between the retrospective and virtual revisit conditions (p = .554).

Finally, participants in the concurrent think aloud condition viewed the homepage significantly

more times than participants in the retrospective condition (p = .001) and the virtual revisit

condition (p = .002), while no significant difference was found between the retrospective and

virtual revisit conditions (p = .985).

Video viewing. Post hoc comparisons using Tukey HSD showed that participants in the

retrospective condition spent significantly more time viewing videos than participants in the

concurrent condition (p = .033) and also stopped fewer videos midway than participants in the

concurrent condition (p = .000). Similarly, participants in the virtual revisit condition spent

significantly more time viewing videos than teachers in the concurrent condition (p = .015) and

stopped fewer videos midway than teachers in the concurrent condition (p = .001). There were

no significant differences in time spent viewing videos and the number of videos stopped

midway between the retrospective and virtual revisit conditions. These findings indicate that

participants in the retrospective and virtual revisit conditions spent more time viewing videos

than participants in the concurrent condition.

Taken together, the overall results of the first time sampling observation analysis

involving the web-based actions suggest that participants in the retrospective and virtual revisit

conditions had higher levels of engagement with the information, images, tools, and videos on

each page of the website during their 20-minute navigation than participants in the concurrent

condition. One possible reason relates to the dual activity of completing the website task while

simultaneously thinking aloud. As previously described, the heavy cognitive load on working

memory disrupted both concurrent participants’ verbalizations and their task performance.

57

Efforts made to verbalize thoughts may have changed the way the participants in the concurrent

condition used the website (Guan et al., 2006). In contrast, participants in the retrospective and

virtual revisit conditions were able to focus exclusively on their navigation without the

additional activity of simultaneously verbalizing their thoughts.

A second possible reason for the differences in web-based actions relates to the stage in

which the instructions were given to participants. Prior to completing the website task,

participants in the retrospective and virtual revisit conditions were only given the first part of the

instructions: to use the website as they normally would when seeking information related to

their practice. In contrast, participants in the concurrent condition were given the web task

instruction as well as the think aloud instruction. Despite the practice think aloud that

participants in the concurrent condition completed prior to the study think aloud, it appears that

the preconceived notion of the think aloud task may have interfered with the concurrent

participants’ web navigation. As suggested by Ericsson and Moxley (2011), instructing the

participants to think aloud while simultaneously using the website, may have a negative effect

on participants’ task performance. Reactivity, “influences of the verbalizations on the decision

process” (Ranyard & Svenson, 2010, p. 119), appeared to negatively affect the concurrent

condition—participants may have altered their navigation due to their awareness of the task.

Web-based actions with the corresponding think aloud recordings. A second time

sampling analysis using the screen-capture recordings and think aloud audio recordings was

conducted to determine the correspondence between participants’ actions and verbalizations. In

general, participants in the retrospective condition verbalized their thoughts for an average of 4

minutes 30 seconds with a range of 1:59-6:52. In contrast, all but two participants in the other

two conditions verbalized their thoughts for 20 minutes. The two participants who did not

verbalize their thoughts for 20 minutes were from the concurrent condition and requested to stop

58

the think aloud at 18 minutes and at 18 minutes 40 seconds respectively. The final

verbalizations from these two participants were: “What else can I say? I’m not exactly sure.

Let’s stop there,” and “I’m done talking, I think.”

Results of this second time sampling analysis revealed the correspondences between

web-based actions and verbalizations. Agreements, discrepancies, and silent intervals are

reported for each condition.

Concurrent condition. During the 10-second intervals, agreements between the web-

based actions and verbalizations were recorded 93.5% of the time for participants in the

concurrent condition. That is, the action that was observed on the screen recording directly

corresponded with the verbalization. Discrepancies accounted for 6.5% of the time and

included intervals of silence and participants’ request to end the website task and think aloud

early. Intervals of silence occurred for 14 participants in the concurrent condition and

accounted for 5.4% of the total time. These intervals of silence occurred during a video. In

addition, two participants ended their think aloud early (accounting for 1.1%).

Retrospective condition. Analysis of the retrospective condition showed agreements

between web-based actions and verbalizations 17.3% of the total time and discrepancies

between participants’ actions and verbalizations 82.69% of the total time. Discrepancies were

found to occur between the first action and the first verbalization. This type of discrepancy

occurred for seven participants assigned to the retrospective condition (47%). For example, the

first verbalization from one participant was, “I think I went to year one first.” This verbalization

was discrepant with this participant’s first web-based action: the screen recording showed this

participant opening a content page and then navigating to a food group page. Another

participant in the retrospective condition stated that she immediately navigated to a web page on

vocabulary development; however, this participant initially spends time on the homepage, then

59

visits the introductory page and views two videos. She finally navigates to a web page on

vocabulary at 3 minutes 30 seconds into her navigation. The think aloud omitted the first three

main actions of her web navigation. As a further example, a participant began her think aloud

by stating, “I went to the recipes right away;” however, observations of the screen recording

show this participant initially entering a virtual classroom tour for grades one and two.

With respect to the eight participants in the retrospective condition whose first action did

correspond to their first verbalization (53%), discrepancies appeared, on average, at 49 seconds

into the think aloud. Omissions were most often observed. For example, a screen recording

shows one participant in the retrospective condition taking notes in a Word document during her

20-minute navigation. Her first note occurs at 6 minutes 20 seconds followed by seven

additional notes during the 20-minute navigation; however this participant omits this encoding

strategy entirely during her think aloud. Another participant in the retrospective condition visits

content pages and virtual tours during her navigation; however omits these actions during her

think aloud. Similarly, a participant in the retrospective condition is observed viewing

information related to assessment and writing conventions for a total of 10 minutes; however

this participant omits this information during her think aloud. The above finding is consistent

with Kuusela and Paul’s (2000) study which found retrospective reports to be more prone to

errors of omission. One possible reason that participants omitted information from their think

aloud relates to short term memory recall. Participants in the retrospective condition recalled

information without a memory aid. In the absence of a memory aid, think alouds were

incomplete (Branch, 2006).

Web-based actions that were correctly recalled occurred at the beginning of the 20-

minute website task and towards the end of the 20-minute website task. This finding is

consistent with the serial position effect—items most likely recalled from a series tend to be

60

those items from the beginning or end of a given list (Ebbinghaus, 1964). Participants in the

retrospective condition were more likely to recall their first action and their most recent action

from the website navigation. Many of their intermediate web-based actions were omitted from

their think aloud.

Finally, the second time sampling analysis revealed participants’ difficulties in recalling

information. Nine retrospective participants (60%) expressed these difficulties during the think

aloud. For example, at two minutes 56 seconds into the think aloud one participant in the

retrospective condition asked, “What else did I do?” At three minutes 43 seconds, another

participant stated, “I'm trying to think what else.” Similarly, towards the end of their think

alouds, participants in the retrospective condition asked themselves, “What else did I look at?”

“What else did I do” and “What else was I thinking?” On average, these nine participants

verbalized their difficulties in recalling their behaviours at two minutes 58 seconds into their

think aloud.

Virtual revisit condition. Agreements between web-based actions and verbalizations

were recorded for the virtual revisit condition 99.2% of the time. The analysis revealed

intervals of silence as a video played for two participants in the virtual revisit condition,

accounting for 0.8% of the total time.

Summary of Results and Discussion for Research Question 2

This section identified and discussed significant findings pertaining to research question

2: How do think aloud tasks differ across the three think aloud conditions? Taken together, the

results of both time sampling observation analyses reveal differences in the think aloud tasks

across the three conditions. Specifically, participants across the conditions differed with respect

to the time spent on each web page, the number of visits made to web pages, the types of pages

they visited most often, the time spent viewing videos, and the number of videos that were

61

stopped before the end. In addition, differences were found in the correspondences between the

web-based actions and think aloud audio recordings across the three conditions. Specifically,

discrepancies between web-based actions and verbalizations occurred most often in the

retrospective condition. The next main section presents the results and discussion of the

interview analysis.

Research Question 3: To What Extent Does Comfort Level with Thinking Aloud Differ

Across the Three Think Aloud Methods?

This section continues the results and discussion of study 1 by presenting the results of

the interview analysis in two parts (see Figure 3.4). The first part presents the results and

discussion of the interview analysis involving the reported comfort levels with thinking aloud,

and highlights the main effects, post hoc results, and significant findings. The second part of

this section shares the qualitative findings of the interview analysis and discusses possible

reasons for participants’ comfort level ratings.

Research Question 3: To what extent does comfort level with

thinking aloud differ across the three think aloud methods?

1. Comfort level with thinking aloud

a. Main effect

b. Post hoc results and discussion—significant findings

2. Qualitative findings and discussion

Figure 3.4. Topics pertaining to research question 3 results and discussion.

Comfort level with thinking aloud. A one-way between subjects ANOVA was

performed to compare the comfort levels of thinking aloud reported by the participants during

the post-task interview. Participants were asked to rate their comfort level with thinking aloud

on a 7-point scale (1 indicating very uncomfortable to 7 indicating very comfortable). On

average, participants in the concurrent condition reported their comfort level as 5.07 (SD =

62

1.22), participants in the retrospective condition reported their comfort level as 6.07 (SD =

0.96), and participants in the virtual revisit condition reported their comfort level as 6.27 (SD =

0.88). The main effect for comfort level reported by participants was significant, F(2, 42) =

5.81, p = .006, η2

= .217. Post hoc comparisons using Tukey HSD indicated that the participants

in the retrospective condition felt significantly more comfortable thinking aloud than

participants in the concurrent condition (p = .030). Participants in the virtual revisit think aloud

condition also felt significantly more comfortable thinking aloud than participants in the

concurrent think aloud condition (p = .008). No significant differences were found between the

retrospective and virtual revisit conditions (p = .857). These results indicate that participants in

the retrospective and virtual revisit conditions felt significantly more comfortable thinking aloud

than participants in the concurrent condition.

Pearson correlation results were computed between reported comfort levels and the

cognitive processes. Results revealed negative correlations between comfort level and

describing website features (r = -.37, p < .05) and comfort level and reading (r = -.34, p < .05).

These results corroborate the ANOVA findings, and suggest that participants in the concurrent

condition, who described website features and read most often during their think aloud, felt the

least comfortable with the process of thinking aloud.

Given the previous discussion related to cognitive load, it is not surprising that

participants in the concurrent condition rated their comfort level significantly lower than

participants in the other two conditions. Participants in the concurrent condition had the added

demand of completing the website task during their think aloud. Factors causing reactivity may

have also influenced participants’ comfort level. For instance, auditory feedback and task

instructions could have interfered with concurrent participants’ comfort level with thinking

aloud. This is consistent with Van den Haak and her colleagues’ findings (2003) that the

63

concurrent think aloud may result in greater reactivity and therefore greater discomfort with the

process of thinking aloud. The next section provides additional reasons for differences in

comfort level with thinking aloud across the three conditions.

Qualitative findings and discussion. As previously mentioned, interview responses

were further examined qualitatively. The purpose of performing this qualitative analysis was to

provide possible reasons for participants’ reported comfort levels. A general inductive approach

to analysis resulted in six themes related to participants’ reported comfort level: (1) Self-

awareness, (2) Negative feelings, (3) Memory retrieval, (4) Thinking aloud over time, (5)

Classroom experience, and (6) Benefits of thinking aloud. Table 3.10 summarizes each theme.

The remaining section discusses each theme and includes selected quotes to support each theme.

Table 3.10

Themes Related to Reported Comfort Level

Theme Description

1. Self-Aware An awareness of their experience and the study task

2. Negative Feelings Negative feelings of distractedness, awkwardness, and stress

3. Memory Retrieval Difficulties recalling actions and thoughts

4. Thinking Aloud Over Time Referring to an increase in comfort level over time during the

think aloud

5. Classroom Experience Use of think alouds in the classroom

6. Benefits of Thinking Aloud The value of thinking aloud as a teaching and learning tool

Theme 1: Self-awareness. Five participants in the concurrent condition (33%) described

how they were conscious of the think aloud experience during the website task. For instance,

one participant made an explicit reference to the study task and awareness of time: “I was

uncomfortable because I was conscious of time and whether I would have things to say after a

64

while.” Another participant in the concurrent condition reported a level 3 “because I felt like I

knew that someone was listening so that kind of makes it uncomfortable for me personally.”

Theme 2: Negative feelings. Ten total participants (22%) who rated their comfort level

as five or less described how thinking aloud contributed to negative feelings of distractedness,

awkwardness, and stress (concurrent, n = 4; retrospective, n = 5; virtual revisit, n = 2). One

participant in the concurrent condition stated that thinking aloud is “not something I really do…I

found myself distracted, reading to myself half the time.” Another participant in the concurrent

condition felt that “it was just kind of strange…I’m never really aware of what I’m thinking so

to have to verbalize it you have this metacognitive piece that’s not usually there.” Participants

in the retrospective condition also described their negative feelings towards thinking aloud. For

instance, one participant described how being conscious of her own voice contributed to

negative feelings: “I don’t really like talking out loud and hearing my own voice speak of

things.” Another participant in the retrospective condition described how thinking aloud was a

stressful experience and that a visual aid would have eased her stress: “It can be stressful, it’s

hard to put someone on the spot like that…I think thinking aloud for me is easier if I have some

type of visual to go along with it.” In addition, a participant in the virtual revisit condition felt

that thinking aloud was “a little unnatural, so it feels weird.”

Theme 3: Memory retrieval. Four participants in the retrospective condition (27%)

referred to the difficulty of recalling their web-based actions. For instance, one participant

reported that “sometimes it’s hard to remember what you did…it’s easier to think aloud in the

moment as opposed to thinking aloud about something that just happened.” Another participant

stated: “I was worried that I was not saying it in the right order and that I didn’t remember what

I did.”

65

Theme 4: Thinking aloud over time. Six of the total participants (13%) stated that as

the think aloud continued they became more comfortable with the process (concurrent, n = 2;

retrospective, n = 1; virtual revisit, n = 3). For instance, a participant in the concurrent condition

stated: “It was definitely an awkward experience but it was getting better as I was going through

it.” Another concurrent participant said, “Once I got going [thinking aloud] was fine.” A

participant in the retrospective condition thought thinking aloud was “a little bit weird to be

doing, but then once you get started you’re fine.” Similarly, a participant in the virtual revisit

condition shared: “At first I was a little bit worried—would I remember what I was thinking

about 20 minutes ago?” This same participant continued by stating:

But when I saw where my mouse was headed and where I was scrolling up and down I

remembered pretty clearly and easily what I was thinking about at those particular times

so it wasn’t quite as hard to remember as I thought.

Finally, a teacher in the virtual revisit condition who rated her comfort level as a 7 stated: “I

think people get better at it over time and I think I’ve improved in the ability to think aloud by

doing it.”

Theme 5: Classroom experience. Sixteen of the total participants (36%) connected their

comfort level to their teaching experience (concurrent, n = 3; retrospective, n = 7; virtual revisit,

n = 6). These participants referred to the teaching strategy, modeling thinking aloud. Their

prior experience with this strategy in the classroom contributed to a higher level of comfort

during the think aloud. For instance, a participant in the retrospective condition who rated her

comfort level as a 7 stated: “We’re taught to think aloud in front of the students all the time, so

when we think aloud they think about how their thinking processes could be, so we’re supposed

to model all that.” Another participant in the retrospective condition said: “I spend my day in

front of children being really silly, so [thinking aloud] was fine.” A participant in the virtual

66

revisit condition referred to her experience of thinking aloud during her reading program: “I

definitely do it with my children all the time, when reading stories. So I think it’s really valuable

as a comprehension strategy and gets you thinking why you do what you do.” Similarly, a

participant in the virtual revisit condition said:

I’m used to thinking aloud in front of my students. I do think alouds a lot and I

remembered so much of what I was thinking that it wasn’t hard for me…I was kind of

surprised…just seeing the image again watching the cursor. It was really the cursor that

was the connector to my thoughts.

Finally, a participant in the virtual revisit condition suggested that, “as a teacher you get very

accustomed to that concept of thinking aloud.”

Theme 6: Benefits of thinking aloud. Seventeen of the total participants (38%)

described benefits of thinking aloud (concurrent, n = 6; retrospective, n = 4; virtual revisit, n =

7). For example, one participant in the retrospective condition suggested that thinking aloud

could be “a valuable tool for learning…it could be for children or even adults to consolidate

what you’ve seen.” Furthermore, a participant in the virtual revisit condition “thought it was

really interesting” and began to think about “how I could use it with the kids and where it would

be valuable.”

Summary of Results and Discussion for Research Question 3

This section identified and discussed the significant findings pertaining to research

question 3: To what extent does comfort level with thinking aloud differ across the three think

aloud methods? Findings revealed differences in reported comfort level across the three

conditions. Participants in the retrospective and virtual revisit conditions reported feeling more

comfortable with the process of thinking aloud than participants in the concurrent condition.

67

This section also identified possible reasons for participants’ comfort level ratings. Cognitive

load and reactivity were discussed as possible reasons why participants in the concurrent

condition felt less comfortable with thinking aloud than those in the other two conditions. In

addition, the qualitative analysis revealed seven themes related to reasons why participants felt

comfortable or uncomfortable with thinking aloud: self-awareness, negative feelings, memory

retrieval, thinking aloud over time, and classroom experience. The qualitative findings also

showed benefits of thinking aloud as a teaching and learning strategy.

General Discussion of Study 1

The previous section presented the results and discussion of the three analyses: think

aloud analysis, time sampling analysis, and interview analysis. Based on the results and

discussion, this section presents the general discussion of study 1 in two parts (see Figure 3.5).

The general discussion begins with descriptions of participants’ typical experiences with each

think aloud condition. The first description includes the website task and the think aloud for the

concurrent condition. The second description includes the website task for both the

retrospective and virtual revisit conditions. The third description includes the think aloud for

the retrospective condition. The final description includes the think aloud for the virtual revisit

condition. These descriptions are representations of participants’ experiences and include

characteristic components of each think aloud condition based on the findings of study 1. The

purpose of including the characteristic components of each think aloud is to present a theoretical

portrait of how participants experienced each think aloud condition.

The second part of the general discussion highlights the benefits and limitations of

employing each think aloud method based on the findings of study 1. Chapter 3 ends with study

1 limitations, and significance and implications. Figure 3.5 outlines the topics pertaining to the

general discussion.

68

Study 1 General Discussion for Study 1

1. Descriptions of participants’ typical experiences

a. A participant’s experience with the concurrent think aloud

method

b. A participant’s experience with the retrospective and virtual

revisit website task

i. A participant’s experience with the retrospective

think aloud

ii. A participant’s experience with the virtual revisit

think aloud

2. Benefits and limitations

a. Concurrent think aloud method

b. Retrospective think aloud method

c. Virtual revisit think aloud method

3. Study 1 Limitations

4. Significance and Educational Implications

Figure 3.5. Topics pertaining to the general discussion of study 1.

A Participant’s Experience with the Concurrent Think Aloud Method

“I’m always [thinking] in my head, but I find when I’m talking I probably don’t do as

good of a job as when I’m just thinking…I’m maybe not absorbing as much information

either” (interview transcript, concurrent condition)

A participant in the concurrent condition receives the full set of instructions (the website

task and think aloud) and completes the practice task on a second educational website. The

participant immediately develops a sense of self-awareness, and feels awkward and slightly

69

stressed about the proceeding task of navigating the website while simultaneously thinking

aloud. The homepage of The Balanced Literacy Diet website appears on the laptop directly in

front of the participant. A digital recorder is activated, ready to capture the participants’

verbalizations as she navigates the website.

Initial verbalizations describe the setup of the homepage and include text, headings and

subheadings. The participant scans the text and reads words and phrases. She does this as she

quickly scrolls down the page. Then the participant notices the tabs at the top of the homepage.

She scrolls over the tabs before she selects a possible topic of interest: reading comprehension.

The reading comprehension page opens in a new window and the participant shares her

thoughts—she states that she seeks information related to her current grade.

The participant continues her navigation—she moves the computer mouse over the text

of the food group pages and sometimes highlights the text as she reads out headings,

subheadings, or the first sentence of a paragraph. At about the ten-minute mark, the participant

notices the embedded video on the right side of the screen and clicks on the video image. The

video is just over one minute in length; however, the participant stops it within the first 20

seconds. She remarks, slightly laughing to herself, that she is impatient and unable to watch the

entire video. Then the participant refers to her general website seeking behaviour—she usually

skims through websites until a concept catches her eye.

The navigational pattern continues over the course of the 20 minute task: the participant

opens pages of interest, skims the text, hovers over text and images, reads sections of text,

describes what she sees, and views snippets of video clips. Occasionally the participant states

her opinion about website features, like the quality of the photographs, and how the tools are

user friendly or cause confusion. Occasionally the participant states her understanding of a

concept. Most often the participant reads the text and describes the website features. The

70

participant rarely expands on her decisions or provides an explanation for her behaviour. The

participant’s navigation and think aloud end at the 20-minute mark.

A Participant’s Experience with the Retrospective and Virtual Revisit Website Task

A participant is given the first set of instructions: the website task instructions. The

screen capture program begins to record the computer screen and the participant begins her 20-

minute navigation on the homepage. First, she moves the computer mouse over the text and

over images, and slowly scrolls down the page. After approximately two minutes, the

participant notices the virtual tour monthly features on the left side of the screen. She selects the

first and second grade virtual tour. The virtual tour opens and the participant hovers over the

hot spots to view the titles of the video clips. Then, she uses the arrows to maneuver around the

virtual tour; occasionally she stops to view an image on the wall. The participant opens three

videos during her virtual classroom tour. She watches two in their entirety and stops the third

video as it approaches the end. She closes the virtual tour and navigates back to the homepage.

From here, the participant hovers over all of the tabs before she selects motivation for literacy

within the food groups tab. When this web page appears, the participant once again hovers over

text and images before she slowly scrolls down the screen. Then the participant selects an

embedded video. As the video plays, the participant maneuvers around this food group page;

she scrolls down again and opens a recipe page. She views the recipe while the video on the

food group page continues to play—this participant is multitasking. When the video ends the

participant opens another page from the food groups tab, writing conventions. The same pattern

of behaviours continues on this page and the remaining pages the participant visits: she slowly

scrolls down each page, views videos, and opens links to educational resources and photo

galleries.

71

A Participant’s Experience with the Retrospective Think Aloud

“To me I think [thinking aloud] is an amazing tool. If you’re not shy about things and

you just trust the process there’s just a lot of learning that can come out it” (interview

transcript, retrospective condition)

A participant is given the second set of instructions, the think aloud instruction, and the

audio recorder begins to record the participant’s verbalizations. The participant begins her think

aloud with confidence and certainty of her initial web-based actions. She immediately connects

her web-based behaviour to her current practice. Occasionally the participant states her opinion

and shares how she would like to implement certain aspects from the website into her practice.

At about two minutes into her think aloud, the participant pauses and asks, “What else did I

do?” She lists a number of actions, and then shares her overall sense of the website and her

general experience with online resources. The participant concludes her think aloud by saying:

“That’s all, I think.” The audio recorder is stopped at four minutes.

A Participant’s Experience with the Virtual Revisit Think Aloud

“It’s like you’ve got these two narratives: you’ve got this physical manifestation of what

you’ve done and you know what you were thinking…and the storytelling of what was

going on…It’s like this disembodiment, but as you’re doing the storytelling you’re

bringing them back together again” (interview transcript, virtual revisit condition)

A participant is given the second set of instructions: the think aloud instructions. The

screen recording of this participants’ website navigation begins to play on the laptop computer

directly in front of him. At the same time an audio recorder begins to capture the participant’s

verbalizations. As the participant views the screen recording of his navigation, he describes his

actions on the homepage and includes a reason for his decisions. As the participant views the

screen recording of his navigation he completed only moments ago, he views the cursor on a

72

link to the charitable organization associated with the development of the website. The

participant shares his reason for making this decision—he is interested in the credibility of the

site. After the participant spends time looking at this outside link, he views the screen recording

return to the homepage. The participant states that he was interested in learning more about the

framework of the site. This is the reason he provides for selecting the introductory page. As the

introductory page appears, the participant verbalizes his understanding of the website in relation

to his prior knowledge about the stages of literacy and literacy instruction. Then, the participant

reflects that at this point in his navigation he was ready to delve deeper into the site. The

participant says that he was seeking information related to his current practice. Specifically, he

expresses interest in implementing integrated art and writing activities with his students. A

pattern of verbalizations related to planning and professional goals, as well as to his students’

interests and needs continues. The participant often remarks on the ease of use of the website

tools and features as they appear and is likely to provide an explanation or reason for his

navigational decisions. At one point, the participant’s think aloud begins to veer off in an

unrelated direction, but he is immediately pulled back into his navigation as the screen changes

and a video appears. At the 20-minute mark the computer screen becomes blank and the

participant completes his final thought related to his final action.

Benefits and Limitations of Employing Each Type of Think Aloud Method

The previous section of the general discussion shared portraits of participants’

experience with each type of think aloud method. These descriptions made implicit references

to the benefits and limitations of employing each type of think aloud method in the context of a

professional development website. This section discusses the main benefits and limitations of

employing each type of think aloud method based on the above theoretical portraits and results

of study 1. Table 3.11 provides a summary of this discussion.

73

Concurrent think aloud. Findings from this study reveal one main benefit and three

limitations of employing the concurrent think aloud method in the context of a professional

development website. First, the concurrent think aloud generates direct data about the ongoing

cognitive processes that occur during task performance. Since the two activities, thinking aloud

and task performance occur simultaneously the verbalizations are valid forms of information—

the verbalizations contain direct data about participants’ thoughts in real time. The limitations

of the concurrent method, however, may outweigh this benefit. The first limitation of the

concurrent think aloud method is the fact that this think aloud produces fewer verbalizations

related to higher order cognitive processes. Secondly, the concurrent think aloud method results

in a high cognitive load on working memory. In addition, the concurrent think aloud may be an

uncomfortable procedure for participants.

Retrospective think aloud method. Findings from this study show three benefits and

two limitations of employing the retrospective think aloud method. First, the main benefit of

employing the retrospective procedure is that this method produces verbalizations related to

higher level cognitive processes (planning, connecting, and reflecting). Secondly, participants

are likely to demonstrate a high level of engagement with the website task. In addition,

participants will most likely be comfortable during the process of thinking aloud. The

limitations of employing the retrospective think aloud method, however, are significant.

Participants will omit most of their online actions and navigational decisions. Furthermore,

participants will most likely have difficulty recalling their intermediate web-based actions and

reasons for these decisions.

Virtual revisit think aloud method. Findings from this study show four benefits and

one limitation of employing the virtual revisit think aloud method. First, the main benefit of

employing the virtual revisit procedure is that this method produces verbalizations related to

74

higher level cognitive processes (planning, reasoning, and reflecting). Secondly, participants

can rely on a visual aid to help them recall their navigational decisions and why they made

them. The third benefit of employing the virtual revisit think aloud method in the context of a

professional development website is that participants are likely to demonstrate a high level of

engagement with the website task. Finally, participants will likely feel comfortable with the

process of thinking aloud. One limitation of the virtual revisit think aloud is the time required to

complete both the website task and think aloud task. In the current study, the total time for each

participant in the virtual revisit condition was 40 minutes. For some researchers, this may be

costly. Moreover, the time required to employ the virtual revisit think aloud may limit

participant involvement.

Table 3.11

Benefits and Limitations of Employing Each Type of Think Aloud Method

Concurrent Think Aloud

Method

Retrospective Think

Aloud Method

Virtual Revisit Think

Aloud Method

Benefits Generates direct data

about the ongoing

cognitive processes

that occur during task

performance

Produces

verbalizations related

to higher level

cognitive processes

(planning,

connecting,

reflecting)

Participants likely

will feel comfortable

with thinking aloud

High task

engagement

Produces

verbalizations related

to higher level

cognitive processes

(planning, reasoning,

reflecting)

Participants can rely

on a visual aid to

recall decisions

Participants likely

will feel comfortable

with thinking aloud

High task

engagement

Limitations Produces

verbalizations related

to lower level

cognitive processes

(descriptive, reading)

Errors of omission

Difficulty recalling

intermediate actions

Time consuming to

conduct

75

High cognitive load

on working memory

Participants may feel

uncomfortable with

thinking aloud

Collectively, the findings of study 1 provide support for the virtual revisit think aloud as

an effective method for examining elementary teachers’ cognitive processes as they use a

professional development website. Results indicate that the virtual revisit can avoid the

limitations of the concurrent and retrospective procedures and provide thorough and descriptive

thought units and insights into how teachers use and learn from a professional development

website.

Study 1 Limitations

There were four main limitations to this study: (1) a wide range of variability in

participants within the conditions, (2) a small sample size, (3) factors that may have caused

reactivity, and (4) the use of one professional development website. First, some of the variables

had a large standard deviation indicating that the data points were distant from the mean or more

spread from the mean. While the effect sizes for the significant findings were fairly strong, the

wide range of variability needs to be taken into consideration when interpreting the findings.

Individual differences may account for the wide range of variability, including background

knowledge, motivating factors, and comfort level with speaking in front of adults.

Secondly, the small sample of teachers assigned to each condition indicates a decrease in

power of the statistical tests and therefore a cautious interpretation of the findings is

recommended (Field, 2009). While future research could benefit from a larger sample size of

20 or more teachers in each condition, the effort and cost in recruiting practicing teachers and

conducting the think aloud tasks are noted. Recruiting 45 practicing teachers to volunteer to

76

participate in the current study was a challenge and alternative modes of recruitment, including

word of mouth, were employed. Gift cards valued at $20 were given to teachers who

participated in this study; however a greater incentive to participate might result in a larger

sample size.

Thirdly, factors may have caused reactivity during the think aloud procedure. As

previously mentioned, reactivity occurs when task performance is altered as a result of an

awareness of the study task. Reactivity may have occurred as a result of participants’ awareness

that they were completing a task in the presence of the primary investigator. A “motivational

shift” in which the participants anticipate exposure of their think aloud protocol may have

occurred when the participants were informed of the think aloud procedure (Russo, 1989).

Another factor that may have caused reactivity is hearing one’s own voice. As previously

mentioned, the additional aural stimulation may have interfered with the concurrent navigation

(Russo, 1989). In general, reactivity was reduced as much as possible during the one-on-one

meeting by: staying neutral during the task and think aloud, keeping verbal and nonverbal cues

to a minimum, and providing participants in the retrospective and virtual revisit conditions with

the second part of the instructions after they had completed the website task. However, the

factors described above should be considered when interpreting the findings.

Finally, the current study was context-specific and used one professional development

website. Future research comparing the three think aloud conditions should be conducted with

additional websites and online resources. Conducting similar studies with alternative

professional development websites will enhance the credibility and transferability of the results.

Significance and Educational Implications

The significance of this research is that it compares the effectiveness of two traditional

think aloud methods—concurrent and retrospective—with a think aloud method combining a

77

retrospective procedure with screen capture technology (the virtual revisit). While think aloud

research is extensive, studies that compare the effectiveness of different think aloud

methodologies for understanding teachers’ cognitive processes as they navigate online resources

are scarce. Based on the findings of this study, the virtual revisit think aloud is an effective

method for examining elementary teachers’ cognitive processes as they use a professional

development website. The virtual revisit think aloud method produces rich and thorough

verbalizations related to teacher planning; the virtual revisit allows participants to incorporate

reasons behind the decision-making process; and the virtual revisit provides opportunities for

reflection. In addition, because participants think aloud immediately following the website task,

participants engage with the website as they might normally do when seeking information

online related to their practice. Participants will also likely feel comfortable with the think

aloud procedure.

Another significance of the current study is that the participants were practicing teachers

with between one and over 25 years of classroom experience. Much of the research examining

teachers’ beliefs and attitudes towards learning in online environments has studied preservice

teachers. Studying the cognitive processes of practicing teachers has the potential to contribute

to the understanding of teacher professional development and teacher cognition.

Furthermore, the virtual revisit think aloud method also has potential that transcends the

specific domain in which it is applied in the present study. Virtual revisits with think aloud

could be applied to examine cognitive processes of participants in research involving online

learning and website use in many domains in education. If researchers provide feedback to

educational website developers based on the virtual revisit methodology, improvements can be

made to the design and content of their sites.

78

Additionally, the virtual revisit think aloud method could be used as a tool for teacher

professional learning. The benefits of commenting on a screen recording of navigational

behviours in the context of professional learning may include encoding information and long

term memory storage. Many of the participants in the virtual revisit condition discussed their

own learning and self-reflection during their think aloud and how the process of viewing their

online behaviours while simultaneously thinking aloud had an impact on their reflective process

of teaching and learning. This process produced connections to prior knowledge allowing

teachers to encode information to long term memory.

Conclusion

Teachers learn in many different aspects of practice, including their classrooms, their

school communities, professional development courses, and online environments (Borko, 2004).

To understand teacher learning, it must be studied within these multiple contexts and it must be

studied with an effective methodology that provides rich and thorough data about the reasoning

process. Gaining greater insights into teachers’ cognitive processes as they navigate online

environments can lead to the reconsideration of the design of online learning environments so

that they “are more conducive to informal learning…so that they further develop the ability of

professionals to solve problems and learn independently” (Lohman, 2006, p. 144). Based on the

findings of study 1, the virtual revisit think aloud methodology provides a comprehensive

picture of elementary teachers’ cognitive processes as they use and learn from a multimedia

professional development website.

79

CHAPTER 4

Understanding Teachers’ Learning Experiences as They Use a Professional Development

Website: A Qualitative Study

The aim of study 2 is to contribute to the field of teacher cognition and online learning

by exploring how elementary teachers use and learn from an online professional resource.

Study 2 utilizes the virtual revisit think aloud method and two additional data sources (a

demographic questionnaire and a semi-structured interview) to examine elementary teachers’

learning experiences as they navigate a multimedia professional development website. As

previously described, the virtual revisit think aloud is a method that combines a retrospective

think aloud and screen capture technology. Results from study 1 suggest that the virtual revisit

think aloud method generates comprehensive information about teachers’ cognitive processes as

they make online choices. In the current study, the virtual revisit think aloud offers insights into

elementary teachers’ learning experiences as they use The Balanced Literacy Diet website.

Chapter 4 presents study 2 in six major sections: (1) literature review, (2) study purpose

and research questions, (3) methods, (4) findings and discussion, (5) general discussion, and (6)

limitations, implications, and future directions. First, a discussion of relevant literature in the

areas of adult learning, and formal and informal professional development sets the foundation

for study 2. Next, this chapter presents the study purpose and research questions. The third

major section of this chapter is the methods section which begins with participant demographics

relevant to study 2. The methods section continues with an overview of the research design,

data analysis, and the techniques used for establishing trustworthiness of the qualitative

findings. The fourth major section of this chapter presents the findings and discussion, and

highlights nine major themes related to the research questions. A general discussion follows the

nine themes. Finally, this chapter concludes with a discussion of the limitations, significance

and implications, and future directions.

80

The following review of the literature includes two main topics relevant to study 2: (1)

models of adult learning, and (2) formal and informal professional development. The first topic

highlights three models of adult learning. The first model is Malcolm Knowles’ theory of adult

learning (1980). Knowles was a theorist of adult education who popularized a theory comprised

of several underlying characteristics of the adult learner. Because study 2 investigates teachers’

learning experiences as they navigate a professional development website, Knowles’ theory

offers the current study a foundational view of the nature of teachers as learners.

The second model of adult learning is self-directed learning. As one of the primary

characteristics in Knowles’ theory of adult learning and particularly relevant to the use of

informal online environments, self-directed learning warrants further attention. The model of

self-directed learning incorporates three primary principles: (1) a self-initiated process of

learning, (2) personal autonomy, and (3) greater learner control (Caffarella, 1993). These

principles relate to teacher motivation for learning and to the context of informal online

professional development.

The third model presented in this chapter is Shulman and Shulman’s (2004) model of

teacher learning. Shulman and Shulman’s model emphasizes five aspects related to teacher

learning: vision, motivation, understanding, practice, and reflection. These aspects comprise a

process of teacher learning which has applications for online professional development

activities.

The second section of the literature review presents two types of professional

development: formal and informal. While the emphasis of the current study is informal

professional development in online environments, this chapter briefly discusses formal

professional development. Formal and informal professional development activities are

81

discussed with respect to elements of teacher learning in the context of online professional

development.

Theoretical Models for Understanding Teacher Learning

Knowles’ Theory of Adult Learning

Knowles’ theory of adult learning is based on the concept andragogy: “the art and

science of helping adults learn” (Knowles, 1977, p. 211). Grounded in the concept that adults

learn differently from children, Knowles popularized the term andragogy in the late 1960s when

he developed a theory about adult learning. Five underlying characteristics of the adult learner

can be drawn from Knowles’ theory of adult learning: (1) self-directedness, (2) the use of

experience as a source of information, (3) immediate application of knowledge, (4) need to

know, and (5) intrinsic motivation (Knowles, 1980; Merriam, 2001). What follows is a

discussion of each characteristic in relation to teacher professional development.

Self-directed learners are free from external control and constraint. Teachers are self-

directed learners when they choose educational topics that directly relate to their individual

practice and classroom context (Trotter, 2006). Choice in professional development activities is

important for two reasons: it reinforces teachers’ status as professional, and it increases the

chances that activities will apply to an individual teacher’s learning needs and professional

interests (Brockett & Hiemstra, 1985; Haymore Sandholtz, 2002; Knowles, 1980; Merriam,

2001). Teachers who pursue meaningful topics are active participants during the learning

process—they independently make choices that relate to their distinct learning pursuits

(Cercone, 2008; Knowles, 1980).

Professional experience is a source of information for teachers and can be highly

valuable during the process of learning. That is, when teachers create connections between new

incoming information and past experiences, the new information is more likely to be retained

82

for future use. Teachers can use their experience to verify the relevance of newly learned

material and to envision possible future implications (Cercone, 2008).

Teachers value material that they can easily and immediately transfer to their practice.

As teachers construct new ideas from professional development resources, they might envision

how the new information relates to their classroom and ultimately how it can contribute to

meaningful student learning experiences. Initially, teachers might experiment with their newly

formed knowledge, often tailoring new information to their unique environment. Teachers are

also more likely to commit to new instructional methods on a long-term basis when these

methods are successful with their students (Guskey, 2002).

Teachers can be considered goal-oriented and often need to know the reason behind the

learning. Specifically, teachers value the application of theory to practice. When teachers can

visualize the practical application of a theory, they may be more likely to embed theory into

their daily instructional activities. In addition, teachers may be more likely to invest in material

that they deem meaningful to their classroom environment and professional goals.

The application of newly acquired knowledge is more likely to occur when teachers are

intrinsically motivated—when engagement in learning is inherent without the mediating effects

of external rewards (Gorozidis & Papaioannour, 2014). That is, an internal drive to learn and

willingness to engage in professional development activities increases when the demands of

external factors (for example, mandatory workshops) are absent. In addition, engagement

increases when the material is relevant and personally meaningful to the learner (Chametzky,

2014).

Taken together, the above characteristics of the adult learner are relevant to teacher

professional development (Knowles, 1975; Knowles et al., 2011; Merriam, 2001; Schellenbach-

Zell & Grasel, 2010; Trotter, 2006). Teachers may be more willing to engage in professional

83

development opportunities that they perceive as meaningful; they may be more likely to gain

greater subject knowledge and transform their instructional practices when professional

development fosters self-directedness, connects to their experiences, and is easily and

immediately transferrable. In addition, when professional development is internally driven,

positive outcomes in teacher learning and in the application of theory to practice are more likely

to occur. In sum, teachers actively construct knowledge through “an interactive process of

interpretation, integration, and transformation of [their] experiential world (Pratt, 1993, p. 17).

For the purposes of study 2, Knowles’ theory provides a foundational view of the nature

of adult learning and can be applied to teacher learning in the context of online professional

development; however three criticisms of his theory require attention. The first criticism

pertains to the theory’s focus—the characteristics focus on the individual rather than on the

learning process (Hartree, 1984; Rachel, 2002). While an understanding of the characteristics of

adult learners can contribute to instructional designs and methods that best meet the needs of

adult learners, Knowles theory omits elements involved during the learning process; for

example: stages of learning, cognitive strategies, and potential learning outcomes. In addition,

the characteristics do not necessarily occur in a particular order; for instance, self-directedness is

not a requirement for using experience as a source of information.

A second criticism of Knowles’ theory relates to a developmental continuum; the

characteristics that describe the adult learner are not necessarily true for all adults, and can also

be true for children. Regardless of age, learners should be viewed along a continuum of each of

the characteristics, and individual differences affect how learners approach different situations

and learning activities. For instance, experienced teachers might feel very comfortable self-

directing their learning, whereas novice teachers might initially require guidance from an

administrator.

84

Finally, Knowles’ theory has been criticized for the limitations in the validity of the

theory—few researchers have performed empirical studies that directly investigate the theory

(Pratt, 1993; Rachel, 2002). Studies that do investigate Knowles’ theory show “considerable

variation in the implementation of andragogical methodologies” (Rachel, 2002, p. 217). That is,

Knowles’ theory lacks specific procedures for adult educators to undertake to best meet the

needs of adult learners (Taylor & Kroth, 2009). This lack of empirical evidence suggests

Knowles’ theory can only guide our understanding of how adults learn.

While the above criticisms are worth mentioning, Knowles’ characteristics of the adult

learner can be considered during the design and development of professional development

activities. Recognizing the characteristics of adult learners can foster a learner-centred approach

to teacher professional development (Cercone, 2008, p. 141).

Self-Directed Learning Theory

Self-directed learning is a well-established notion (Brockett & Hiemstra, 1985; Knowles,

1975) and stems from an individual’s desire to understand a phenomenon, an incident, or a

concept (Ponti, 2014). As previously mentioned, the model of self-directed learning

incorporates three principles: a self-initiated process of learning, personal autonomy, and greater

learner control (Caffarella, 1993). Each of these concepts is discussed with respect to teacher

professional development.

As a self-initiated process of learning, self-directed learning emphasizes “the ability of

individuals to plan and manage their own learning” (Caffarella, 1993, p. 25). The expectation of

self-directed learning is that the individual learner assumes “primary responsibility for their own

learning” depending on their unique needs and individual goals (Caffarella, 1993, p. 26). When

given the opportunity, teachers might self-direct their learning by first identifying their needs.

Then, teachers may decide which professional development approach will best meet their needs.

85

Approaches to professional development might include online professional development

activities, such as distance education courses or professional development websites. Teachers

might use Internet technologies and web-based resources to construct knowledge and identify

instructional materials relevant to their teaching context and professional goals.

A central trait of a self-directed learner is autonomy. Personal autonomy implies

“independence, freedom of choice, and rational reflection” (Loyens, Magda, & Rikers, 2008, p.

414); teachers may independently make choices according to their needs and interests. In

addition, personal autonomy is positively related to personal values and beliefs (Caffarella,

1993). A stronger sense of personal values and beliefs may contribute to a “solid foundation for

conceiving goals and plans” (Caffarella, 1993, p. 29). Teachers who conceive professional

goals and instructional plans that are personally valuable may be more likely to integrate these

ideas into their practice.

In addition, autonomy is individualistic (Candy, 1988) and autonomous behaviour ranges

according to several factors: the learners’ “level of technical skills, their familiarity with the

subject matter, their sense of personal competence as learners, and the context of the learning

event” (Caffarella, 1993, p. 30). These factors may affect how teachers approach professional

development, particularly when accessing web-based technologies. That is, experience with

technology may contribute to increases in autonomy during online learning activities.

Greater learner control implies that the learner is the one who considers the content,

approach, and value to the learning experience (Caffarella, 1993). With greater learner control

individual needs are more likely to be met in teachers’ quest for pedagogical knowledge and

instructional materials. Moreover, greater learning control may be linked to higher levels of

motivation for learning.

86

Taken together, professional development activities that foster a self-initiated process of

learning, personal autonomy, and greater learner control may contribute to higher levels of

motivation for learning and the application of pedagogical knowledge to the classroom.

Informal online learning environments lend themselves to a “self-directed learning experience”

(Song & Hill, 2007, p. 35). Teachers who initiate online professional development activities

with a sense of self-directedness, autonomy, and learner control may benefit from the

experience—they become motivated to learn and to implement best practices in their classroom.

Shulman and Shulman’s Model of Teacher Learning

The third theoretical model presented in this chapter for understanding how teachers

learn is a model proposed by Shulman and Shulman (2004). Shulman and Shulman highlight

five aspects of teacher learning: vision, motivation, understanding, practice, and reflection

(2004). Each aspect is discussed next.

Shulman and Shulman’s model of teacher learning begins with vision. As teachers enter

the classroom for the first time in their career, at the beginning of a new school year, or each day

they have a vision about their students’ learning, classroom environment, and instructional

methods. This vision guides teachers in their practice and can influence how they approach

professional development activities.

Along with vision, a teacher’s motivation and willingness to learn contributes to positive

outcomes, including the construction of new pedagogical knowledge and the implementation of

meaningful activities in the classroom. As previously described, teachers tend to be motivated

to learn; particularly in the absence of external demands as well as when material is relevant and

personally meaningful (Chametzky, 2014).

Through experience teachers develop an understanding of the elements that form the

knowledge-base of teaching; for example, pedagogical content knowledge, curriculum and

87

subject knowledge, and a developmental perspective of the learner. These understandings

contribute to their daily practice.

According to Shulman and Shulman (2004), a teacher’s practice is set upon a foundation

of vision, motivation, and understanding. When this foundation has been established, a teacher

can more easily consider a range of ongoing factors related to their practice; such as, the design

and adaptation of curriculum, classroom management, formal and informal assessments, and

student diversity (Shulman & Shulman, 2004). A teacher’s practice develops and has the

potential to transform through ongoing reflection. That is, when teachers are conscious of their

own understandings, performances, and dispositions, their capacity to learn is enhanced

(Shulman & Shulman, 2004).

Taken together, Shulman and Shulman’s model of teacher learning highlights five stages

of the learning process. Each stage can be applied to the context of online professional

development. That is, as teachers use the Internet for their professional learning they initiate

their search for pedagogical information with a vision about their students, teaching context, and

instructional methods. This vision is enhanced by a teacher’s willingness to learn—an internal

motivation. The construction of knowledge and development of deep understandings via online

resources contributes to meaningful practice. As teachers reflect on their learning and practice,

new visions arise and the cycle of professional learning continues.

In sum the three theoretical models discussed above highlight teacher learning as an

integrated set of various factors. Given that the current study examines the processes of teacher

learning in the context of a professional development website, the next section discusses formal

and informal online learning environments.

88

Formal and Informal Professional Development

Learning is an activity that occurs in a range of contexts. In any learning context—

formal, informal, or along a continuum—there is the potential for knowledge growth, changes in

beliefs, and transformations in practice (Bakkenes et al., 2010). Formal professional

development can introduce teachers to educational innovations and evidence-based methods of

teaching in content areas (Jurasaite-Harbison & Rex, 2010), and includes, but is not limited to,

face-to-face workshops and distance education courses. These types of professional

development activities are often structured, and usually guided by an instructor or set of

expectations. In contrast, informal professional development is usually open-ended, flexible,

and self-directed. Informal professional development activities are likely to coincide with a

teacher’s interests and often consider individual processes, real-life activities, and contextual

factors (Callanan, Cervantes, & Loomis, 2011). Teachers choose informal approaches,

including conversations between colleagues and online resources, to informally construct

knowledge and improve their instructional practice. Informal professional development

provides teachers with innovative strategies and research-based methods of instruction in a way

that directly relates to goals and interests of individual teachers.

Within the domain of informal professional development is independent teacher

learning: “learning activities that teachers engage in on their own initiative and accord, and

which possess no connection to their organization” (Jones & Dexter, 2014, p. 371). Independent

learning aligns with the characteristics of adult learning, as described above. For instance, a

central component of independent learning is self-direction, in which teachers independently

choose to pursue meaningful topics that directly relate to their personal interests and individual

needs. Independent learning contexts also promote the use of experience as a source of

information and allow meaningful connections between new incoming information and past

89

experience. Moreover, independent learning contexts are not restricted by time and location;

they are accessible, and convenient. This anytime-anywhere approach to professional

development promotes the immediate application of teachers’ newly formed knowledge, and the

reason behind the learning is often known. Furthermore, independent learning is often driven by

an internal motivation to learn—adults have a basic need for autonomy (Deci & Ryan, 2008). In

sum, independent learning contexts can consider the characteristics of the adult learner. In

addition, independent learning contexts lend themselves to a self-directed learning experience—

one that fosters autonomy and greater learner control.

A recent study compared formal, informal, and independent learning (Jones & Dexter,

2014). Findings indicate that while teachers reported general satisfaction with formal learning

opportunities, they identified constraints, lack of choice, and ongoing support as limitations of

these formal activities (Jones & Dexter, 2014). The findings also specified informal and

independent learning (i.e., Internet resources) as key components of teachers’ overall

professional learning experiences and independent learning as a primary activity crucial to

teacher learning (Jones & Dexter, 2014). The study also found that informal learning activities

were often initiated by teachers; participants in the study self-directed their own learning.

In another study that examined factors that influence teachers’ engagement in informal

learning activities, personal characteristics appeared to enhance teachers’ motivation to engage

in informal learning (Lohman, 2006). These personal characteristics include initiative, self-

efficacy, love of learning, interest in the profession, commitment to professional development, a

nurturing personality, and an outgoing personality. Findings also identified three factors that

inhibit teacher professional development: a lack of time, a lack of proximity to colleagues’ work

areas, and insufficient funds. These findings suggest that accessible online professional

90

development can offer an alternative to the more traditional and formal professional

development activities.

Informal online learning environments have become increasingly popular to teacher

professional development and the outcomes of informal online learning are positive: for

instance, informal online learning fosters self-directedness, topics of interest are easily

accessible, and there is a sense of control and choice in the learning. However, there are also

challenges to informal and self-directed online professional development. One challenge of

informal online learning is “that it is up to each teacher to interpret the professional

development content and translate it into specific action in their classroom” (Fishman et al.,

2013, p. 427). Teachers may have access to a curriculum leader or administrator for

clarification and guidance; however, the teacher independently accesses and interprets given

information. Another challenge of informal online learning is the lack of hands-on experience

and collaboration between colleagues. It is well-known that teachers come together to share

their experiences and guide each other in their learning—“teachers can be valuable resources for

each other’s learning in professional development” (Fishman et al., 2013, p. 427). During self-

directed online learning, collaboration may be limited. One way to overcome this limitation is

to encourage teachers to share learning resources during professional learning communities and

staff development sessions, especially resources teachers deem as credible and highly valuable.

The current study acknowledges collaboration as an important aspect to teacher

professional development and considers informal online learning as just one of the many facets

of professional development. Teacher professional development is complex and multifaceted,

and as teachers continue to use the Internet for their professional learning, it is necessary to

examine the processes and experiences involved during informal online learning. Informal

learning experiences have been shown to have a positive effect on, and even change, teachers’

91

beliefs, acquired knowledge and pedagogical approaches (Cervetti et al., 2012). Therefore, the

aim of study 2 is to contribute to the scholarship of teacher learning and professional

development by examining how teachers independently use and learn from one informal

professional development website.

Study Purpose and Research Questions

The purpose of this qualitative study is to gain an in-depth understanding of elementary

teachers’ learning experiences as they navigate a multimedia professional development website.

Despite extensive research on teacher learning and professional development, there are few in-

depth studies that examine elementary teachers’ learning experiences as they utilize informal

online resources. Specifically, little to no research has examined how teachers use and learn

from a multimedia professional development website, the cognitive processes and strategies

involved, and potential outcomes related to teacher learning and professional goals.

In-depth analyses of elementary teachers’ online learning experiences can offer a deeper

understanding of teacher learning within the complex field of continued professional

development. In addition, research in the area of informal online learning can offer website

developers and professional development administrators with suggestions pertaining to the

development and refinement of professional development websites. The following research

questions guided this study:

1. How do elementary teachers use a multimedia literacy professional development

website?

2. What factors influence how teachers use a literacy professional development

website?

3. What cognitive processes and strategies do teachers employ as they navigate a

literacy professional development website?

92

4. What are some of the potential outcomes of using and learning from a literacy

professional development website?

Methods

The methods section of this chapter describes the participants, research design process,

data analysis, and techniques for establishing trustworthiness of the findings. The methods

section begins with a description of the demographic characteristics of the participants in study

2 (the participants from the virtual revisit condition). Then, the methods section of this chapter

presents the research design and the overall process for study 2. Next, the data analysis

highlights three phases of coding: open coding, axial coding, and selective coding. The methods

section of study 2 ends with a description of the strategies and techniques used for establishing

trustworthiness of the qualitative findings.

Participants

Study 2 analyzes a sub-sample of participant data from study 1. Specifically, study 2

further examines data from the virtual revisit condition. The results of the demographic

questionnaire revealed that, in the virtual revisit sample, (N = 15) male participants (n = 3, 20%)

represented a smaller proportion than female participants (n = 12, 80%). The majority of

participants from the virtual revisit sample were between the age of 30 and 39 (n = 8, 54%) and

most participants had between five and nine years of teaching experience (n = 5, 33%).

Furthermore, most of the participants in the virtual revisit sample taught at the primary level (n

= 7, 47%). In addition, 13 participants (87%) reported feeling very comfortable using the

Internet for professional purposes and two participants (13%) reported feeling somewhat

comfortable using the Internet for professional purposes. The majority of participants in the

virtual revisit condition reported using the Internet for professional purposes at least once a day

93

(n = 7, 47%). Table 4.1 provides details concerning the demographic characteristics of the

participants in study 2.

Table 4.1 Demographic Characteristics for Study 2

Characteristic Frequency and Proportion

n (%)

Gender

Male 3 (20%)

Female 12 (80%)

Age

25-29 3 (20%)

30-34 4 (27%)

35-39 4 (27%)

40-44 2 (13%)

45-49 1 (6%)

50-54 0

55+ 1 (6%)

Years Teaching

2-4 years 4 (27%)

5-9 years 5 (33%)

10-14 years 4 (27%)

15-19 years 1 (6%)

20-24 years 1 (6%)

25+ years 0

Current Grade

Kindergarten 0

Primary 7 (47%)

Junior 6 (40%)

Primary/Junior 2 (13%)

Comfort Using Internet for

Professional Purposes

Very 13 (87%)

Somewhat 2 (13%)

Not very 0

Frequency of Internet Use

for Professional Purposes

> once/day 6 (40%)

Once/day 7 (47%)

Once/week 1 (6%)

Once/month 1 (6%)

94

Research Design

Study 2 employs qualitative methods to gain an in-depth understanding of elementary

teachers’ learning experiences as they utilize a multimedia professional development website.

Qualitative methods follow the post-positivist paradigm, which uses a naturalistic approach to

understand phenomena in a context-specific setting (Golafshani, 2003). In contrast with the

positivist paradigm, which seeks to explain, predict, and generalize to other contexts, the

primary aim of qualitative research methods is to describe, understand, and interpret phenomena

using an inductive approach to data analysis (Lincoln & Guba, 1985). An inductive approach to

analysis provides insights into how individuals experience a particular phenomenon and the

processes involved in a particular situation (Rowan & Huston, 1997).

There are several approaches to qualitative research (e.g., case study, phenomenology,

and ethnography) and choosing an approach depends on the study purpose and context. The

current study employs a grounded theory approach (Glaser & Strauss, 1967). The goal of

grounded theory is to generate a general explanation or theory of a process, action, or interaction

(Creswell, 2007). This general explanation or theory “is derived from data and then illustrated

by characteristic examples of data” (Glaser & Strauss, 1967, p. 5). Grounded theory was

deemed an appropriate qualitative approach for the current study for two reasons: (1) there is a

limited amount of research that examines in-depth how elementary teachers use and learn from

multimedia online learning environments, and (2) a theoretical model for elementary teachers’

online learning experiences as they use a professional development website can guide

researchers in the development and refinement of professional development websites.

The grounded theory research process for study 2 begins with three different data

sources from the virtual revisit condition: virtual revisit think aloud, interview, and demographic

data. Based on these data sources, the data set for study 2 includes three components: think

95

aloud transcripts, interview transcripts, and demographic questionnaire responses. These

components were analyzed during three phases of coding. The first and second phases of

coding, open coding and axial coding, involved an examination of the think aloud transcripts, as

well as memo writing and member checking. The interview transcripts and demographic

questionnaire responses were analyzed during the latter stage of axial coding. During the third

phase of coding, selective coding, the data were re-examined and a theoretical model was

developed.

The coding process led to nine major themes. These nine themes were organized into a

theoretical model, according to the grounded theory framework, and are discussed in the

findings and discussion section of this chapter. Figure 4.1 outlines the overall process for study

2. The following data analysis section describes the three phases of coding, memo writing, and

the member checking process.

Data Collection

Date sources:

demographic questionnaire

think aloud

semi-structured interview

Study Two Data Set

Three components:

15 think aloud transcripts

15 interview transcripts

15 demographic questionnaires

Phases of Coding and Procedures

1. Open coding:

Think aloud transcripts

Memo writing

2. Axial coding:

Think aloud transcripts

Interview transcripts

Demographic data

Member checking

96

Memo writing

3. Selective coding:

Think aloud transcripts

Interview transcripts

Demographic data

Nine Major Themes

1. Perception of professional learning

2. Focusing on student needs and

instructional goals

3. Individual differences

4. Navigating a professional

development website

5. Evaluating information

6. Encoding information for future

retrieval

7. Reflecting on learning

8. Continued professional learning

9. Intentions for practice

Theoretical Model Based on the

Grounded Theory Framework

Figure 4.1. Outline of overall process for study 2

Data Analysis

Data analysis in qualitative research involves reducing the data into themes as a result of

repeated coding, comparisons, and categorizations (Creswell, 2007). As mentioned above,

study 2 follows a grounded theory approach to data analysis and employs three phases of

coding: open coding, axial coding, and selective coding. These phases of coding involved data

categorization, comparisons, synthesis, and refinement, and led to a theoretical model for

elementary teachers’ online learning experiences as they used a professional development

website. As Charmaz (2014) states, “coding is the pivotal link between collecting data and

developing an emergent theory to explain these data…codes are created by defining what we see

in the data” (p. 46). The remainder of the data analysis section describes each phase of coding.

97

Open coding. In this study, open coding followed three stages. First, the think aloud

transcripts were read reflectively to gain a general sense of the participants’ thought processes.

Secondly, phrases within the transcripts were unitized based on meaningful pieces of

information that were “interpretable in the absence of any additional information” (Lincoln &

Guba, 1985, p. 345). The units consisted of simple sentences and longer phrases as opposed to a

specified number of words or lines within the transcripts. As the transcripts were unitized,

initial codes were generated based on participants’ actions and processes (Charmaz, 2014). For

example, the following unit was coded as tailoring information to teaching practice: “I was

reading the quick tips, and it seems easy enough that you can start a writing centre in any

classroom. You can tailor it very easily to the needs of your students.” The third stage of open

coding involved a process of constant comparison (Glaser & Strauss, 1967). As units were

coded, they were compared for similarities and differences. Units that were found to be similar

were grouped together. This resulted in an initial list of categories.

The initial list was extensive and included 77 categories (see Appendix C-1); so there

was an attempt to refine the list by, again, looking for instances of similarity. For example, the

units coded as connecting website content to future planning and classroom set up were found

to be similar to the units coded as constructing new ideas for future practice; therefore these two

categories were grouped together as knowledge construction. This process of comparing and

synthesizing categories began the next phase of coding and resulted in 18 main categories.

Memos were also produced during open coding. Memo writing provided a record of

initial thoughts and consisted of ideas related to possible connections between categories and the

grounded theory framework. As an example, the following memo describes initial thoughts

about the core phenomenon, navigating a professional development website, and the strategy,

encoding information for future retrieval:

98

As I’m beginning to read through the transcripts, it appears that participants reflect on

how they take in as much information as possible: they’re noticing images and materials

on the walls of classrooms, details behind a teacher who is speaking in a video, the

spatial organization of the classroom in relation to their own classroom and practice.

Participants attempt to absorb and remember information by encoding: they’re

connecting, taking notes, and planning.

Memo writing continued for the remaining two phases. Appendix C-2 presents a sample of

memos written during the analysis.

Axial coding. The second phase of analysis, axial coding, involved a review of the units

and categories. This review helped to establish connections between the categories and

subcategories and resulted in a refined list of categories. For example, the categories, current

grade and classroom context, narrowing down the focus, and reflecting on student needs and

goals, were grouped together under the theme, focusing on student needs and goals. This phase

of analysis identified seven preliminary themes:

(1) Focusing on student needs and goals;

(2) Evaluating information;

(3) Encoding information for future retrieval;

(4) Intentions for practice and professional learning goals;

(5) Reflecting on their learning and use of website;

(6) Valuing best practices in literacy and teaching;

(7) Feeling motivated for learning and applying new knowledge

To test the validity of these seven preliminary themes, member checking occurred next.

Member checking is a valid approach to establish the meaningfulness of the initial findings and

involves taking the ideas back to the research participants for their confirmation, elaboration, or

99

disagreement (Charmaz, 2014; Lincoln & Guba, 1985). Participants play a significant role in

contributing to the credibility of the findings; therefore, the participants (N= 15) were contacted

through email and asked if they were interested in providing feedback on the seven preliminary

themes. Specifically, participants received the following email script:

I am currently working on my analysis and as part of this process I am interested in

participant feedback. It will only take a few minutes of your time and your feedback

will be greatly appreciated. If you are interested in being involved I will share (email)

my preliminary findings with you and ask you to comment (as little or as much as

you would like) on seven preliminary themes. I will also email you the transcript of

your think aloud so that if you would like to, you can look over what you said during the

think aloud. Only I will have access to your comments and your name will not be used

in any paper or report.

Ten participants replied and were willing to provide feedback on the preliminary themes. These

participants were asked to review and comment on the preliminary themes in a follow-up email.

Appendix C-3 presents the themes and short descriptions sent to the participants. In general,

participants confirmed and elaborated on the seven themes: all of the participants involved in the

member check agreed with the themes. Appendix C-4 presents participants’ ‘member check’

comments.

Member checking contributed to the analysis in two ways: (1) by verifying the major

themes, and (2) by refining the major themes. First, the member check verified the validity of

the preliminary themes, as in the following example. In response to the theme, focusing on

student needs and goals, one participant stated, “I would definitely agree with this—when I

search, the most important thing I would initially be thinking about is: Is this relevant and

applicable to my students (age/grade/level) and will they benefit from it?” Another participant

100

made a similar connection: “This makes sense to me, as a teacher’s first goal is to support their

students. It’s the first check of how I evaluate any and all new resources.”

Member checking also aided in the refinement of themes. For example, the preliminary

theme, feeling motivated for learning and applying new knowledge, was refined into two

subthemes of a potential outcome, intentions for practice.

As a result of axial coding and member checking, nine major themes emerged. These

major themes were used to code the interview and questionnaire data. The purpose of analyzing

the interview and questionnaire data was to determine the consistency of the nine themes across

data sources. Appendix C-5 presents a distribution chart showing the frequency of themes

across data sources and across participants. The following section describes the last phase of the

grounded theory analysis, selective coding.

Selective coding. Selective coding is also known as theoretical coding. This process

involves selecting the core category and specifying relationships between the core category and

other categories to develop a theoretical model (Charmaz, 2014). In the current study, the

process involved organizing the nine major themes into a model based on four components of

the grounded theory framework:

(1) Conditions that affect the core category;

(2) The core category also known as the central phenomenon;

(3) Strategies as a result of the core category; and

(4) Potential outcomes related from the core category

The central phenomenon, navigating a professional development literacy website, was deemed

the core category because of the following three reasons, as outlined by Corbin and Strauss

(2008):

(1) the central phenomenon represents the main theme of the research;

101

(2) all other categories can be related to it; and

(3) it appears frequently and consistently in the data.

The findings and discussion section presents the model for elementary teachers’ learning

experiences as they navigated a professional development literacy website. The findings and

discussion section also describes the nine major themes and includes direct quotes to support

each theme. Preceding the findings and discussion of study 2, the remaining methods section

highlights the strategies and techniques used for establishing trustworthiness of the qualitative

findings.

Establishing Trustworthiness

Lincoln and Guba (1985) provide a framework for establishing trustworthiness in their

book, Naturalistic Inquiry. This framework guides researchers in establishing rigor of

qualitative findings (Agostinho, 2005; Corbin & Strauss, 2008; Glesne, 1999; Golafshani, 2003;

Leech & Onwuegbuzie, 2007; Shenton, 2004) by outlining four strategies of trustworthiness

(credibility, transferability, dependability, and confirmability), and providing techniques to

enhance the trustworthiness of qualitative findings. Following Lincoln and Guba’s (1985)

framework, study 2 achieved trustworthiness through the following six techniques: (1) member

checking, (2) triangulation, (3) frequency tallies, (4) thick descriptions, (5) compiling an audit

trail, and (6) screen shots of the analysis process. The remainder of this section describes each

technique under the headings: credibility, transferability, dependability, and confirmability.

Figure 4.2 outlines the techniques used to achieve trustworthiness of the findings.

Establishing Trustworthiness

1. Credibility

a. Member checking

b. Triangulation

c. Frequency of categories across participants

102

2. Transferability

a. Thick descriptions

3. Dependability

a. Audit Trail

4. Confirmability

a. Triangulation

b. Audit Trail

Figure 4.2. Topics pertaining to establishing trustworthiness

1. Credibility. The first criterion for establishing trustworthiness is credibility—the

believability of the findings and interpretations; it is the truth value and validity of the findings

(Lincoln & Guba, 1985). To enhance the credibility of the findings and interpretations of study

2, three techniques were used: member checking, triangulation, and tallying the frequency of

categories across participants.

a. Member checking. Member checking has been suggested as “the most crucial

technique for establishing credibility” (Lincoln & Guba, 1985, p. 314). As previously

mentioned member checking occurred during the axial phase of analysis. The process involved

the participants’ feedback on seven preliminary themes and is described above. Member

checking contributed to the credibility of the findings by verifying the validity of the themes

through participant feedback.

b. Triangulation. Triangulation involves the convergence among multiple sources of

information (Golafshani, 2003). According to Lincoln and Guba (1985) four different modes of

triangulation exist—the use of multiple and different sources, methods, investigators, and

theories. The current study collected and analyzed three different sources of data—think alouds,

interviews, and questionnaires. Analyzing the interview and questionnaire data, in addition to

the think aloud transcripts, helped establish the consistency and frequency of categories beyond

the think aloud transcripts (see Appendix C-5). Furthermore, the interview allowed participants

103

to expand upon their think aloud and answer direct questions about their website navigation.

Because prompts, questions, and redirections were limited during the think aloud, the post-think

aloud interview provided opportunities for participants to expand upon their ideas and follow up

on their decision making strategies. Triangulation contributed to the credibility of the findings

by adding depth to the findings. Selected quotes from the three data sources are included in the

results and discussion section.

c. Tallying the frequency of categories across participants. A frequency chart was

developed to determine the frequency of the categories across participants. As previously

stated, categories across participants were tallied for each data source. This process contributed

to the credibility of the findings by confirming the consistency of the final categories across

participants.

2. Transferability. The second criterion for establishing trustworthiness is

transferability—the potential applicability of the results to other contexts. Transferability

depends on the degree of similarity between the original research context and contexts in which

the results may be transferable (Lincoln & Guba, 1985). Lincoln and Guba (1985) caution

qualitative researchers that “the naturalist can only set out working hypotheses together with a

description of the time and context in which they were found to hold” (p. 316). In qualitative

research, transferability is not about generalizing the findings; rather transferability is about

“providing the database that makes transferability judgements possible” (p. 316). To enhance

transferability in the current study, thick descriptions were used.

a. Thick descriptions. In this study, thick descriptions include details of the context and

study participants, and direct quotes and interpretations presented in the findings and discussion

section. That is, participants’ actions and experiences are “thickly described…so that thick

interpretations of the actions can be made, presented in written form, and be made available to a

104

wide audience of readers” (Ponterotto, 2006). Thick descriptions enhance the potential

transferability of the findings. That is, descriptions of the website used in the current study will

allow future researchers to undertake similar studies using any professional development

website across a range of domains. In addition, extensive participant quotes can provide

researchers with a basis of the think aloud transcripts and units of analysis used in this study.

3. Dependability. The third criterion for establishing trustworthiness is dependability—

the consistency of the findings. Dependability is typically demonstrated by replication (Lincoln

& Guba, 1985). That is, if two or more studies yield similar findings then the reliability of the

findings has been established. Future studies attempting to replicate the findings of the current

study must consider the process of data analysis; therefore, the current study enhanced the

dependability of the findings through the compilation of an audit trail.

a. Audit trail. The audit trail in Appendix C-6 includes samples of raw data and a sample

screen shot from NVivo that shows the nine major themes. In addition, the audit trail includes

figures of the theory construction. An audit trail contributed to the dependability of the findings

by documenting the development of the analysis, and providing, as much as possible, an account

of the research decisions and activities throughout the study (Carcary, 2009).

4. Confirmability. The last criterion cited by Lincoln and Guba (1985) is

confirmability—ensuring, as much as possible, that the findings emerged from the data and not

the predispositions of the researcher (Shenton, 2004). Confirmability was achieved in this study

through triangulation and the audit trail. As previously described, these techniques provide the

reader with information about the consistency of the findings (triangulation) and the research

activities throughout the course of the study (audit trail). In addition, the primary investigator

was also aware of the potential overlap in categories between study 1 and study 2. While it may

be impossible to reduce researcher bias completely, the primary investigator approached the

105

analysis of study 2 with an open mind and attempted to code units of analysis by staying as

close to the data as possible.

In sum, several techniques were employed to enhance the trustworthiness of qualitative

findings: (1) member checking, (2) triangulation, (3) frequency tallies, (4) thick descriptions, (5)

compiling an audit trail, and (6) screen shots of the analysis process. While these techniques

have been cited by many qualitative researchers as strategies to enhance the trustworthiness of

the qualitative findings, a cautious interpretation of the findings is recommended since the

current study is context-specific and exploratory in nature.

Findings and Discussion

The previous section described the methods of study 2, including participant

demographics, the research design process, data analysis, and techniques for establishing

trustworthiness. This section presents the findings and discussion of study 2. As previously

described, the primary purpose of study 2 was to gain an in-depth understanding of elementary

teachers’ learning experiences as they navigated a professional development website. The

findings presented in this section represent the participants’ learning experiences as they used

The Balanced Literacy Diet website. These findings are organized according to nine major

themes that resulted from the analysis. Table 4.2 presents a description of each theme and an

example of each theme from the think aloud transcripts.

Table 4.2

Study 2 Themes

Theme Description Example from the Transcripts

1. Perception of

Professional

Learning

Ideas and attitudes related to professional

learning and literacy instruction,

including credibility and trustworthiness,

and research-based practices.

“If the information is accurate and

well-presented, teachers will go

back to the website. There are so

many websites, there’s so much

information...I need to know the

information here is valid.”

2. Focusing on

Student Needs and

A connection in reference to a teaching

context, students, and current and future

“I’m gathering ideas and

information that are relevant to

106

Instructional Goals

instructional practices. my students’ needs and interests.”

3. Individual

Differences

Factors relating to motivation, online user

behaviour, prior experience, and

professional knowledge.

“I find it really helpful because I

can see what the classrooms are,

and sometimes there are lessons

and links that I can look into

which helps me as a learner and as

a teacher.”

4. Navigating a

Professional

Development

Website

Descriptions and explanations of website

actions and behaviours based on

participants’ exploration and

understanding of the website.

“First I was trying to get a general

sense of the website and what was

on it.”

5. Evaluating

Information

Evaluative statements (assessments,

opinions) in relation to participants’

navigation.

“It was nice also that you could

sort by grade and then re-filter by

the food groups.”

6. Encoding

Information for

Future Retrieval

Strategies for remembering and recalling

information including note taking,

bookmarking, and emailing.

“I definitely at this point wanted

to record the title of this book, so

I’m going to open up a document

so I can start typing out my

ideas.”

7. Reflecting on

Learning

Reflective statements about learning from

the website and seeing one’s self as a

learner, including feelings towards the

website, future use of the website, and

navigational choices.

“As a new teacher it’s really

important to go over things and

see examples of what people

expect.”

8. Continued

Professional

Learning

Statements relating to the advantages of

online professional learning, increased

self-efficacy, feeling inspired, and

knowledge construction.

“Here I’m just having a look at

this classroom…and it can remind

you of things, it can show you

things that you haven’t thought of

and give you inspiration and new

ideas.”

9. Intentions for

Practice

Statements about planning, adapting

information to meet student needs, and

professional goals.

“I can tell with this activity that

it’s multi-layered, it can be turned

into a whole unit and it can allow

me to develop many different

skills, literacy skills and other

skills with the students.”

107

The nine themes listed in Table 4.2 were organized into a theoretical model based on the

current study’s research questions as well as four components of the grounded theory

framework (Strauss & Corbin, 1998). Figure 4.3 presents the theoretical model for elementary

teachers’ learning experiences as they navigated a professional development website. The

remaining chapter presents and discusses each theme and pertinent topics under the following

headings: (1) conditions affecting elementary teachers’ website navigation, (2) the central

phenomenon resulting from the conditions, (3) strategies for navigating a professional

development website, and (4) potential outcomes resulting from elementary teachers’ website

navigation. Direct quotes are included to support each theme.

Elementary Teachers’ Learning Experiences as they Navigate a

Professional Development Website

Conditions Affecting Central Phenomenon Strategies for Navigating

Navigation a Professional Development

Website

Theme 1:

Perception of

Professional Learning

Theme 4:

Navigating a

Professional

Development Website

Theme 5:

Evaluating

Information

Theme 2:

Focusing on Student

Needs and Instructional

Goals

Theme 6:

Encoding

Information for

Future Retrieval

Theme 3:

Individual

Differences

Potential Outcomes of Navigating a Professional Development Website

Theme 7:

Reflecting on Learning

Theme 8:

Continued Professional

Learning

Theme 9:

Intentions for Practice

Figure 4.3. Theoretical model for elementary teachers’ learning experiences as they navigated a

professional development literacy website.

108

Conditions Affecting Elementary Teachers’ Website Navigation

The first three themes of the theoretical model are the conditions affecting elementary

teachers’ website navigation. These themes emerged as a result of the analysis and include:

Theme 1: Perception of Professional Learning; Theme 2: Focusing on Student Needs and

Instructional Goals; and Theme 3: Individual Differences. Each theme and pertinent topics are

discussed separately below.

Theme 1: Perception of Professional Learning

In the current study, Theme 1: Perception of Professional Learning refers to the

participants’ ideas, attitudes, and beliefs related to professional learning resources and

instructional practices. Participants’ thought units were classified through the data analysis into

two main groups: (a) quality of professional resources, and (b) beliefs about literacy instruction.

Figure 4.4 presents the topics pertaining to the first theme.

Theme 1: Perception of Professional Learning

1. Quality of Professional Resources

a. Website authenticity

b. Background information

c. Source of recommendation

2. Beliefs about Literacy Instruction

Figure 4.4. Topics pertaining to theme 1: Perception of professional learning

1. Quality of Professional Resources

In the current study, participants identified three source features that they believe can

affect the credibility, trustworthiness, and overall quality of a given professional resource: (a)

website authenticity—whether the content is believable and reflects realistic contexts and real-

life strategies, (b) background information—the expertise of the developers and supporting

organization, and (c) source of recommendation—such as, a trustworthy colleague or

109

administrator. All of the participants felt confident in their use of professional resources with

the presence of these source features.

a. Website authenticity. Participants were willing to trust the information on the given

website because it included authentic contexts and realistic examples—the information appeared

believable. For example, one participant commented on the authenticity of the virtual

classrooms: “I think it’s a great website, I think that there’s so much value on it because it’s

authentic in the fact that you can see into people’s classrooms.” In addition, a participant who

viewed an activity related to vocabulary growth thought “it was interesting in that they found

one specific word in an article it looks like, and they used that as their challenging word. So

that’s the teacher showing a real life example.” Similarly, another participant described in an

interview how “the real life photographs and images of the different parts were the most eye-

catching…it was a real classroom, it wasn’t a theoretical place.”

The above examples suggest that elementary teachers value professional information that

accurately reflects their own practice. By viewing real-life examples of classroom environments

and authentic teaching practices, elementary teachers can make connections between new

information and their own practice as well as visualize possible future activities. Moreover,

real-life examples can contribute to a heightened teacher self-efficacy—teachers gain

confidence in their own teaching abilities when they view realistic models to which they can

relate.

b. Background information. Along with authentic examples, participants noted how

the background information—for example, the supporting organization—affects a website’s

credibility and overall quality. For example, one participant described her thoughts about the

credibility of the given website and included her interests in understanding the background

information:

110

I was still reading about the Institute’s mission and what its goals were. And I guess I’m

doing this to make sure that the website’s credible and that maybe if nothing jumps out

that contradicts my own pedagogical practice or my own ideas around teaching

literacy…And then I was interested in who was sort of behind the Institution so I clicked

on the ‘about us’ button and read a little bit about the history about why the Institution

was set up. And then I scrolled down to read about the members of the board.

As a result of gaining information about the website’s background, this participant determined

whether the website corresponded to her own practice and beliefs about teaching literacy.

Another participant acknowledged the website’s university association during an

interview: “I like that [the website is] from the [university] site, because I think that does give it

credibility.” Again, information about the website’s developers and supporting organizations

contributed to participants’ willingness to trust the information presented on the given website.

The background information was easily accessible and offered the participants a sense of ease in

their quest for professional knowledge and materials.

c. Source of recommendation. The source of recommendation (a colleague or school

administrator) also appeared to affect the website’s credibility. For example, as one participant

viewed the screen recording of his navigation he wondered about the credibility of the

supporting organization, one with which he was unfamiliar:

I’m also wondering what the Melissa Institute is, because I didn’t go here, you know I

didn’t navigate here from somewhere else in a way that makes sense to me. Or I didn’t

hear about this particular website from someone that said, ‘hey you really got to check

this out’; you know, from a colleague or from a workshop of some kind.

Another participant had a similar concern: “Because no one recommended [the website] to me,

there’s the element of trust missing initially.” Similarly, in a questionnaire response a

111

participant presented the benefits of using online professional resources and reasons why she

favoured school-board websites: “It brings together links to resources in one easily accessible

place…the links have already been checked for authenticity and are ‘Board Approved’ and this

takes the leg-work out of the process for me.” These examples suggest that teachers value their

colleagues’ recommendations and opinions about professional learning. In turn, source

recommendations appear to influence teachers’ use of a professional resource.

Taken together, the website’s authenticity, background information, and source of

recommendation appear to affect the credibility, trustworthiness, and overall quality of the given

website. Over half of the participants in the current study sought information related to the

website’s credentials; some participants noted the journal citations on various web pages; and

participants stated that in general, credibility and trustworthiness impacts their future use of any

type of professional development resource.

It is well-known that online resources vary in terms of the quality of information they

provide. Research that has investigated the trustworthiness of online resources is based on

university student and high school student responses (Iding, Crosby, Auernheimer, & Klemm,

2009; Wiley et al., 2009). Findings of such studies indicate that many Internet users interpret

“superficial or inadequate source features (e.g., popularity)” when judging a website’s

trustworthiness (Andreassen & Braten, 2013, p. 823). The findings pertaining to theme 1of the

current study reveal the use of effective source features by elementary teachers when judging a

website’s trustworthiness. Elementary teachers want to ensure that their professional knowledge

and instructional materials stem from credible sources, and examining these source features

during professional learning enhances their confidence in the resource.

112

2. Beliefs about Literacy Instruction

In the current study, all of the participants held positive beliefs about research-based

practices in literacy and often referred to the components of effective literacy instruction.

Specifically, participants perceived The Balanced Literacy Diet website as believable and

expert-driven because it presented “good evidence of modern research and instructional

strategies.” The website presented “great ideas because they connect to the research and how

that’s all supported.”

One participant stated in the questionnaire that she “thinks it’s important to keep up with

best practices in education,” and that she enjoys “interacting with experts and other teachers” to

gain knowledge about meaningful teaching methods. Additionally, participants described the

importance of staying “current to learn about innovative teaching practices.” Participants also

sought innovative strategies that align with their views of literacy instruction. For example, one

participant stated that he “was happy that there were good examples of criteria lists…its

reinforcing effective discussion within the classroom and [the website] reinforces the concepts

of what the ministry and the literacy/numeracy secretariat is trying to promote.”

Participants referred to several other research-based literacy practices during their think

aloud. For example, one participant referred to the importance of including a variety of print

materials in the learning environment: “Remembering that the balanced literacy program

includes fiction, non-fiction, all types of literature to make sure that, you know, kids are getting

a good variety.” In addition, a participant described the practice of teacher-student

collaboration:

So it’s really important to me that the students are clear on the learning goals of each

subject area of the class. We brainstorm together what the learning goals will be in each

subject and the students actually come and write the learning goals with me. They

113

become a reference point for them if they are ever wondering what we’re working on

they can easily come to the board.

Participants also referred to:

Differentiated instruction—“I liked the idea that it was reinforcing differentiated

instruction and having books at different levels,”

Motivation for literacy through games—“I like this idea of, you know, connecting new

vocabulary with a game and some sort of fun activity that we can do with it,”

Integrated instruction—“And I liked again, the link to a different subject area, so linking

that to social studies and science, I thought it was really great that she combined her

math with language,” and

Student accommodations—“I also like the idea of colour coding. That’s something that I

should do more. I think it can support some of my students with learning

disabilities…some students I think would find it helpful, when we’re writing paragraphs

or any sort of writing that can be kind of compartmentalized to learn the form.”

The above examples demonstrate participants’ positive beliefs about research-based

practices. Many of these examples are consistent with the evidence-based components of high

quality literacy instruction (Gambrell & Anders Mazzoni, 1999; National Reading Panel, 2000;

The National Early Literacy Panel, 2008). That is, there is a general agreement of what

constitutes best practices in literacy instruction (see Table 4.3). The findings pertaining to

theme 1 suggest that teachers are cognizant of these practices when they approach professional

development websites.

114

Table 4.3

General Agreement of What Constitutes Best Practices in Literacy Instruction

Best Practices in Literacy Instruction

Teach reading for authentic meaning-making literacy experiences

Use high-quality literature,

Integrate a comprehensive word study/phonics program into reading/writing instruction,

Use multiple texts that link and expand concepts,

Balance teacher- and student-led discussions,

Give students direct instruction in decoding and comprehension strategies that promote

independent reading,

Use a variety of assessment techniques to inform instruction,

Explicit instruction in reading skills, including letter-sounds, phonics, spelling, vocabulary,

word recognition, and comprehension skills

Note: From Gambrell & Anders Mazzoni, 1999; National Reading Panel, 2000; The National

Early Literacy Panel, 2008.

Taken together, the findings pertaining to theme 1 suggest that elementary teachers

approach professional learning with particular beliefs, values, and ideas. In turn, their

perception of professional learning may affect the short- and long-term use of an online

professional resource.

Theme 2: Focusing on Student Needs and Instructional Goals

As teachers approach online resources they carefully reflect on the relevance and

applicability of the given resource. All of the participants in the current study sought

information specific to their practice; they were mindful of the current and future needs of their

students and their instructional goals. Theme 2: Focusing on Student Needs and Instructional

Goals is the second theme that emerged from the analysis and consists of the following topics:

(1) student needs, (2) teaching context, and (3) current and future instructional goals. Figure 4.5

presents the topics pertaining to the second theme.

115

Theme 2: Focusing on Student Needs and Instructional Goals

1. Student Needs

2. Classroom Context

3. Current and Future Instructional Goals

Figure 4.5. Topics pertaining to theme 2: Focusing on student needs and instructional goals.

1. Student Needs

As participants navigated The Balanced Literacy Diet website their web-based actions

and behaviours were affected by the needs of their current students. Participants were interested

in “gathering ideas and information that [were] relevant to [their] students’ needs and interests.”

For example, one participant stated: “when I was reading the different recipe elements I was

specifically looking for key words that target some of the things I do in my special education

classroom.” As this quote demonstrates, participants were consistently referring to their

students’ needs and interests during the 20-minute website navigation. Participants had

individual interests as they sought instructional materials. This finding suggests that teachers

value professional resources that support their students.

2. Classroom Context

Participants also based their navigational decisions on their classroom context. For

instance, one participant stated:

I teach grade one/two, so clicking on the first to second grade virtual tour and just having

a look at the classroom it shows and then looking down below to see some of the

particular recipes I guess that are in that tour…And looking at just the little snapshots

and the description underneath for anything I might be interested in.

This participant chose topics directly relating to her individual classroom context. In this sense,

she was self-directing her learning. This finding suggests that teachers who are given the

116

opportunity to choose pertinent topics are likely to gain relevant information with immediate

applications.

3. Current and Future Instructional Goals

As participants navigated the website, they also reflected on their current and future units

of study. For example, as one participant viewed a photo gallery of an integrated math and

literacy activity, she stated: “we’re looking at symmetry right now, so that seemed really

interesting.” Similarly, as another participant viewed a recipe about literature circles she

referred to a future instructional goal: “I’m wanting to start literature circles with my class, like

maybe this week or next week, so here I’m having a look at literature circles and just reading

through this one in some detail.” In addition, while navigating the given website, a participant

focused on a programming goal: “I’m hunting for anything related to grade two and lately I’ve

been wanting more word work activities…so I’m looking for any word work or sight words.”

In sum, participants in the current study sought information that directly applied to their

students’ needs, classroom context, and instructional goals. This finding is consistent with two

characteristics of the adult learner: self-directed learning, and immediate application of

knowledge. Given that the participants were directing their navigation themselves, they were

free to choose topics that best fit their needs and current context. There was also a sense of

immediate application—participants had a vision for their practice and recognized that the

information could be immediately applied to their current and future units of study.

Theme 3: Individual Differences

Individual Differences is the third theme of the current study’s theoretical model.

Individual differences were classified through the data analysis into three categories: (1)

motivating factors to use online resources, (2) online user behaviour, and (3) prior experience

and professional knowledge. Figure 4.6 presents the topics pertaining to theme 3.

117

Theme 3: Individual Differences

1. Motivating Factors to Use Online Resources

a. Ease of use

b. Personal interest to expand professional knowledge

2. Online User Behaviour

a. Forms of communication

b. Multitasking behaviours

c. Type of learner

3. Prior Experience and Professional Knowledge

a. Professional challenges

b. Area of expertise

Figure 4.6. Topics pertaining to theme 3: Individual differences.

1. Motivating Factors to Use Online Resources

Teachers are generally motivated to use online resources for their professional learning

(Charalambousa & Ioannou, 2011). In the current study, 11 participants referred to two factors

that contribute to their motivation to use online professional resources: (i) the ease of use of

online resources, and (ii) their personal interest to expand professional knowledge.

a. Ease of use. Participants agreed that the ease and accessibility of the Internet is a

motivating factor to use professional development websites. For example, one participant

preferred the convenience of online resources as opposed to professional texts:

I find [professional development websites] really convenient…instead of going to the

library and taking out books, things are already narrowed down for you and the

important things are already there, in a sense that you don’t have to search and waste so

much time…it’s a lot easier.

Similarly, another participant stated that “the Internet is very portable these days…you can use

it on your phone, outside the house…it’s very convenient to use online resources.” In addition,

a participant described the convenience of the Internet over a professional library of texts and

118

books: “I find [the Internet] is much nicer than having to collect books and having a huge library

of books…sometimes you might only want a small segment of it or something like that…so I

appreciate having access to online resources.”

These examples suggest that access to professional development websites offers

information that teachers may not as easily access through more traditional face-to-face

professional development activities. This is consistent with other research that has examined

teachers’ motivations towards online learning (Jones & Dexter, 2014; Recker et al., 2007).

Specifically, teacher reports have indicated that informal online environments are flexible in

terms of time and location, and can provide on-going and “just-in-time” support (Jones &

Dexter, 2014). These just-in-time supports can include accessible and easy-to-use professional

development websites—teachers can expand their professional knowledge and repertoire of

instructional strategies at their own time and convenience.

Additionally, online professional resources make it easier for teachers to easily and

immediately transfer material to their practice. This is consistent with one of the characteristics

of the adult learner: adults value material that they can easily and immediately transfer to their

practice. Professional development websites offer elementary teachers with timely professional

knowledge and instructional material; a teacher might wish to immediately follow up on her

students’ learning, and can quickly search for relevant supports and strategies.

b. Personal interest to expand professional knowledge. Participants referred to their

motivation to expand their repertoire of professional knowledge by seeking out research-based

practices. For example, one participant, who identified herself as a new teacher, stated in a

questionnaire response “that it is important for [her] to continue learning new strategies

(especially in the area of literacy).” She reported that “conferences and workshops are inspiring

to go to…they provide new and creative ways to teach.” Similarly, another participant reported

119

that “it’s important to keep up with best practices in education.” This participant stated that she

“enjoys interacting with experts and other teachers,” and that she is “always looking for new

ideas to bring back to [her] classroom.”

This finding is consistent with the characteristics of adult learners. As previously

discussed, teachers’ willingness to engage in professional development increases when the

material is relevant and personally meaningful to them (Chametzky, 2014). Professional

development websites lend themselves to greater learner control and autonomy, and ultimately

intrinsic motivation to learn. As a result of self-directing their own learning, teachers may be

more willing to implement best practices in the classroom.

2. Online User Behaviour

The second individual difference found as a result of the analysis is differences in

participants’ online user behaviour. Three main differences in online user behaviour emerged

from the analysis: (i) forms of communication, (ii) multitasking behaviour, and (iii) types of

online learners.

a. Forms of communication. Participants in the current study most often referred to two

forms of communication: visual media—visual representations of information—and

multimedia—a form of communication that includes more than one type of media element

(Mayer, 2002). Both forms of communication appeared to play a role in participants’

navigational experiences. Specifically, three participants described how visual media, including

photo galleries and virtual tours, contributed to their online learning experience. For example,

one participant stated:

I find [visual tools] really helpful because I can see what the classrooms are, and

sometimes there are lessons and links that I can look into which helps me as a learner

and as a teacher, so I can be like, ‘oh this is a good idea, maybe I can use in my class’.

120

Similarly, a participant indicated: “I’m always looking at visuals and how things are organized. I

get a lot visually from websites. I like the layout of the movie and the print. All of this is a nice

design.” In addition, a participant verbalized her thoughts about visual media as an important

form of communication during planning: “I find it really important to see actual examples and

that’s what I tend to do when I’m using online resources to help me plan my lessons.”

One participant who accessed a video of an exemplary teacher describing a lesson thinks

aloud about the benefits of this audio-visual modality:

Wow, this is really interesting—that there are videos too. It’s really helpful to me

because that’s the best way I like to see things—get a really quick look at what’s going

on, and what they have to say.

Furthermore, three participants described how the videos fostered a personal approach to

professional learning and that this personal approach contributed to the retention of information.

For example, one participant referred to the personal approach taken with the videos:

It’s neat because it feels like she’s actually talking to me which makes it personal in a

way…instead of just looking at pictures on, let’s say Pintrest of anchor charts, she

explains to me what is expected and the steps that you can do…if you just looked at a

photo you wouldn’t understand it, but there’s an explanation to it which is very

beneficial.

Taken together, participants appeared to prefer two forms of communication during their

20-minute website navigation: visual media and audio-visual modalities. Additionally, many

participants linked their use of multimedia forms of communication to their ability to understand

and retain information. This finding is consistent with research that has examined the use of

multimedia design and memory. Specifically, recent research findings indicate multimedia

designs that incorporate audio-visual modalities promote recall and retention performances

121

(Mutlu Bayraktar & Altun, 2014). This suggests that when professional development websites

contain elements of multimedia, teachers may be more likely to create memorable learning

experiences.

b. Multitasking behaviour. Half of the participants described multitasking behaviour.

In particularly, one participant “figured that [she] could probably start typing and [she] was able

to listen to the video at the same time and start [her] document.” Similarly, another participant

suggested that when using online resources “you’re always multi-tasking.” She went on to

explain:

Even though I’m listening, I’m still looking around…I’m thinking about all the other

choices or taking in more information...I like the way [the video] can just be played at

the side, and then you can be reading about the next thing that you’re going to go to.

According to the demographic data, the participants who multitasked during their 20-

minute website navigation ranged in terms of age and years of experience. In addition,

participants who multitasked during their navigation indicated that they felt very comfortable

using the Internet and technology for professional purposes. These findings suggest that

comfort with technology may be related to website use, whereas age may not be related. This is

consistent with the third characteristic of the adult learner—the use of experience as a source of

information. As teachers’ experience with online learning environments develops their comfort

with technology also develops; thus, the way they engage with a given website changes over

time.

c. Type of online learner. Eleven participants also identified themselves as specific

types of learners. Some participants described themselves as “skimmers” where they scan the

web pages first before focusing on their objectives. For example, one participant “really liked

the transcript feature because [she] didn’t feel like [she] needed to watch the video, so it was

122

good to skim.” Another participant identified herself as “a digital native.” As a digital native,

web users “whip through things pretty quickly.”

In sum, the findings pertaining to the topic, online user behaviours can be considered in

the design of professional development websites. That is, professional development resources

for teachers should be geared towards a broad audience, to both experts and novices. In

addition, the design should include a variety of visual and multimedia web-based tools.

Moreover, professional development that incorporates informal online resources should allow

for multiple experiences with the technology over time. Taking these aspects of design into

consideration can contribute to elementary teachers’ willingness to engage as active learners and

to therefore transfer their knowledge to practice. Additionally, these considerations may

influence teachers’ continued use of high quality professional development websites.

3. Prior Experience and Professional Knowledge

The third individual difference that emerged as a result of the analysis is prior experience

and professional knowledge. Two main differences were found with respect to participants’

prior experience and professional knowledge: (i) professional challenges, and (ii) area of

expertise.

a. Professional challenges. Five participants described professional challenges they

encounter when they develop their programs. For example, one participant described the

challenge of developing and implementing a junior level reading program:

“I find it tough teaching, like developing and teaching a reading program to grades five

six…it’s easy to think that in grades five and six you’re done teaching reading, and of

course we’re not…I’m still struggling with figuring out exactly how my reading program

should look and I find that reading gets neglected in favour of writing.”

123

This example demonstrates how this participant reflects on her professional challenges as she

uses a professional development website. As a result, she is likely to seek out information that

will aid her in program planning.

b. Area of expertise. Participants’ navigational choices also appeared to be influenced

by their area of expertise with respect to curriculum content. As one participant viewed an

integrated arts lesson, she identified her qualifications: “I have a visual arts background… I’m

also a special education teacher, so this was a good amalgamation of my two interests.”

Another participant referred to the obtainment of a recent additional qualification and how this

influenced her choice to view Kindergarten material: “I thought I might check out the earlier

grades. I took a Kindergarten AQ last year and I’m really interested in full day kindergarten…so

I thought I’d take a look and see how that was.”

Taken together, these findings demonstrate that some participants used experience as a

source of information. For instance, in the former example, the participant reflected on her

experience and recognized her professional challenges. She then attempted to move beyond her

professional challenges to refine her reading program. In the latter examples, the participants

refer to their expertise and make connections between their knowledge and related instructional

methods.

In sum, individual differences appeared to affect how participants navigated The

Balanced Literacy Diet website. Specifically, the findings pertaining to theme 3 suggest that

elementary teachers demonstrate several differences as they approach and utilize online

professional development websites: elementary teachers show differences in their motivation to

use online resources; elementary teachers vary in their user behaviours and prefer specific forms

of communication for their learning; elementary teachers view themselves as a particular type of

124

online learner; and prior experience and professional knowledge appear to affect how

elementary teachers navigate professional development websites.

The significance of the findings pertaining to theme 3 is twofold: (1) the findings

emphasize elementary teachers as individual learners, and (2) the findings identify specific

differences in elementary teachers’ use of a professional development website. These findings

can be taken into consideration in the design of professional development websites—the design

should be geared towards a broad audience and various tools should be accessible.

The Central Phenomenon Resulting from the Conditions

The previous section described the first three themes of the theoretical model:

Elementary teachers’ learning experiences as they navigate a professional development website.

Specifically, perceptions, foci, and individual differences emerged as the conditions affecting

participants’ website navigation. This section presents Theme 4: Navigating a Professional

Development Website. As previously mentioned, theme 4 was deemed the core or central

phenomenon of the theoretical model because it represents the main theme of the research, all

the other categories can be related to it, and it appeared frequently and consistently in the data.

Figure 4.7 presents the topics pertaining to Theme 4: Navigating a Professional Development

Website.

Theme 4: Navigating a Professional Development Website

1. Gaining a General Sense of the Website

2. Exploring In-Depth

3. Understanding the Website

Figure 4.7. Topics pertaining to theme 4: Navigating a professional development website.

Theme 4: Navigating a professional development website

While there were differences in how the participants approached and used the website

(i.e., motivational factors, foci, area of expertise), there was a general pattern of navigation for

125

all participants as they used The Balanced Literacy Diet website. This navigational pattern

followed three stages: (1) gaining a general sense of the website, (2) exploring in-depth, and (3)

understanding the website.

1. Gaining a General Sense of the Website

As participants began their website navigation they were interested in gaining a general

sense of the website’s framework, the web pages, and the various web-based tools. The

participants considered these elements to “get a general sense of the website and what was on

it,” and to “explore the recipes, the lessons, and understand the thinking behind the website.”

During their initial navigation, participants were interested in “how the recipes were created or

the lessons were laid out.” For example, one participant described his web-based behaviour

when approaching any new website.

I think initially when you come to a new website you want to be able to play around,

move around through your anxiety, put it aside and then soon after get a sense of what’s

there and how to navigate, and how many layers there are. Then I think you can get into

it more and relax more…I was able to do that.

Participants were generally interested in gaining “more of a feel for what these recipes or

lessons were before using the recipe finder and starting to actually lose [themselves] in reading

the recipes more specifically.” As one participant viewed a classroom virtual tour, she explained

that she was “just trying to take everything in, to see how all the different subjects are integrated

in the classroom, how things are being displayed, and just the use of wall space and shelf

space.”

2. Exploring In-Depth

As participants maneuvered through the website, they began to “explore the website in a

little bit more depth” and connect and plan according to their current practice, classroom

126

context, and student needs. For example, during one participant’s think aloud she stated: “At

this point I was trying to read through the transcript. I was trying to figure out, make sure I had

all the details for that because it was an activity I thought I could possibly use.” Similarly,

another participant decided “to go in and read in a little bit more detail and read about the oral

language food group, how it’s described by the website.” In addition, a participant reflected, “In

the beginning I was more focussed on how to use it or what was available and now I’m actually

almost, like thinking about how to put it into practice.”

3. Understanding the Website

As participants continued to explore the website in depth they began to understand the

features and underlying architecture of the website. For example, as a participant viewed the

screen recording of her navigation she thought back to navigational choices and “realized that

you can actually scroll by teacher…I said, ‘oh you can actually see things by teacher,’ so you

actually might find favourite teachers that you like to follow.”

Participants also developed an understanding of the organization of the website. For

example, one participant reflected on how he was “discovering the way the website’s organized,

at the bottom of each page describing each food group. I can also use that as a filter to find

activities.” Similarly, another participant grasped the workings of particular web-based tools: “I

figured out that if you rested on the red dot you could find out what the [learning] centre was. So

I started reading more of them…there’s so much in there.” In addition, a participant described

her process of understanding how the virtual tour worked as a professional tool:

And I wasn’t too sure, I mean I’m not too familiar with virtual websites so I wasn’t too

sure what was going to happen. I thought I was just going to visit the classroom but then

I realized that as I clicked on things I got information, like what is displayed right now

127

about the chart. And then I think I went back and clicked on one of the red dots and

realized that this little video popped up which was kind of neat.

In sum, participants’ 20-minute navigations followed a pattern of three stages. Initially,

participants were interested in gaining a general sense of the website. This general overview

was followed by an in-depth exploration during which time participants were likely to relate

website content to their teaching practice. As participants explored in-depth they developed a

greater understanding of the website features and various web-based tools. This general pattern

of web seeking behaviour has implications for educational website developers. Understanding

teachers’ website navigational patterns can contribute to the development of online resources

that best support users’ online behaviours. Online learning platforms can be designed in a way

that easily permits first time users to navigate web pages as well as link back to the homepage or

central hub of the resource. By gaining a general sense of the online resource, the user can

decide to continue to explore in-depth.

Strategies for Navigating a Professional Development Website

The previous section described Theme 4: Navigating a Professional Development

Website and highlighted a pattern of three stages of a participants’ website navigation. This

section describes two strategies that participants employed while navigating a professional

development website. These strategies emerged as a result of the analysis and include: Theme

5: Evaluating Information; and Theme 6: Encoding Information for Future Retrieval. As

participants navigated The Balanced Literacy Diet website they employed these strategies to aid

them in their learning and their use of the given website. Each theme and its relevant topics are

discussed separately below.

128

Theme 5: Evaluating Information.

Evaluation involves the collection, organization, and analysis of information. In the

current study, participants evaluated a range of components during their 20-minute web

navigation. These components coincide with six criteria for evaluating informal online learning

environments as identified by Song and Lee (2014): content richness, functionality of the

technology, extent of technology integration, novelty of the technology, uniqueness of the

learning environment, and potential for learning (see Table 4.4 for a summary of Song’s

criteria). Figure 4.8 presents the topics pertaining to Theme 5: Evaluating Information.

Theme 5: Evaluating Information

1. Content Richness

2. Functionality of the Technology

3. Extent of Technology Integration

4. Novelty of the Technology and Uniqueness of the Learning Environment

5. Potential for Learning

Figure 4.8. Topics pertaining to theme 5: Evaluating information.

Table 4.4

Song and Lee’s Criteria for Evaluating Informal Online Environments

Criteria Description

1. Content richness Adequacy of content for learning purposes, credibility and currency

of the information

2. Functionality of the

technology

Ease of access and navigation

3. Extent of

technology

integration

Wide range of technologies, interactive and collaborative tool

4. Novelty of the

technology

Utilization of emerging technology for learner excitement and

motivation

5. Uniqueness of the

learning

environment

Opportunities to explore real-world issues through authentic

learning experiences

129

6. Potential for

learning

Focus on achievement of learning goals, tools for tracking learning

Note: From Song & Lee, 2014

1. Content Richness

Participants evaluated the content richness of the given website—“the adequacy of

content for learning purposes, credibility and currency of the information” (Song & Lee, 2014).

For instance, one participant stated:

It felt like it was a very comprehensive website, so it had a lot of information and also

that it had a lot of practical teaching strategies, so really useful things like seeing a real

classroom and hearing that teacher speak to her classroom, I found that really useful.

The quality of the content contributed to the participants’ learning experience by

ensuring the trustworthiness of the information presented on the website. Zaharias (2009)

emphasizes that the basis of an online learning environment “should be the quality of content”

(47). Content should not only be relevant, but it should also come from reliable and cited

sources (Zaharias, 2009).

2. Functionality of the Technology

Participants also evaluated the functionality of the technology—“the ease of access and

navigation” (Song & Lee, 2014). For instance, a participant described his “general

feelings…that [the website] was a relatively clean and well-organized…there were lots of

headings and boxes and colours and fonts but it wasn’t overwhelming…there was a consistent

colour scheme which I think is very helpful.” The website was also described as user-friendly.

For example, one participant reflected on the visuals and accessibility of information: “I liked

how visual it was, it was really user-friendly in the sense that all the videos played when you

clicked them. You could expand the size, you could change the volume…I thought that was

really accessible.”

130

Half of the participants also evaluated the website’s organization—how the website’s

underlying architecture contributed to the ease of use. During the think aloud, participants

described “how everything is searchable in so many different ways…by either the component of

the diet or by your grade or by video.” For example, one participant described how the ease of

navigation contributed to the access of information and potential choices for learning.

I found that the general website was attractive, it was pretty easy to navigate the tabs at

the top, and in the recipe part there were so many choices that you could go and look so

that was, I found that nice to have all those different choices to look at.

The ease of use was facilitated by the organization of the website which allowed

participants to connect their practice to relevant information in an efficient way.

3. Extent of Technology Integration

Participants evaluated the extent of technology integration. That is, the website included

a wide range of interactive technologies, including the videos which were found to be “really

helpful.” Participants could “hear how [the teacher] questions and frames the lessons and gets

the students thinking.”

Participants also commented on specific web-based features. For example, one

participant thought “it was interesting to see that there are a lot of personal views of teaching

videos” alongside the virtual tours. She continued to explain why:

I think that that is really important, that [the personal views of teaching videos] were

side-by-side with the virtual tour…I think that it certainly is the easiest to learn from

someone where there’s a similarity of style and lessons out of context don’t always

transfer very well because when the other components aren’t built into it then it won’t

work as well.

131

Another participant described how a number of features contributed to her planning and

assessment:

I feel like [the website] has everything to make it helpful for me…it has lots of pictures and

when I click on the lessons, you can print out the lesson plan, which is amazing. It breaks

down the components of the food groups and how they help. I really especially like the

assessment component which is something I’m always grappling with so it helps as well.

4. Novelty of the Technology and Uniqueness of the Learning Environment

Participants evaluated the novelty of the technology and described how the virtual

classroom tours utilized “emerging technology” and that this contributed to their motivation to

use online resources (Song & Lee, 2014, p. 517). The virtual classroom tours also contributed

to the uniqueness of the learning environment. Participants had “opportunities to explore real-

world issues through authentic learning experiences” (Song & Lee, 2014, p. 517). For instance,

while one participant reflected on her classroom virtual tour she stated: “The most appealing

thing once I recognized it was the virtual tours and the videos I would say, actually hearing the

teachers describe whatever they do and being able to see them describe it in their classroom.”

5. Potential for Learning

Participants also envisioned potential for learning by evaluating the content with respect

to their own understanding. For instance, one participant stated:

I like the way this website broke everything down. It had a well-structured lesson plan

with exactly what the students are expected to do and what we need to do in order to

facilitate the activity.

In sum, participants evaluated many aspects of the given website—participants formed

opinions about the web-based features, the organization of the website, the ease of accessing

information, and they assessed the quality and authenticity of information, as well as the

132

potential for learning. Evaluating web-based features such as these may contribute to a website

user’s ability to plan during their learning. That is, collecting, organizing, and analyzing

information can facilitate the decision-making process as teachers use a professional

development website. In addition, gaining feedback from the real user can provide meaningful

information about what works and necessary improvements.

Theme 6: Encoding Information for Future Retrieval

Encoding information for future retrieval was the second strategy participants employed

while navigating The Balanced Literacy Diet website. Participants synthesized new information

by utilizing external aids: they took notes during their navigation and also referred to

bookmarking and emailing as strategies for retaining newly learned information. Figure 4.9

presents the topics pertaining to the sixth theme.

Theme 6: Encoding Information for Future Retrieval

1. Note-Taking

2. Saving Ideas for Future Use

Figure 4.9. Topics pertaining to theme 6: Encoding information for future retrieval.

1. Note-Taking

More than half of the participants referred to note-taking as a strategy for encoding

information. Participants who took notes believed “that note taking is always a good active way

to interact with information.” One participant stated, “I like to take notes to remind myself”

while another participant explained, “if I see something that interests me I would be jotting

down little notes to remind myself…key concepts and how it was laid out so I can copy that at

some point.” Similarly, another participant stated that she “would usually take notes on the

side…jot the ideas down for [herself] to make sure [she] can revisit [the information].”

133

Participants referred to note-taking as a general strategy for encoding information while

navigating any online professional resource. For example, the following quote describes one

participant’s thought process about note taking:

The teacher [referring to a teacher in a video] seemed to have a nice comprehensive program

and she gave some nice tips off-hand that I actually wanted to write down …I do end up

opening a window to start typing down some ideas because I would want to go back to these

quickly, refer back to these quickly the next time I wanted to go through this video…or the

next time I wanted to think about some of the ideas from this so I’d have a summary for

myself.

2. Saving Ideas for Future Use

Four participants also referred to bookmarking and emailing as strategies for encoding

information while using online professional resources. For instance, one participant stated, “If

this was my computer I would have bookmarked these…just so that I could go back to them.”

Similarly, another participant “thought about emailing to [herself] because that’s usually a good

way to remind yourself of good things that you’ve done recently.”

In sum, participants in the current study encoded information during their 20-minute

website navigation by utilizing external aids, including note-taking, bookmarking, and emailing.

Many participants made references to the contribution of these external aids: for instance, the

retention and future retrieval of newly acquired information. These types of external aids can be

thought of as learning strategies and were employed by the participants (the learners) to

accomplish a specific task (to seek out information related to their teaching practice).

Furthermore, participants used these external aids in a deliberate way: participants intentionally

chose to take notes and made explicit references to the benefits of this strategy. This is

consistent with studies that report adults using deliberate strategies to enhance memory (Intons-

134

Peterson & Fournier, 1986; Schryer & Ross, 2013). Note taking, in particular has been found to

facilitate memory even when the notes are not available as retrieval cues (Intons-Peterson &

Fournier, 1986). That is, the process of taking notes in itself has a positive effect on the way in

which newly learned material is encoded. In the current study, it is possible that participants

who took notes during their website navigation may have been more likely to encode, recall, and

ultimately implement the newly learned material into their teaching practice. In addition, the

findings pertaining to theme 6 suggest that as teachers use informal online learning

environments they do so with the intention to integrate their newly formed knowledge into their

classroom practice.

Potential Outcomes Resulting from Elementary Teachers’ Website Navigation

Three themes related to the potential outcomes of the theoretical model resulted from

elementary teachers’ website navigation and emerged from the analysis, and include: Theme 7:

Reflecting on Learning; Theme 8: Continued Professional Learning; and Theme 9: Intentions

for Practice. Each theme and its relevant topics are discussed separately below.

Theme 7: Reflecting on Learning

During participants’ think aloud they reflected on their learning and their use of the

given website. Reflective statements in this study were classified through the data analysis into

two categories: (1) general feelings about the website, and (2) future use of the website. Figure

4.10 presents the topics pertaining to the seventh theme.

Theme 7: Reflecting on Learning

1. General Feelings about the Website

2. Future Use of the Website

Figure 4.10. Topics pertaining to theme 7: Reflecting on learning.

135

1. General Feelings about the Website

During their think aloud, all of the participants reflected upon their general feelings

towards The Balanced Literacy Diet website. For example, one participant perceived “almost

an overwhelming amount of information at times. It felt like [she] could explore it for hours.”

Another participant echoed this thought and also described her sense of being overwhelmed: “It

was a bit overwhelming to see the number of things that go on in the room and there are several

that I would have liked to go back and look at again.” While some participants expressed a

sense of being overwhelmed, others reflected on their sense of comfort as they used the given

website and that their comfort with the technology increased over the course of the 20-minute

task. For example, while viewing the screen recording of her navigation, one participant stated:

I’m also at this point having less panic about, you know, there’s a lot in here. Now I’m

getting a sense of what is in here…and that each component definitely has a lot, but it’s

at this point that I’m sort of going, ‘okay now I know what’s here’…as opposed to that

initial, ‘what is this and how much is in here?’

2. Future Use of the Website

In addition to their general feelings towards the website, participants also reflected on

their future use of the given website. For example, one participant reflected on features of the

website she thought she would explore during her next visit. She “probably would look at some

of the other teachers’ videos because every teacher is unique and they’re probably highlighting

different parts of their program which is pretty cool.” Similarly, another participant stated:

If I had to go back to the website I think I would have watched more of the videos, like

there was a tab that I didn’t explore, and I think that was next to the virtual tours,

something about the recipes in action…I think I would have had a look at that.

136

In sum, participants in the current study reflected on their general feelings about the

website. For some participants, this was an initial sense of feeling overwhelmed while other

participants gained a sense of comfort with the technology and ease of accessing web-based

features and tools. Participants also noted certain features and web pages that they would use in

the future. Taken together, participants’ self-reflections about their learning experience align

with Shulman and Shulman’s model of teacher learning. As previously stated, when teachers

are more conscious of their own understandings, performances, and dispositions, their capacity

to learn is enhanced (Shulman & Shulman, 2004). In the current study, participants’ reflections

about their use of and general feelings towards the website, and their performance during their

website navigation may contribute to their future use of the website—how they will narrow

down their focus to gain personally relevant pedagogical information about literacy instruction.

Theme 8: Continued Professional Learning

As teachers move forward in their career they build a repertoire of knowledge related to

pedagogy and curriculum. Their knowledge is constructed through a variety of experiences and

professional learning opportunities. In the current study, continued professional learning was

classified through the data analysis into four categories: (1) advantages of online professional

learning, (2) increased self-efficacy, (3) feeling inspired, and (4) knowledge construction.

Figure 4.11 presents the topics pertaining to the Theme 8.

Theme 8: Continued Professional Learning

1. Advantages of Online Professional Learning

2. Increased Self-Efficacy

3. Feeling Inspired

4. Knowledge Construction

Figure 4.11. Topics pertaining to theme 8: Continued professional learning.

137

1. Advantages of Online Professional Learning

All of the participants referred to the advantages of online learning, in contrast to face-

to-face workshops, packaged texts, and teacher guides. As stated by one participant: “online

resources are kind of like a one-stop shop for me because I don’t need to go anywhere

else…they offer a range of ideas for teachers.” Another participant noted the accessibility of

online resources:

[Online resources] allow me to find new ways of teaching something or new activities to

reinforce student learning that I wouldn't otherwise have access to. It is also great to see

what other teachers are doing and pick what I like and can use in my own classroom.

Practicing teachers are often under time constraints and they “don’t have that time to walk

around and to visit people’s classrooms.” When online resources and technology are easily

accessible and include real-life examples, “it makes you feel like you’re really experiencing the

lesson.”

This finding is consistent with the current study’s factors that motivate teachers to use

online resources. As previously discussed, participants’ use of online resources is largely due to

increasingly available and convenient online resources. As a potential outcome for the current

study’s model, the advantages of online resources for continued professional learning may

contribute to elementary teachers’ ongoing use of professional development websites. This

finding is also consistent with self-directedness in adult learning. Online resources are

advantageous in that they foster a self-initiated process of learning, personal autonomy, and

greater sense of learner control.

2. Increased Self-Efficacy

Teacher efficacy can be described as “teachers’ beliefs about their own capacities as

teachers” (Tschannen-Moran, Woolfolk Hoy, & Hoy, 1998). Factors that impact a teacher’s

138

efficacy include performance accomplishments, vicarious experiences, and social persuasion

(Protheroe, 2008). In the current study, vicarious experiences appeared to contribute to

participants’ increased self-efficacy. Vicarious experiences occurred as participants observed a

model teacher using an effective method of instruction. As a result, participants referred to

gains in their confidence—participants believed that they too could successfully implement a

lesson or literacy-related strategy (Protheroe, 2008). Specifically, 12 participants shared

positive reflections about their own ability to successfully employ similar instructional strategies

after they viewed videos of model teachers implementing an effective strategy. For example,

one participant stated: “I really did like exploring that teacher’s classroom…I found what she

was saying from the get-go seemed to resonate with my understanding of a literacy program.”

Similarly, another participant referred to the virtual tours as an approach to overcome feeling

isolated:

Sometimes when you are teaching, I mean I find as a teacher, once you’re actually

teaching you don’t get to spend that much time in other classrooms. This is something

that most teachers will say—that it can be kind of isolating…I know that when I see

other classrooms it’s kind of neat to go, ‘oh this teacher has an anchor chart about parts

of speech on the wall’, and I guess especially as a new teacher it can kind of affirm, oh

yeah, it does make sense that I’m doing that.

In addition, a participant described the positive impact of seeing strategies reinforced by

model teachers: “I think it’s nice to see things that you’re doing reinforced in what other people

are doing. It just keeps your confidence going…okay, I’m on the right track I’m doing the right

thing…you’re always questioning yourself.”

In sum, participants’ sense of self-efficacy appeared to be related to the success of a

model teacher they viewed in a video or virtual classroom. This finding suggests that teachers’

139

informal online learning experiences are enhanced by videos of model teachers demonstrating

effective instructional practices. These videos appear to have a positive impact on (and can

potentially change) a teacher’s sense of self-efficacy. Moreover, participants described a sense

of affirmation—a confirmation of their instructional decisions. Research that has investigated

teacher efficacy has found a positive relationship between higher levels of teacher efficacy and

classroom quality and literacy instruction (Guo, Piasta, Justice, & Kaderavek, 2010; Tschannen-

Moran & Johnson, 2011). Thus, by viewing videos of effective classroom instruction during

elementary teachers’ informal online learning experiences, teacher efficacy may increase and

contribute to the quality of classroom practice.

3. Feeling Inspired

More than half of the participants were inspired as they navigated the given website.

The Balanced Literacy Diet website “can show you things that you haven’t thought of and give

you inspiration and new ideas.” For example, one participant stated that “it’s good to get

inspired.” She went on to explain how the use of online resources for professional learning

allows her to reflect and ask:

Is there something I can form or reform or clarify? Or something that I’ve done in the

past that just has not worked at all that I’m interested in revising, like the social studies

for grade two that we’re currently mired in?

Professional development websites allow elementary teachers to look “for things that

you have seen and you want to build on and then things that you’ve never seen before.” One

participant, who identified herself as “a relatively new teacher,” recognized the impact of

discovering new ideas and “refreshing my knowledge of current practices that I can implement

in my own classroom to enhance my program and help me to become a better teacher.”

140

In the current study, the process of acquiring information via a professional development

website appeared to stimulate participants’ thinking about their individual practice—how they

could enhance their program and instructional methods. For novice teachers, the use of a

professional development website may elicit fresh ideas or clarify particular teaching

approaches. For the experienced teacher, professional development websites may reactivate

ideas and re-energize their willingness to implement new instructional strategies.

4. Knowledge Construction

As participants navigated the given website they constructed knowledge about literacy

instruction and literacy teaching strategies. For example, one participant, who “watched a video

in detail,” was “just trying to figure out how to present the concept of success criteria to the

students and how to write it out in a clear way.” Consistent with Shulman and Shulman’s model

of teacher learning, this participant began to understand the content from the website in relation

to her own practice. In addition, a participant referred to the “idea of colour coding” as an

accommodation for students with learning disabilities: “Some students I think would find it

helpful when we’re writing paragraphs or any sort of writing that can be kind of

compartmentalized to learn the form.” This participant was making sense of the information on

the website by building onto her current knowledge about classroom accommodations.

As participants navigated The Balanced Literacy Diet website, they formed concepts and

constructed ideas in relation to their individual practice. That is, they verbalized how they

would use the information presented on the website in their practice, an important element of

professional learning.

Theme 9: Intentions for Practice.

During participants’ think aloud, all of the participants shared their intentions to

incorporate newly acquired information into their teaching practice. Theme 9: Intentions for

141

Practice was classified through the data analysis into two categories: (1) adapting information,

and (2) professional goals. Figure 4.12 presents the topics pertaining to the ninth theme.

Theme 9: Intentions for Practice

1. Adapting Information

2. Professional Goals

Figure 4.12. Topics pertaining to theme 9: Intentions for practice.

1. Adapting Information

Participants shared their intentions to adapt information to their student needs. They

discussed how they would “tweak” lesson examples from the website. For instance, one

participant described how she would adapt a writing idea to her drama class: “That was

interesting, ‘All about Me Autobiographies,’ because I was starting to think that about how it

could translate into a drama activity fairly quickly.” Similarly, as participants navigated the

given website they realized “that you could combine a number of these ideas and make them

your own, see what works for you.” For example, one participant explained how a primary

level activity could be adapted to the junior level:

I’m thinking about the idea of having students have their individual cubby

holders…what would I do because I’m working with junior students, is maybe give them

a binder that they can work with…I look at these [lessons] and just know what I can do

and I have an idea of how to set things up.

More than half of the participants looked beyond the given information on The Balanced

Literacy Diet website—they extended ideas to meet their individual needs. One participant

stated, “I very rarely use anything verbatim that I find on the Internet. I usually adapt it to

something I’ve already done or I fold it into something that I’m doing.” Similarly, another

participant described an integrated activity that stemmed from a writing lesson:

142

I’m looking at text structures here because thinking about drama, there’s a lot you can do

teaching text structures through dramatic improvisation or games or exploration, and so I

decided to have a quick look and see how it’s explained.

In sum, participants in the current study made explicit references to how they would

adapt information to their own practice. This finding is consistent with Shulman and Shulman’s

model of teacher learning in that elementary teachers have a clear vision about their goals and

adapt information according to their student needs.

2. Professional Goals

Participants also described curriculum areas and aspects of their practice they believed

they could improve upon—areas they believed were necessary “to continue working on.” For

instance, one participant described how she “chose the image of the writing centre because

that’s an area that [she] would like to do better.” Similarly, another participant stated, “one area

that I’m still a little bit nervous with is doing writing, so I like to get other people’s ideas and

how they approach strategies for writing.”

Participants also referred to school and board-wide initiatives. For example, one

participant stressed how “right now there’s a lot of emphasis on anchor charts and it’s

something that I know I need more experience on, what it looks like and how I can do to model

it.” Similarly, another participant described that “at the first school that I worked at we were

involved in moderated marking throughout the whole school, from grade one to eight, with the

writing continuums.” This participant was interested in viewing “writing assessment

tools…because [she] always finds [writing assessments] tricky.”

In sum, the potential outcomes as a result of navigating The Balanced Literacy Diet

website are the “changes in cognition or behaviour resulting from engagement in activity”

(Hoekstra et al., 2009, p. 665). The primary goal of any resource is to contribute meaningful

143

information to professional practice. The findings pertaining to theme 9 are consistent with

Bakkenes and her colleagues’ findings (2010)—teachers try new practices in the context of

educational innovation and change.

General Discussion

The aim of study 2 was to utilize the virtual revisit think aloud method and two

additional data sources to examine elementary teachers’ learning experiences as they navigated

a multimedia professional development website. The findings provide a comprehensive picture

of the participants’ use of a professional development website; thorough descriptions and

explanations related to higher cognitive processes provide insights into four aspects of the

participants’ online learning experience: (1) the conditions that affect elementary teachers’ use

of a professional development website, (2) elementary teachers’ general pattern of website

navigation, (3) the navigational strategies elementary teachers employ when using a professional

development website, and (4) the potential outcomes of utilizing a professional development

website. These aspects were organized into the theoretical model for elementary teachers’

learning processes as they navigated the given website. This model offers contributions to the

literature, both for understanding teacher cognition and learning, and for informal online

professional development. Moreover, this model was developed utilizing the virtual revisit

think aloud, a method that tracked participants’ cognitive processes as they used and learned

from The Balanced Literacy Diet website; thus, this model also offers support for the virtual

revisit as an effective think aloud methodology in the context of online learning.

The general discussion section of this chapter continues with an overview of the major

findings. Next, this section presents the study limitations. Then, this section discusses the

significance of this research and possible future directions.

144

Overview of Major Findings

Based on study 2’s model for elementary teachers’ learning experience as they navigated

a professional development website, 14 major findings are summarized below. These include

conditions that affect elementary teacher’s use of a professional development website (key

conditions), strategies that elementary teachers employ while navigating a professional

development website (key navigational strategies), and potential outcomes of elementary

teachers’ use of professional development websites (key outcomes). While the term, elementary

teachers is used, the major findings are based on 15 participants from the current study.

Key Conditions:

1. Elementary teachers who utilize the Internet for their professional learning want to ensure

that professional online resources are credible and trustworthy. Specifically, elementary

teachers trust online resources that they perceive as authentic; teachers want to ensure that

the website is supported by credible organizations; and teachers value a professional

resource when it is recommended by a credible source.

2. Elementary teachers hold positive beliefs about research-based practices including best

practices in literacy instruction. In their quest for professional knowledge and instructional

strategies, elementary teachers seek information according to these beliefs.

3. Elementary teachers focus on their student needs and instructional goals as they seek

information related to their professional practice. Specifically, elementary teachers are

cognizant of their students’ needs, unique teaching context, and current and future

instructional goals.

4. Elementary teachers are motivated to use online resources for their professional learning

because these resources are convenient and accessible, and they remove time, location, and

145

situational barriers. In addition, elementary teachers are intrinsically motivated to expand

their repertoire of professional knowledge via informal online learning environments.

5. Elementary teachers vary in their use of online professional development websites:

elementary teachers have preferences for audio and visual website tools and they sometimes

multitask as they seek information related to their practice. In addition, elementary teachers

view themselves as a specific type of online learner.

6. Elementary teachers’ prior experience and professional knowledge, including the area of

their expertise, influence their navigational choices.

Key Navigational Strategies:

7. As elementary teachers navigate professional development websites they evaluate website

features and content. Elementary teachers want to ensure the applicability of the

information they are seeking to their teaching context.

8. Elementary teachers who access online resources employ encoding strategies as they use

professional development websites. Specifically, external aids (note taking, emailing, and

bookmarking) may facilitate the retention and recollection of newly learned material.

Key Potential Outcomes:

9. Elementary teachers are self-reflective during online learning. As elementary teachers use

professional development websites they reflect on their feelings towards the resource, its

content and organization, and whether they would invest in the resource over time.

10. Elementary teachers value informal online environments as a source of continued

professional development and see several advantages to this facet of learning (e.g.,

accessibility).

11. Utilizing online resources as a form of continued professional development contributes to an

increase in elementary teachers’ self-efficacy. When observing model teachers

146

implementing an effective strategy in a video or classroom virtual tour, viewers gain

confidence in their own ability to successfully employ similar strategies.

12. Utilizing professional development websites inspires elementary teachers to implement best

practices in their classroom.

13. Professional development websites contribute to elementary teachers’ knowledge

construction by providing opportunities to build upon previously acquired knowledge.

14. Elementary teachers adapt professional development to their instructional needs and

consider their professional goals when navigating online resources.

Study 2 Limitations

There are several limitations to this study that need to be considered when interpreting

the findings. The first limitation of study 2 is the spread of data collection over time. Data

collection occurred over a period of eight months and this period may have skewed the findings.

That is, the participants may have varied in their use of the website depending on the time of

year (e.g., how one teacher plans and learns at the beginning of the school year may look quite

different than how a teacher plans and learns midyear). Due to the low response rate, the eight

month period was necessary to collect data from all participants; however a shorter period of

data collection may increase the validity of the findings.

The second limitation of study 2 relates to the low response rate and the difficulty

recruiting elementary teachers to volunteer to participate during the school year. Practicing

teachers are extremely busy, and although multiple sources of participant recruitment were

employed, the response rate was low. Studies with a larger sample size that can utilize

alternative modes of recruitment, such as a greater monetary incentive to participate, will be

useful to verify the findings of this study.

147

The participants who did participate ranged in terms of their years of teaching

experience. While a range of experience contributes findings from a broader population,

experience may also influence how a teacher approaches professional development. Future

research could examine the differences in online learning experiences between expert and

novice teachers as well as between practicing and preservice teachers.

Although strategies for establishing trustworthiness were taken, the ability to generalize

qualitative findings is limited. While generalizing the findings is limited, the small sample size

does generate data that is rich and thorough, with descriptions that are thick with details about

the context, participants, and direct quotes. Therefore, the findings can guide researchers and

future work in this field of study.

A further limitation relates to reactivity which may have occurred as a result of

participants’ awareness of completing the website task and think aloud. As described in study 1,

reactivity was reduced as much as possible during the one-on-one meeting. However, reactivity

may have contributed to the verbalization of specific words related to Theme 7: Reflecting on

Learning. Some participants expressed feeling overwhelmed, stressed, anxious, and panicked in

relation to their initial use of the website. While these verbalizations were interpreted as

reflections on learning, reactivity is noted as a factor that may have influenced participants’

word choice.

Finally, it is difficult to completely eliminate researcher bias. Steps were taken to reduce

researcher bias as much as possible: staying close to the data during analysis, the application of

a well-established research approach (grounded theory), and strategies for establishing the

credibility of the findings were taken (i.e., the member check). However, this limitation must be

considered when interpreting the findings. To avoid researcher bias, future research could

involve a team of researchers with varying backgrounds, particularly during data analysis.

148

Significance and Future Directions

The significance of this study is threefold: (1) this research describes a professional

learning experience in which elementary teachers engage on a regular basis; (2) this research

identifies key conditions, strategies, and potential outcomes of elementary teachers’ use of a

professional development website; and (3) finally this research utilizes the virtual revisit think

aloud method to gain insights into the learning experiences of elementary teachers’ as they use a

multimedia professional development website.

First, this study describes how elementary teachers’ use and learn from a professional

development website—an experience in which elementary teachers often engage. Teacher

professional development is complex and multifaceted—there are multiple contexts in which

teachers’ gain professional knowledge and instructional materials. As previously mentioned,

elementary teachers are increasingly turning to the Internet for their professional learning;

therefore, it is imperative to study teacher thinking and learning in the context of informal online

learning environments. The study of teachers’ thinking in online environments can contribute to

research in teacher cognition—what teachers know, think, believe, and value. While research in

the area of teacher cognition is very extensive, research that explores elementary teachers’

thinking during informal online professional learning is limited. Thus, the findings from this

study provide a representation of teachers’ learning experiences in one domain of teacher

professional development. In addition, this representation can benefit educational website

designers and policy makers by helping them anticipate and meet the needs of teachers in their

search for high-quality professional resources.

Secondly, this research identifies key conditions, navigational strategies, and potential

outcomes related to elementary teachers’ use of a professional development website. Research

in the field of adult learning is vast and, as discussed in the introduction, models of the adult

149

learner as well as theories about how adults learn can support the design and implementation of

professional development across many domains. The findings of study 2 portray an in-depth

representation of teacher learning within informal online environments. How elementary

teachers use and learn from informal professional development websites is a relatively under-

examined area of adult learning and teacher professional development. However, identifying

specific aspects of teachers’ informal online learning experiences can contribute to a holistic

understanding of teacher professional development; that teacher learning is complex and

multifaceted.

Finally, this study utilizes the virtual revisit think aloud method to explore elementary

teachers’ informal online learning experiences. As previously discussed, the virtual revisit think

aloud provides direct data about the reasoning process: combining the retrospective think aloud

and screen capture recording allows participants to reason about their decisions; they describe

their choices and why they made them. The findings related to the virtual revisit further

substantiate study 1 findings: the virtual revisit appears to be an effective approach for exploring

the use of Internet-based resources by elementary teachers. In study 2, participants were able to

access their interactive decision making immediately after viewing a recording of their

navigational experience. They viewed details of the website and offered reasons for their web-

based behavoiurs—participants clarified their choices, explained their decisions, and provided

details about their navigation. Access to participants’ cognitive processes was possible via the

virtual revisit think aloud, and contributed to a theoretical model for elementary teachers’

learning experiences as they navigated a professional development literacy website.

The virtual revisit allowed participants to focus on what they were doing at a particular

time while freeing them from having to complete the task and simultaneously talk about it.

Furthermore, the limitations of the concurrent and retrospective techniques may have been

150

avoided with the virtual revisit think aloud method. The virtual revisit think aloud allowed

participants to recall their decisions and why they made them. Future research that investigates

the virtual revisit think aloud methodology for examining teacher cognition within a variety of

informal and formal online environments will contribute to the validity of this methodology. In

addition, the virtual revisit think aloud could be applied across a range of age groups and

domains in education. If this methodology can provide access into elementary teachers’

thinking as they utilize a professional development website, the application of the virtual revisit

to any online learning experience is possible.

151

CHAPTER 5

General Conclusions

The aim of this dissertation was twofold: (1) to compare three think aloud methods—

concurrent, retrospective and virtual revisit—for examining elementary teachers’ cognitive

processes as they use a multimedia professional development website; and (2) to gain an in-

depth understanding of the teachers’ learning experiences from the virtual revisit condition as

they navigate a professional development website. The findings of study 1 suggest that the there

are benefits and limitations to each type of think aloud method. In addition, findings of study 1

indicate that the virtual revisit think aloud methodology can overcome the limitations of the

concurrent and retrospective think alouds. In contrast to the concurrent think aloud, the virtual

revisit method produces rich and thorough verbalizations related to teacher planning, reasoning,

and reflection. In contrast to the retrospective think aloud, the virtual revisit method produces

details concerning the entire task—little to no information is omitted, as in the retrospective

procedure. Thus, in addressing the research questions related to study 1, this exploratory study

suggests that there are differences in verbalizations between the concurrent and virtual revisit

think aloud as well as between the concurrent and retrospective think aloud methods. In

addition, differences in the think aloud task and comfort level with thinking aloud across the

think aloud methods.

Given the findings of study 1, data from the virtual revisit condition was further

examined in a second study using a grounded theory approach to analysis. The findings of

study 2 provided a theoretical model of elementary teachers’ learning experiences as they used a

multimedia professional development website. The nine themes that emerged as a result of the

analysis present three conditions that affect elementary teachers’ use of a professional

development website, the general pattern of website navigation, two navigational strategies

elementary teachers employ when using a professional development website, and three potential

152

outcomes of utilizing a professional development website. As discussed in the previous chapter,

this model offers contributions to the literature, both for understanding teacher cognition and

learning, and for informal online professional development.

The overall findings of this research suggest that elementary teachers actively construct

pedagogical knowledge and instructional teaching methods as they use online learning

environments. To capture their thought processes as accurately as possible, a methodology like

the virtual revisit think aloud must be employed. As elementary teachers increasingly turn to

the Internet for their professional learning, it is essential to examine how teachers use

professional websites and web-based technologies. By utilizing the virtual revisit method,

researchers can gain direct insights into elementary teachers’ cognitive processes and learning

experiences (conditions, navigational strategies, potential outcomes) as they use and learn from

professional development websites. This information can contribute to the ongoing

development of high quality online learning environments. If researchers provide feedback to

educational website developers based on the methodology used in the present study,

improvements can be made to the design and content of their sites. Similarly, findings may

benefit policy makers and administrators of professional development by providing insights

concerning elementary teachers’ use of informal online resources as an aspect of their

professional development.

153

References

Agostinho, S. (2005). Naturalistic inquiry in e-learning research. International Journal of

Qualitative Methods, 4(1), 1-13.

Anderson, J. R. (1995). Cognitive Psychology and Its Implications. (4th edition). New York: W.

H. Freeman and Company.

Andreassen, R., & Braten, I. (2013). Teachers’ source evaluation self-efficacy predicts their use

of relevant source features when evaluating the trustworthiness of web sources on

special education. British Journal of Educational Technology, 44(5), 821-836.

Aranyi, G., van Schaik, P., & Barker, P. (2012). Using think-aloud and psychometrics to explore

users’ experience with a news website. Interacting with Computers, 24, 69-77.

Armstrong, D. (2011). Students’ perceptions of online learning and instructional tools: A

qualitative study of undergraduate students’ use of online tools. The Turkish Online

Journal of Educational Technology, 10(3), 222-226.

Bakkenes, I., Vermunt, J.D., Wubbels, T. (2010). Teacher learning in the context of educational

innovation: Learning activities and learning outcomes of experienced teachers. Learning

and Instruction 20, 533-548.

Barak, M. (2010). Motivating self-regulated learning in technology education. International

Journal of Technology and Design Education, 20(4), 381-401.

Barzilai, S. & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online

sources. Cognition and Instruction, 30(1), 39-85.

Boren, M.T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE

Transactions on Professional Communication, 43(3), 261-278.

Borg, S. (2006). Teacher cognition and language education: Research and practice. London:

Continuum.

154

Borko, H. (2004). Professional development and teacher learning: Mapping the Terrain.

Educational Researcher, 33 (8), 3-15.

Borko, H., Livingston, C., & Shavelson, R.J. (1990). Teachers’ thinking about instruction.

Remedial and Special Education, 11(6), 40-49.

Branch, J. (2006). Using think alouds, think afters, and think togethers to research adolescents’

inquiry experiences. Alberta Journal of Educational Research, 52(3), 148-159.

Brockett, R.G., & Hiemstra, R.H. (1985). Bridging the theory-practice gap in self-directed

learning. New Directions for Continuing Education, 25, 31-40.

Callanan, M., Cervantes, C., & Loomis, M. (2011). Informal learning. Wiley Interdisciplinary

Review: Cognitive Science, 2(6), 646-655.

Caffarella, R. (1993). Self-directed learning. New Directions for Adult and Continuing

Education 57, 25-35.

Callahan, C., Saye, J., & Brush, T. (2013). Designing web-based educative curriculum

materials for the social studies. Contemporary Issues in Technology and Teacher

Education, 13(2), 126-155.

Candy, P. (1988). Key issues for research in self-directed learning. Studies in Continuing

Education, 10(2), 104-124.

Carcary, M (2009). The research audit trail: Enhancing trustworthiness in qualitative inquiry.

Electronic Journal of Business Research Methods, 7(1), 11-24.

Cercone, K. (2008). Characteristics of adult learners with implications for online learning

design. Association for the Advancement of Computing in Education Journal, 16(2),

137-159.

Cervetti, G.N., Kulikowich, J., Drummond, M., & Billman, A. (2012). Influence of educative

155

curriculum materials on teachers’ learning and teaching. Paper presented at the annual

meeting of the American Educational Research Association, Vancouver, B.B. Retrieved

August 8, 2014 from

http://www.sandi.net/cms/lib/CA01001235/Centricity/Domain/121/R_R/DistResearch/R

esultReports/585_Cervetti_G_et_al_UCBerkeley_2012B.pdf

Charalambousa, K., & Ioannou, I. (2011). The attitudes and opinions of Cypriot primary

teachers about the use of the Internet for their professional development and as an

educational tool. Learning, Media and Technology, 33(1), 45-57.

Chametzky, B. (2014). Andragogy and engagement in online learning: Tenets and solutions.

Creative Education, 5, 813-821.

Charmaz, K. (2014). Constructing grounded theory: A practical guide through qualitative

Analysis (2nd

ed.). Thousand Oaks, CA: SAGE Publications.

Clark, C.M., & Peterson, P.L. (1986). Teachers' thought processes. In M.C. Wittrock, (Ed.),

Handbook Of Research Teaching (3rd

ed.) (pp.255-296). New York: MacMillan

Publishing Company.

Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usability test method: A

technical communication approach. IEEE Transactions on Professional Communication,

53(3), 202-215.

Corbin, J.M., & Strauss, A.L. (2008). Basics of qualitative research: Techniques and

procedures for developing grounded theory (3rd

ed.). Thousand Oaks, CA: SAGE

Publications.

Creswell, J.W., & Piano Clark, V.L. (2007). Designing and conducting mixed methods research.

Thousand Oaks: Sage Publication.

Cunningham, A.E., Zibulsky, J., Stanovich, K.E., & Stanovich, P.J. (2009). How teachers would

156

spend their time teaching language arts. Journal of Learning Disabilities, 42(5), 418-

430.

Damico, J., & Balidon, M. (2007). Examining ways readers engage with websites during think-

aloud sessions. Journal of Adolescent and Adult Literacy, 51(3), 254-263.

Darling-Hammond, L., & Richardson, N. (2009). Teacher learning: What matters? Educational

Leadership, 46-53.

Davis, E.A., & Krajcik, J.S. (2005). Designing educative curriculum materials to promote

teacher learning. Educational Researcher, 34(3), 3-14.

Davis, E.A., Palincsar, A.S., Arias, A.M., Bismack, A.S., Marulis, L.M., & Iwashyna, S.K.

(2014). Designing educative curriculum materials: A theoretically and empirically driven

process. Harvard Educational Review, 84(1), 24-52.

Deci, E. L., & Ryan, R. M. (2008). Self-determination theory: A macrotheory of human

motivation, development, and health. Canadian Psychology, 49(3), 182-185.

Desimone, L. M. (2009). Improving impact studies of teachers’ professional development:

Toward better conceptualizations and measures. Educational Research, 38(3), 181-199.

de Vries, S., van de Grift, W., & Jansen, E. (2013). Teachers’ beliefs and continuing

professional development. Journal of Educational Administration, 51(2), 213-231.

de Vries, S., van de Grift, W., & Jansen, E. (2014). How teachers’ beliefs about learning and

teaching related to their continuing professional development. Teachers and Teaching:

Theory and Practice, 20(3), 338-357.

Dillon, A. (2001). Beyond usability: process, outcome and affect in human-computer

interactions. Canadian Journal of Library and Information Science, 26(4), 57-69.

Downer, J.T., Pianta, R.C., Fan, X., Hamre, B.K., Mashburn, A., & Justice, L. (2012). Effecits

157

of web-mediated teacher professional development on the language and literacy skills of

children enrolled in prekindergarten programs. NHSA Dialog, 14(4), 189-212.

Duncan-Howell, J. (2010). Teachers making connections: Online communities as a source of

professional learning. British Journal of Educational Technology, 41(2), 324-340.

Ebbinghaus, H. (1964). Memory: A Contribution to Experimental Psychology. New York:

Dover.

Ericsson, K. A., & Moxley, J. H. (2011). Thinking aloud protocols: Concurrent verbalizations of

thinking during performance tasks involving decision making. In Schulte-Mecklenbeck,

M., Kuhberger, A., & Ranyard, R. (Eds.), The Handbook of Process Tracing Methods

for Decision Research: A Critical Review and User’s Guide. (p. 89-114). New York,

NY: Psychology Press.

Ericsson, K. A., & Simon H. (1984). Protocol analysis: Verbal reports as data. Cambridge,

MA: MIT Press.

Ericsson, K. A., & Simon, H. (1993). Protocol analysis: Verbal reports as data (Rev. ed.).

Cambridge, MA: MIT Press.

Ericsson, K. (2002). Towards a procedure for eliciting verbal expression of non-verbal

experience without reactivity: Interpreting the verbal overshadowing effect within the

theoretical framework for protocol analysis. Applied Cognitive Psychology, 16, 981-987.

Ericsson, K. (2003). Valid and non-reactive verbalization of thoughts during performance of

tasks: Towards a solution to the central problems of introspection as a source of

scientific data. Journal of Consciousness Studies, 10(9-10), 1-18.

Field, A. O. (2009). Discovering statistics using SPSS. London, England: SAGE.

Fishman, B., Konstantopoulos, S., Kubitskey, B.W., Vath, R., Park, G., Johnson, H., & Edelson,

158

D.C., 2013. Comparing the impact of online and face-to-face professional development

in the context of curriculum implementation. Journal of Teacher Education, 64(5), 426-

438.

Fourie, I., & Bothma, T. (2007). Information seeking: An overview of web tracking and the

criteria for tracking softeware. Aslib Proceedings, 59 (3), 264-284.

Gaissmaier, W., Fific, M., & Rieskamp, J. (2010). Analyzing response times to understand

decision processes. In M. Schulte-Mecklenbeck, A. Kuhberger, & R. Ranyard (Eds.),

The Handbook of Process Tracing Methods for Decision Research: A Critical Review

and User’s Guide. (p. 89-114). New York, NY: Psychology Press.

Gambrell, L.B., Malloy, J.A., & Anders Mazzoni, S. (1999). Evidence-based best practices in

comprehensive literacy instruction. In Mandel Marrow, L. & Gambrell, L.B. (Eds.), Best

practices in literacy instruction (11-36). New York: The Guilford Press.

Garner, R., & Gillingham, (1991). Topic knowledge, cognitive interest, and text recall. The

Journal of Experimental Education, 59(4), 310-319.

Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory: Strategies for

qualitative research. Chicago: Aldine Publishing Co.

Glesne, C. (1999). Becoming qualitative researchers: An introduction (2nd ed.). New York:

Longman.

Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The

Qualitative Report, 8(4), 597-607.

Gorozidis, G., & Papaioannour, A.G. (2014). Teachers’ motivation to participate in training and

to implement innovations. Teaching and Teacher Education, 39, 1-11.

Guan, Z., Lee, S., Cuddihy, E., & Ramey. J. (2006). The validity of the stimulated retrospective

think-aloud method as measured by eye tracking. CHI, April 22-27.

159

Guskey, T.R. (2002). Professional development and teacher change. Teachers and Teaching:

Theory and Practice, 8(4), 381-391.

Guo, Y., Piasta, S.B., Justice, L.M., & Kaderavek, J.N. (2010). Relations among preschool

teachers’ self-efficacy, classroom quality, and children’s language and literacy gains.

Teaching and Teacher Education, 26, 1094-1103.

Hartree, A. (1984). Malcolm Knowles’ theory of andragogy: A critique. International Journal of

Lifelong Education, 3(3), 203-210.

Haymore Sandholtz, J. (2002). Inservice training or professional development: Contrasting

opportunities in a school/university partnership. Teaching and Teacher Education, 18,

815-830.

Hoekstra, A., Brekelmans, M., Beijaard, D., & Korthagen, F. (2009). Experienced teachers’

informal learning: Learning activities and changes in behaviour and cognition. Teacher

and Teacher Education, 25, 663-673.

Hogan, T., Rabinowitz, M., & Craven, J.A. (2003). Representation in teaching: Inferences from

research of expert and novice teachers. Educational Psychologist, 38(4), 235-247.

Horz, H., & Schnotz, W. (2010). Cognitive load in learning with multiple representations. In

Plass, J.L., Moreno, R., & Brunken, R. (Ed.), Cognitive load theory (229-252). New

York: Cambridge University Press.

Hur, J. W., & Brush, T. A. (2009). Teacher participation in online communities: Why do

teachers want to participate in self-generated online communities of K-12 teachers?

Journal of Research on Technology in Education, 41(3), 279-303.

Iding, M.K., Crosby, M.E., Auernheimer, B., & Klemm, E.B. (2009). Website credibility: Why

do people believe what they believe? Instructional Science, 37, 43-63.

Intons-Peterson, M.J., & Fournier, J. (1986). External and internal memory aids: When and how

160

often do we use them? Journal of Experimental Psychology, 115(3), 267-280.

Jamieson, B. (2007, June). Male presence in teaching continues to decline. Professionally

Speaking. Retrieved from http://professionallyspeaking.oct.ca/june_2007/

male_teachers.asp

Jaspers, M. W. (2009). A comparison of usability methods for testing interactive health

technologies: Methodological aspects and empirical evidence. International Journal of

Medical Informatics, 78, 340-353.

Jaspers, M., Steen, T., van den Bos, C., & Geenen, M. (2004). The think aloud method: A guide

to user interface design. International Journal of Medical Infomatics, 73, 781-795.

Jones, M., & Dexter, S. (2014). How teachers learn: The roles of formal, informal, and

independent learning. Educational Technology Research Development, 62, 367-384.

Jurasaite-Harbison, E., & Rex, L.A. (2010). School cultures as contexts for informal teacher

learning. Teaching and Teacher Education, 26, 267-277.

Kansanen, P., Tirri, K., Meri, M., Krokfors, L., Husu, J., & Jyrhama, Riitta. (2000). Teachers’

pedagogical thinking: Theoretical landscapes, practical challenges. New York: NY, Peter

Lange Publishing, Inc.

Kanuka, H., & Nocente, N. (2003). Exploring the effects of personality type on perceived

satisfaction with web-based learning in continuing professional development. Distance

Education, 24(2), 227-244.

Kao, C. P., Wu, Y. T., & Tsai, C. C. (2011). Elementary school teachers motivation toward

web-based professional development, and the relationship with Internet self-efficacy and

belief about web-based learning. Teacher and Teacher Education, 27, 406-415.

Kennedy, E. (2010). Improving literacy achievement in a high-poverty school: Empowering

classroom teachers through professional development. Reading Research Quarterly,

161

45(4), 384-387.

Knowles, M. (1975). Self-Directed Learning: A Guide for Learners and Teachers, New York,

NY: Association Press.

Knowles, M. S. (1977). The modern practice of adult education: Andragogy versus pedagogy

(8th ed.). New York: Association Press.

Knowles, M. S. (1980). My farewell address...andragogy: no panacea, no ideology. Training

and Development Journal, 48-50.

Knowles, M.S., Holton, E.F., & Swanson, R.A. (2011). The adult learner: The definitive

classic in adult education and human resource development (7th

Ed.). New York, NY:

Elsevier Inc.

Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol

analysis. American Journal of Psychology, 113 (3), 387-404.

Krathwohl, D. R. (2002). A revision of Bloom’s Taxonomy: An overview. Theory into Practice,

41(4), 212-218.

Laberge, J.C., & Scialfa, C.T. (2005). Predictors of web navigation performance in a life span

sample of adults. The Journal of the Human Factors and Ergonomics Society, 47(2),

289-302.

Leech, N.L., & Onwuegbuzie, A.J. (2007). An array of qualitative data analysis tools: A call for

data analysis triangulation. School Psychology Quarterly, 22(4), 557-584.

Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory into Practice, 32(3),

131-137.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.

Livingston, C., & Borko, H. (1989). Expert-novice differences in teaching: A cognitive analysis

and implications for teacher education. Journal of Teacher Education, 40(4), 36-42.

162

Lohman, M. C. (2006). Factors influencing teachers’ engagement in informal learning activities.

Journal of Workplace Learning, 18(3), 141-156.

Lomard, M., Snyder-Duch. J., & Campanella Bracken, C. (2004). Practical resources for

assessing and reporting intercoder reliability in content analysis research projects.

Retrieved from http://astro.temple.edu/~lombard/reliability

Loyens, S.M., Magda, J., & Rikers, R. (2008). Self-directed learning in problem-based learning

and its relationship with self-regulated learning. Educational Psychological Review, 20,

411-427.

Maass, K. (2011). How can teachers’ beliefs affect their professional development? The

International Journal of Mathematics Edcuation, 43(4), 573-586.

Mastin, L. (2010). The human memory: Memory encoding. Retrieved from

http://www.human-memory.net/processes_encoding.html

Masuda, A.M., Ebersole, M.M., & Barrett, D. (2013). A qualitative inquiry: Teachers attitudes

and willingness to engage in professional development experiences at different career

stages. Educational Research, 6-14.

Mayer, R. (2002). Cognitive theory and the design of multimedia instruction: An example of the

two-way street between cognition and instruction. New Directions for Teaching and

Learning, 2002(89), 55-71.

McDonald, L. (2012). Teacher learning, transfer and motivation: facilitating improved outcomes

for students. Journal of Educational Sciences & Psychology, 2(2), 15-24.

McDonald, S., Edwards, H., & Zhao, T. (2012). Exploring think-alouds in usability testing: An

international survey. IEEE Transactions on Professional Communication, 55 (1), 2-19.

McLeod, S. A. (2007). Cognitive Approach in Psychology - Simply Psychology. Retrieved from

http://www.simplypsychology.org/cognitive.html

163

McLoughlin, D., & Mynard, J. (2009). An analysis of higher order thinking in online

discussions. Innovations in Education and Teaching Internationally, 46(2), 147-160.

Mercier, J., Girard, C., Brodeur, M., & Laplante, L. (2010). Collaborative and individual

learning in teacher: A trajectory to expertise in pedagogical reasoning. New York:

Nova Science Publishers Inc.

Merriam, S. (2001). Andragogy and self-directed learning: Pillars of adult learning theory. New

Directions for Adult and Continuing Education, 89, 3-13.

Moos, D. (2014). Setting the stage for the metacognition during hypermedia learning: What

motivation constructs matter? Computers & Education 70, 128-137.

Mutlu Bayraktar, D., & Altun, A. (2014). The effect of multimedia design types on learners’

recall performances with varying short term memory spans. Multimedia Tools and

Applications, 71(3), 1201-1213.

National Early Literacy Panel (2008). Developing early literacy: Report of the national early

literacy panel. National Institute for Literacy.

National Reading Panel (2000). Teaching children to read: An evidence-based assessment of the

scientific research literature on reading and its implications for reading instruction.

Reports of the subgroups. Washington, D.C.: National Institute of Child Health and

Human Development.

Nielsen, J., Clemmensen, T., & Yssing, C. (2002). Getting access to what goes on in people’s

heads?-Reflections on the think-aloud technique. NordiCHI, October 19-23.

Nir, A.E., & Bogler, R. (2008). The antecedents of teacher satisfaction with professional

development programs. Teaching and Teacher Education, 24, 377-386.

NVivo, NVivo qualitative data analysis software; QSR International Pty Ltd. Version 10, 2012.

Olmsted-Hawala, E., Murphy, E., & Hawala, S. (2010). Think-aloud protocols: Analyzing three

164

different think-aloud protocols with counts of verbalized frustrations in a usability study

of an information-rich website. IEEE International Professional Communication

Conference.

Pearson, J., Pearson, A., & Green, D. (2007). Determining the importance of key criteria in web

usability. Management Research News, 30(11), 816-828.

Page K.L., Robson, M.J., & Uncles, M.D. (2012). Perceptions of web knowledge and usability:

When sex and experience matter. International Journal of Human-Computer Studies,

70(12), 907-919.

Paas, F.G., & Van Merrienboer, J. (1994). Variability of worked examples and transfer of

geometrical problem-solving skills: A cognitive-load approach. Journal of Educational

Psychology, 86(1), 122-133.

Ping Lim, C. (2004). Engaging learners in online learning environments. Tech Trends, 48(4),

16-23.

Pitsoe, V.J., & Maila, W.M. (2012). Towards constructivist teacher professional development.

Journal of Social Sciences, 8(3), 318-324.

Podhajski, B., Mather, N., Nathan, A., & Sammons, J. (2009). Professional development in

scientifically based reading instruction: Teacher knowledge and reading outcomes.

Journal of Learning Disabilities, 42(5), 403-417.

Poland, B. D. (1995). Transcription quality as an aspect of rigor in qualitative research.

Qualitative Inquiry, 1(3), 290-310.

Ponterotto, J.G. (2006). Brief note on the origins, evaluation, and meaning of the qualitative

research concept ‘thick descriptions’. The Qualitative Report, 11(3), 538-549.

Ponti, M. (2014). Self-directed learning and guidance in non-formal open courses. Learning,

Media and Technology, 39(2), 154-168.

165

Porche, M.V., Pallante, D.H., & Snow, C.E. (2012). Professional development for reading

achievement. The Elementary School Journal, 112(4), 649-671.

Postholm, M.B. (2012). Teachers’ professional development: A theoretical review. Educational

Research, 54(4), 405-429.

Pratt, D.D. (1993). Andragogy after twenty-five years. New Directions for Adult and Continuing

Education, 57, 15-23.

Pressley, M., Mohan, L., Raphael, L.M., & Fingeret, L. (2007). How does Bennett Woods

Elementary School produce such high reading and writing achievement? Journal of

Educational Psychology, 99(2), 221-240.

Protheroe, N. (2008). Teacher efficacy: What is it and does it matter? Principal, 87(5), 42.

Rachel, J.R. (2002). Andragogy’s detectives: A critique of the present and a proposal for the

future. Adult Education Quarterly, 52(3), 210-227.

Ranyard, R., & Svenson, O. (2010). Verbal data and decision process analysis. In M. Schulte-

Mecklenbeck, A. Kuhberger, & R. Ranyard (Eds.), The Handbook of Process Tracing

Methods for Decision Research: A Critical Review and User’s Guide. (p. 89-114). New

York, NY: Psychology Press.

Recker, M., Walker, A., Giersch, S., Mao, X., Halioris, S., Palmer, B., Johnson, D., Leary, H., &

Robertshaw, M.B. (2007). A study of teachers’ use of online learning resources to design

classroom activities. New Review of Hypermedia and Multimedia, 13(2), 117-134.

Reeves, T.D., & Pedulla, J.J. (2011). Predictors of teacher satisfaction with online professional

development: Evidence form the USA’s e-Learning for Educators initiative. Professional

Development in Education, 37(4), 591-611.

Rich, P., & Hannafin, M. (2008). Capturing and assessing evidence of study teacher inquiry: A

case study. Teaching and Teacher Education, 24(6), 1426-1440.

166

Rosenzweig, C., Krawec, J., & Montague, M. (2011). Metacognitive strategy use of eighth-

grade students with and without learning disabilities during mathematical problem

solving: A think-aloud analysis. Journal of Learning Disabilities, 44(6), 508-520.

Rowan, M., & Huston, P. (1997). Qualitative research articles: Information for authors and peer

reviewers. Canadian Medical Association, 157(10), 1442-1446.

Ryan, R.M., & Deci, E.L. (2000). Self-determination theory and the facilitation of intrinsic

motivation, social development, and well-being. American Psychologist, 55(1), 68-78.

Schellenbach-Zell, J., & Grasel, C. (2010). Teacher motivation for participating in school

innovations—supporting factors. Journal for Educational Research Online, 2(2), 34-54.

Shenton, A.K. (2004). Strategies for ensuring trustworthiness in qualitative research projects.

Education for Information, 22, 63-75.

Schryer, E., & Ross, M. (2013). The use and benefits of external memory aids in older and

younger adults. Applied Cognitive Psychology, 27, 663-671.

Shulman, L.S., & Shulman, J.H. (2004). How and what teachers learn: A shifting perspective.

Journal of Curriculum Studies, 36(2), 257-271.

Song, D., & Lee, J. (2014). Has web 2.0 revitalized informal learning? The relationship between

web 2.0 and informal learning. Journal of Computer Assisted Learning, 30, 511-533.

Song, L., & Hill, J.R. (2007). A conceptual model for understanding self-directed learning in

online environments. Journal of Interactive Online Learning, 6(1), 27-41.

Tan, G. W., & Wei, K. K. (2006). An empirical study of web browsing behaviour: Towards an

effective website design. Electronic Commerce Research and Applications, 5, 261-271.

Taylor, B., & Kroth, M. (2009). A single conversation with a wise man is better than ten years

of study: A model for testing methodologies for pedagogy or andragogy. The Journal of

Scholarship of Teaching and Learning, 9, 42-56.

167

Thomas, D. (2006). A general inductive approach for analyzing qualitative data. American

Journal of Evaluation, 27(2), 237-246.

Tilley, S. A. (2003). “Challenging” research practices: Turning a critical lens on the work of

transcription. Qualitative Inquiry, 9(5). 750-773.

Trotter, Y. (2006). Adult learning theories: Impacting professional development programs. The

Delta Kappa Gamma Bulleti, 72 (2), 8-13.

Tschannen-Moran, M., & Johnson, D. (2011). Exploring literacy teachers’ self-efficacy beliefs:

Potential sources at play. Teaching and Teacher Education, 27, 751-761.

Tschannen-Moran, M., Woolfolk Hoy, A., & Hoy, W.K. (1998). Teacher efficacy: Its meaning

and measure. Review of Educational Research, 68(2), 202-248.

UNESCO. (2009). Open educational resources: Conversations in cyberspace. S. D’Antoni, &

C. Savage, Eds. Paris: UNESCO.

Uppal, T. (2011). Internet use for lesson preparation by pre-service teachers: A mixed methods

study. (Unpublished Master’s thesis.) OISE, University of Toronto, Toronto, Canada.

van den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2003). Retrospective vs. concurrent

think-aloud protocols: Testing the usability of an online library catalogue. Behaviour &

Information Technology, 22(5), 339-52.

van Gog, T., Kester, L., Nievelstein, F., Giesbers, B., & Paas, F. (2009). Uncovering cognitive

processes: Different techniques that can contribute to cognitive load research and

instruction. Computers in Human Behavior, 25, 325-331.

van Gog, T., Paas, F., van Marrienboer, J., & Witte, P. (2005). Uncovering the problem-solving

process: Cued retrospective reporting versus concurrent and retrospective reporting.

Journal of Experimental Psychology: Applied, 11 (4), 237-244.

168

van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. (1994). The think aloud method: A

practical guide to modelling cognitive processes. London: Academic Press.

Vermunt, J.D., & Endedijk, M.D. (2011). Patterns in teacher learning in different phases of the

professional career. Learning and Individual Differences 21, 294-302.

Warren, L.L. (2000). Teacher planning: A literature review. Educational Research Quarterly,

24(2), 37.

Weber, K., Corrigan, M., Fornash, B., & Neupauer, N. (2003). The effect of interest on recall:

An experiment. Communication Research Reports, 20(2), 116-123.

Westerman, D.A. (1991) Expert and novice teacher decision making. Journal of Teacher

Education, 42(4), 292-305.

Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., & Hemmerich, J.A. (2009).

Source evaluation, comprehension, and learning in Internet science inquiry tasks.

American Educational Research Journal, 46(4), 1060-1106.

Wing-mui So, W. (1997). A study of teacher cognition in planning elementary science lessons.

Research in Science Education, 27(1), 71-86.

Zhao, T., & McDonald, S. (2010). Keep talking: An analysis of participant utterances gathered

using two concurrent think-aloud methods. Proceedings: NordiCHI, 16-20.

169

Appendices

Appendix A-1

Sample Pages of The Balanced Literacy Diet Website

Food Group Page Example

Recipe Finder Page Example

170

Recipe Example

Virtual Tour Example

171

YouTube Channel

Content Page Example

172

Appendix A-2

Invitation to Participate Script

My name is Pamela Beach. I am currently a PhD student at the Ontario Institute for Studies in

Education at the University of Toronto. I would like to invite you to be a part of a research

project that I am completing involving the use of educational websites by elementary teachers. I

am conducting this project for my dissertation and to learn more about how elementary teachers

use an educational website to assist in professional development in literacy instruction.

If you decide to participate, you will first be asked to complete an online questionnaire. The

website link to the questionnaire will be emailed to you and should take about 15 minutes to

complete. You will then be contacted to arrange a one-on-one meeting session.

The one-on-one session will take about one hour and can occur at a time that is most convenient

to you. During the session, you will be asked to navigate through a professional development

website and think aloud about your navigational experience. A list of questions will then be

asked concerning your experience of exploring the website as well as your general feelings

towards thinking aloud.

Your involvement would be greatly appreciated and upon completion of the tasks you would

receive a gift certificate from Indigo Book Store. Please let me know if have any questions

regarding this project. If you do decide to participate, please complete a consent form, place it in

the provided envelop and leave it in the designated box. I will return at the end of this class to

collect the box.

173

Appendix A-3

Information Letter and Consent Form

Dear Participant,

My name is Pamela Beach. I am currently a PhD student at the Ontario Institute for Studies in

Education at the University of Toronto. I would like to invite you to be a part of a research

project that I am completing involving the use of educational websites by elementary teachers. I

am conducting this project to learn more about how elementary teachers use an educational

website to assist in professional development in literacy instruction and what elementary

teachers think about while navigating online resources. I am asking you to take part in this

research because I believe that your experiences and opinions are valuable information.

Description of the Project

If you take part in this research, you will be asked to complete an online questionnaire. The

website link to the questionnaire will be emailed to you and should take about 15 minutes to

complete. You will then be contacted to arrange a one-on-one meeting session.

The one-on-one session will take about one hour and can occur at a time that is most convenient

to you. During the session, you will be asked to navigate through a professional development

website and think aloud about your navigational experience. A list of questions will then be

asked concerning your experience of exploring the website as well as your general feelings

towards thinking aloud. You will be asked to complete a second consent form at this time.

Upon completion of the tasks, you will receive a gift certificate to a book store.

Benefits

The main benefit of participating is that you will help us learn more about teachers and the use

of an innovative educational website. This will allow us to evaluate this website and its features

as online professional resources. We will also begin to understand how elementary teachers use

professional development websites for their teaching practice.

Potential Harms and Withdrawal There are no harms associated with taking part in this research. If at any time you feel that you

do not want to answer a question, you can tell the researcher, and talk about it. You may also

say that you want to stop, skip a question, or that you need a break and want to continue at some

other time. Also, if you say that you will take part in the research and then change your mind,

this is okay. You can decide at any time to stop taking part in the research. The only

consequence is that we would not have the information we need to write the report that we

described above.

174

Confidentiality

Your answers will only be used for research. No information that reveals your identity will be

released without consent, unless required by law. I will not put your name or your personal

information on any of the activities we do together, or include them in anything that I write or

present describing the research results. These will be kept in locked filing cabinets in our

research lab. Electronic data will be stored on a secure server. We will use a code on these files

instead of your name and all of the files will be encrypted.

Consent-Participation In This Study Is Your Choice

It is completely up to you to decide if you would like to be part of this research. If you are

interested to participate, you will be asked to sign the consent form. As a participant, you may

skip any questions that you do not want to answer, ask for a break and/or leave the study at any

time. I will be happy to answer any of your questions. If you would like to talk about taking

part in this study with someone who is not involved in it, you may call the Ethics Review Office

at the University of Toronto.

Access to Results

A summary of the overall results of this study can be mailed/e-mailed to you about 1 year after

the study. If you would like a copy of the results, please place a checkmark in the box .

As well, you can always contact Dr. Willows or me at any time with questions that you may

have about the study or its results.

Thank you,

175

Consent:

“I acknowledge that the research procedures described above have been explained to me and

that any questions that I have asked have been answered to my satisfaction. As well, the

potential harms and discomforts have been explained to me and I also understand the benefits of

participating in the research study. I know that I may ask now, or in the future, any questions

that I have about the study. I have been assured that no information will be released or printed

that would disclose my identity without my permission, unless required by law. I understand

that I will receive a copy of this signed consent. I understand that participation is voluntary and

I can withdraw at any time.”

I hereby consent to take part in this research.

Name (please print)

Email (please use the address to which you

would like the survey to be sent)

Signature

Date

“I agree that the information gathered in this study can be used for future data analysis as long

as my personal information (i.e. name) is not used.”

Signature

If you are interested in participating in this study, please fill out the attached consent form

and return it in the envelope provided to the designated box.

176

Appendix A-4

Recruitment Poster

Department of Applied Psychology and Human Development

For more information please contact:

Are you a practicing elementary school teacher?

Do you currently teach at the primary or junior level?

If so, you are invited to participate in a research

study on the use of educational websites by

elementary teachers.

Participation will involve a 15 minute online questionnaire and a one-on-one website task and follow-up interview. The task will take approximately 45 minutes and will occur at a time and location that is most convenient to you. Upon completion of the interview, participants will receive a gift certificate to Chapters.

Practicing Teachers Needed for Research Study on Educational Websites

177

Appendix A-5

Demographic Characteristics across the Three Different Modes of Recruitment

Demographic

Characteristic

Recruitment Type

Poster AQ Course Word of Mouth

Gender

Male 6 1 2

Female 22 3 11

Age

25-29 5 2 2

30-34 10 0 2

35-39 8 1 2

40-44 1 1 4

45-49 2 0 1

50-54 1 0 1

55+ 1 0 1

Years Teaching

2-4 years 6 2 2

5-9 years 14 0 3

10-14 years 6 2 4

15-19 years 1 0 1

20-24 years 1 0 1

25+ years 0 0 2

Current Grade

Kindergarten 1 0 1

Primary 14 0 3

Junior 9 1 9

Primary/Junior 4 3 0

Comfort Using

Internet

Very

Comfortable

25 3 13

Somewhat

Comfortable

3 1 0

Not Very

Comfortable

0 0 0

Frequency of

Internet Use for

Professional

Purposes

> once/day 9 1 4

Once/day 13 1 7

Once/week 5 2 2

Once/month 1 0 0

178

Appendix A-6

Demographic Questionnaire

1. Which grade do you currently teach?

□ Kindergarten □ Grade 4

□ Grade 1 □ Grade 5

□ Grade 2 □ Grade 6

□ Grade 3

2. What grade(s) have you taught in your teaching career?

□ Kindergarten □ Grade 4

□ Grade 1 □ Grade 5

□ Grade 2 □ Grade 6

□ Grade 3

3. What is your gender?

Male

Female

4. What is your age-range?

□ 20-25 years □ 41-45 years

□ 26-30 years □ 46-50 years

□ 31-35 years □ 51-55 years

□ 36-40 years □ 56 + years

5. How long have you been teaching at the elementary level?

years

6. What technology devices do you own?

Desktop Computer

Laptop Computer

PDA (personal data assistant)

179

Ipod/MP3 Player with Internet Access

Smartphone: _____________

Tablet PC

iPad

eReader Device

Other: __________________________________________________________

7. What technology devices do you use?

Desktop Computer

Laptop Computer

PDA (personal data assistant)

Ipod/MP3 Player with Internet Access

Smartphone: _____________

Tablet PC

iPad

eReader Device

Other: __________________________________________________________

8. Do you use the Internet for your teaching practices?

□ Yes

□ No

(a) If yes, please indicate how you use the Internet.

9. How comfortable do you feel using the Internet?

□ Very comfortable

□ Somewhat comfortable

□ Not very comfortable

□ Not comfortable at all

180

10. How frequently do you use the Internet for your teaching practices?

Once a month

Once a week

Once a day

More than once a day

11. What percent of the time would you say you spend using the Internet to plan your literacy

lessons/program?

________%

12. What resources do you use when planning for your classroom set-up/literacy lessons?

□ Books

□ Internet

□ Literacy Coach

□ Workshop Material

□ Professional Learning Community Material

Other:

13. What three websites do you use most often in your teaching practices?

(i)

(ii)

(iii)

14. Why do you like these websites?

15. Which of the following website tools do you use when exploring educational websites?

□ Video clips

□ Photographs

181

□ Blogs

□ Chat

□ Discussion boards

□ Other:

16. Which website tools do you find most beneficial to your teaching practices or placements?

□ Video clips

□ Photographs

□ Blogs

□ Chat

□ Discussion boards

□ Other:

17. How passionate are you about teaching literacy? Please circle one.

1 2 3 4 5 6 7

Not at all Extremely

18. How confident do you feel about teaching literacy? Please circle one.

1 2 3 4 5 6 7

Not at all Extremely

19. How confident do you feel about using website features/tools? Please circle one.

1 2 3 4 5 6 7

Not at all Extremely

20. How often do you attend workshops or conferences? Please circle one.

1 2 3 4

Never Not often Often Very often

(a) What are your reasons for your choice you made in question 23?

21. How often do you attend workshops for literacy? Please circle one.

1 2 3 4

Never Not often Often Very often

182

22. How motivated are you to incorporate a new teaching strategy that you have learned about

into your future program?

1 2 3 4 5 6 7

Not at all Extremely

Thank you

183

Appendix A-7

Think Aloud and Interview Consent Form

Thank you for participating in this research study. During this session, you will be asked to

navigate through a professional development website and think aloud about your navigational

experience. A list of questions will then be asked concerning your experience of exploring the

website as well as your general feelings towards thinking aloud. You may skip any questions if

you do not want to answer them. You may choose to end the interview at any time, without

consequences. The think aloud and interview will be audio-recorded so that we can later

transcribe it to ensure we do not lose any valuable information.

Your signature below indicates that you understand that the think aloud and interview will be

audio-recorded and that you give your consent for this.

_____________________________________ _________________________________

Name (please print) Signature

_ ______

Date

184

Appendix A-8

Screen Shot of Camtasia Studio During a Participant’s Navigation

185

Appendix A-9

Interview Questions

1. What were your general feelings while you explored the Balanced Literacy Diet website?

2. On a scale of 1 to 7, 1 indicating very uncomfortable and 7 indicating very comfortable, how

did you feel about thinking aloud while you were navigating/following your

navigation/while you watched a screen recording of your navigation? Why did you give

yourself this rating?

3. What did you find challenging while you explored the Balanced Literacy Diet website?

4. What features of the Balanced Literacy Diet website did you find most appealing?

5. What other features would you have liked to view/explore in the Balanced Literacy Diet

website?

6. Is there anything else you would like to share about online professional resources?

7. Is there anything else you would like to share about your experience of thinking aloud while

exploring The Balanced Literacy Diet website?

186

Appendix B-1

Initial List of Potential Categories for Coding Think Aloud Transcripts

Potential Categories

1. Impression category

2. Information content (textual, multimedia)

3. Layout (text, graphic elements)

4. Information architecture (structural design)

a. Links

b. Navigation

c. Structure

5. Diversion

a. Confusion

b. Loading time

6. Reading-reading words, phrases or sentences off the screen; reading texts, headings and

links

7. Procedure-describing current or future activity

8. Observation-making an observation about the website or participant’s own behaviour

9. Explanation-explaining, using either present or future tense, the reason for participant’s

behaviour; providing insight into the participant’s motivation for his/her behaviour

10. Verbal fillers

11. Identifying and making sense of new information

12. Evaluating claims and evidence

13. Considering ways to use the site in their own practice

14. Intentions-information representing goals-future states of the subject-shall, will, must

15. Cognitions-information based on attention to selected aspects of the current situation

16. Planning-information representing intermediate constructions to explore sequences of

possibilities

17. Evaluation-explicit or implicit comparisons

18. Comprehension

19. Problem solving

20. Concept formation

21. Prior knowledge

22. Decision making

23. An evaluation of an aspect

24. A causal relation

25. Collection of information

26. Evaluation of information

27. Strategy

28. Choice

29. Relative evaluation of options

30. Attention-refers to the processing of information on the site

31. Perception: relates to the perception of text and graphics on the site

32. Problem-solving: refers to the provision of support for using the sites

33. Memory: in relation to the retrieval of information on the site

34. Cognitive mapping: taking in sensory information (e.g., I see…)

a. Sensory information comprised of:

187

i. Spatial characteristics

ii. Signing of website (hyperlinks, navigational bar)

b. Inferential information:

i. Comments on website environment

c. Memory information:

i. Seldom recalled, when did: “other commercial sites”

35. Decision generation: subjects would often pause before making a decision

36. Action description-describe what they were doing or going to do or just did

37. Action explanation-explain the reason(s) for executing certain actions

38. Result evaluation-summarize understanding or give evaluation of content, links or the

outcome of an action

39. User experience-express positive or negative feelings, aesthetic preferences towards the

websites

40. Problem formulation-verbalize difficulties, including utterances that participants indicate

uncertainty; and utterances that participants not only express a negative feeling or

disapproval, but also indicate that it was caused by system based issue(s)

41. Causal explanation-explain what caused certain outcome or event

42. Recommendation-to improve site

43. Procedural behaviour- specific statements about what subjects did during their task

performance, such as “I read the instructions at the top”

44. Inferential/explanatory statements

a. Logic inference-directly inferred from or generated based on information that users

attended to, such as “I see the top and bottom of this”

b. Perception explanation-such as, “the picture is pretty bright”

c. Strategy explanation—about how subjects completed the task, such as “and this one I

just started doing the subtraction”

45. Reflective comments-contain self-diagnostics about what subjects did or should have done,

such as “for this one I was actually a little confused about what they were asking at first”

46. Meta-comments-such as, “this one kind of took me by surprise”

47. Evaluative statements: “are related to cognitive processes that link pieces of information

obtained through experience and stored in memory of the decision maker”

48. Predictive statements: reference to a future action

49. Decisions: simple statement of the selection of an option

50. Understand-construct meaning from instructional messages, including written and graphic

a. Interpreting-clarifying, paraphrasing, representing

b. Exemplifying-illustrating (finding a specific example or a concept

c. Classifying-categorizing

d. Summarizing-abstracting, generalizing

e. Inferring-concluding, extrapolating (drawing a logical conclusion from presented

information

f. Comparing-contrasting, mapping, matching

g. Explaining-constructing models

51. Analyze: break material into its constituent parts and determine how the parts relate to one

another and to an overall structure or purpose

a. Differentiating-discriminating, distinguishing (relevant from irrelevant parts,

important form unimportant parts of presented material), focusing, selecting

b. Organizing-finding coherence, integrating, outline, structuring (determining how

elements fit or function within a structure)

188

52. Evaluate-make judgements based on criteria and standards

a. Checking-coordinating, monitoring

b. Critiquing-judging

53. Create-put elements together to form a coherent whole; reorganize elements into a new

patter or structure

a. Generating-hypothesizing

b. Planning-designing

189

Appendix B-2

Krathwohl’s Revised Taxonomy

Structure of the Cognitive Process

Dimension of the Revised Taxonomy

(based on Krathwohl, 2002)

1.0 Remember—Retrieving relevant knowledge from long-term memory.

1.1 Recognizing

1.2 Recalling

2.0 Understand—Determining the meaning of instructional messages, including oral, written,

and graphic communication.

2.1 Interpreting

2.2 Exemplifying

2.3 Classifying

2.4 Summarizing

2.5 Inferring

2.6 Comparing

2.7 Explaining 3.0 Apply—Carrying out or using a procedure in a given situation.

3.1 Executing

3.2 Implementing 4.0 Analyze—Breaking material into its constituent parts and detecting how the parts relate to

one another and to an overall structure or purpose.

4.1 Differentiating

4.2 Organizing

4.3 Attributing 5.0 Evaluate—Making judgements based on criteria and standards.

5.1 Checking

5.2 Critiquing 6.0 Create—Putting elements together to form a novel, coherent whole or make an original

product.

6.1 Generating

6.2 Planning

6.3 Producing

190

Appendix B-3

Frequency Counts of Web-based Actions

Concurrent Condition

Behaviour/Action C0

1

C0

2

C0

3

C0

4

C0

5

C0

6

C0

7

C0

8

C0

9

C1

0

C1

1

C1

2

C1

3

C1

4

C1

5

Enters a search term 7 3 0 1 0 0 0 0 0 0 0 0 1 3 2

Maneuvers around virtual

tour 0 0 0 2 5 0 0 3 0 0 4 0 0 0 0

Opens content page 3 1 0 4 0 0 5 0 2 0 4 1 4 0 0

Opens food groups page 8 5 8 6 2 2 4 2 6 5 7 2 12 11 16

Opens homepage 4 1 1 2 2 0 2 1 2 4 3 0 6 2 2

Opens how-to videos 0 2 0 1 1 2 1 2 0 2 1 0 1 2 0

Opens outside link 11 1 1 4 0 4 1 1 2 0 2 0 5 0 0

Opens recipe 10 13 8 8 2 7 10 4 5 3 6 8 2 7 9

Opens recipe finder page 7 4 5 8 2 9 3 4 1 2 2 7 2 8 0

Opens virtual tour 3 4 1 4 2 1 1 1 2 1 1 0 4 0 0

Selects filter option 4 5 2 0 0 0 1 0 1 0 3 0 0 2 0

Starts a video 6 4 2 7 8 3 5 7 2 8 4 2 4 4 5

Stops video before the

end 4 4 2 6 7 2 5 8 2 8 4 1 4 4 4

Takes a note 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Uses virtual tour arrow 4 6 1 11 2 0 0 0 3 0 0 0 3 0 0

Uses virtual tour zoom 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0

Views a photograph 0 8 9 0 0 6 4 8 0 6 9 7 0 20 9

Views a video 13 5 5 22 40 9 4 38 11 19 10 9 10 8 32

Total Number of Pages 46 32 25 37 12 26 30 16 22 18 26 18 37 31 28

191

Retrospective Condition

Behaviour/Action R01 R02 R03 R04 R05 R06 R07 R08 R09 R10 R11 R12 R13 R14 R15

Enters a search term 0 0 0 0 3 1 0 0 1 0 0 0 1 0 0

Opens content page 1 1 1 1 1 2 1 0 1 0 1 0 3 0 0

Opens food groups page 3 1 2 5 3 3 2 3 1 2 4 2 5 3 3

Opens homepage 0 0 0 1 1 1 0 0 1 1 1 0 1 1 1

Opens how-to video 0 0 0 0 1 1 0 0 1 1 0 1 1 0 1

Opens outside link 4 1 0 0 0 1 0 0 1 0 0 0 1 2 2

Opens recipe 3 9 17 6 10 6 10 5 4 6 4 5 2 3 2

Opens recipe finder page 3 6 0 2 7 2 3 0 1 3 3 2 1 0 15

Opens virtual tour 1 0 1 1 1 1 1 1 2 4 2 1 1 1 0

Selects filter option 0 1 0 0 0 0 0 0 7 0 0 0 0 0 5

Starts video 3 0 2 7 3 3 2 6 2 8 1 6 4 4 2

Stops video before the end 1 0 1 3 3 0 2 3 1 3 0 0 0 0 1

Takes a note 0 0 0 1 0 0 0 8 0 0 0 0 0 0 0

Total number of pages 15 18 22 16 23 17 17 10 12 17 15 11 15 11 23

Uses virtual tour arrows 0 0 0 0 2 0 6 0 5 6 0 0 4 0 0

Uses virtual tour

maneuver 9 0 0 0 0 0 2 0 0 0 0 0 1 0 0

Uses virtual tour zoom 3 0 0 0 0 0 1 2 1 0 0 0 0 1 0

Views a photograph 15 9 37 2 17 16 7 0 0 11 0 0 1 0 6

Views video 32 0 12 65 23 30 19 47 12 56 8 66 64 46 11

192

Virtual Revisit Condition

Behaviour/Action V01 V02 V03 V04 V05 V06 V07 V08 V09 V10 V11 V12 V13 V14 V15

Enters a search term 0 0 0 0 0 0 0 1 0 0 2 2 1 0 0

Opens content page 0 0 1 0 1 0 1 0 1 1 0 0 0 1 0

Opens food groups page 2 5 3 4 0 4 5 1 4 3 2 1 2 7 1

Opens home page 3 1 0 0 1 1 1 0 2 0 0 0 0 0 1

Opens how-to video 0 0 0 1 1 0 0 1 0 0 1 0 2 2 0

Opens outside link 0 4 0 1 0 3 1 0 0 0 1 5 1 0 1

Opens recipe 5 10 0 4 9 10 6 6 6 2 8 8 6 5 9

Opens recipe finder page 5 8 1 4 9 8 9 5 0 2 5 4 8 4 6

Opens virtual tour 2 2 1 1 1 0 2 1 2 2 1 2 1 1 1

Selects filter option 0 2 0 1 0 0 6 2 0 0 1 1 3 1 0

Starts video 1 3 7 6 4 4 2 9 4 7 2 2 7 15 8

Stops video before the end 0 2 0 4 0 3 2 4 3 2 1 0 3 12 2

Takes a note 0 0 8 0 0 0 10 0 0 0 3 0 0 0 2

Total Number of Pages 18 31 7 16 23 25 25 15 16 11 19 21 21 21 19

Uses virtual tour arrows 12 2 0 8 0 0 0 1 1 0 0 10 2 0 0

Uses Virtual tour

maneuver 0 0 0 0 0 0 0 0 2 3 5 0 0 0 0

Uses virtual tour zoom 0 0 0 0 0 0 0 0 2 0 0 1 0 0 0

Views a photograph 17 3 0 8 10 24 0 0 24 0 7 0 0 0 0

Views video 9 35 69 47 29 35 6 61 26 45 8 14 43 48 59

193

Appendix B-4

Sample Observations of Web-based Actions with Think Aloud Recordings

Concurrent Condition

*Note: SV = Silent interval, Video playing

0:10 0:20 0:30 0:40 0:50 1:00 1:10 1:20 1:30 1:40 1:50 2:00 2:10 2:20 2:30 2:40

C01 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C02 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C03 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C04 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C05 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C06 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C07 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C08 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y SV

C09 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C10 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C11 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C12 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C13 Y Y Y Y Y Y Y Y Y Y Y Y SV Y Y Y

C14 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

C15 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

Retrospective Condition:

0:10 0:20 0:30 0:40 0:50 1:00 1:10 1:20 1:30 1:40 1:50 2:00 2:10 2:20 2:30 2:40

R01 Y Y Y N N N N Y N N N N N N N N

R02 N N N N N N Y Y Y Y Y Y Y Y Y Y

R03 N N N Y N N N N N N N N N N N N

R04 N N N N N N N N N N Y Y Y Y N N

R05 Y Y Y N N N N N N N N N N N N N

R06 N N N Y Y Y N N N N N N N N N N

R07 Y Y Y Y Y N N N N N N N N N N N

R08 N N N N N N N N N N N N N N N N

R09 Y Y N N N N N N N N N N N N N N

R10 Y Y Y Y Y Y Y N N N N N N N N N

R11 N N N N N N N N N N N

R12 N N N N N N N N N N N N N N N N

R13 Y Y Y Y Y Y Y Y N N N

R14 Y Y Y Y Y Y N N N N N N N N N N

R15 Y Y Y Y Y N N N N N N N N N N

194

Virtual Revisit Condition:

0:10 0:20 0:30 0:40 0:50 1:00 1:10 1:20 1:30 1:40 1:50 2:00 2:10 2:20 2:30 2:40

V01 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V02 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V03 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V04 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V05 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V06 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V07 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V08 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V09 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V10 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V11 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V12 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V13 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V14 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

V15 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

195

Appendix B-5

Data Screening Results and Descriptives

N Mean Skewness Kurtosis 95% Confidence

Interval for Mean

Group Stat. SE Stat. SE Stat. SE Lower

Bound

Upper

Bound

Planning 1 15 0.41 0.18 1.75 0.58 2.25 1.12 0.02 0.80

2 15 3.96 0.99 0.48 0.58 -1.20 1.12 1.94 6.08

3 15 4.17 0.94 1.18 0.58 -0.03 1.12 2.15 6.20

Connecting 1 15 6.04 0.93 -0.04 0.58 -1.12 1.12 4.05 8.03

2 15 13.23 1.69 0.22 0.58 -0.70 1.12 9.61 16.85

3 15 9.08 0.98 0.10 0.58 -1.20 1.12 6.97 11.19

Reasoning 1 15 2.23 0.64 1.22 0.58 0.22 1.12 0.87 2.60

2 15 10.99 1.97 0.58 0.58 1.20 1.12 6.75 15.22

3 15 19.27 2.36 -0.06 0.58 -0.85 1.12 14.20 24.35

Reflecting 1 15 16.71 1.99 0.60 0.58 0.04 1.12 12.45 20.98

2 15 26.86 2.96 -0.15 0.58 -0.54 1.12 20.52 33.20

3 15 25.08 1.93 0.99 0.58 0.83 1.12 20.94 29.22

Website

Content

1

15

10.87

1.28

0.56

0.58

-0.25

1.12

8.12

13.61

2 15 12.18 2.25 -0.82 0.58 -1.26 1.12 7.36 17.00

3 15 15.36 1.36 -0.21 0.58 -0.88 1.12 12.44 18.28

User

Experience

1

15

7.96

1.80

1.07

0.58

0.82

1.12

4.09

11.83

2 15 12.13 2.33 1.20 0.58 3.20 1.12 7.14 17.13

3 15 7.47

0.85 0.16 0.58 -0.36 1.12 5.64 9.30

Understanding 1 15 5.99 0.90 0.73 0.58 -0.24 1.12 4.05 7.93

2 15 2.97 1.15 1.95 0.58 4.23 1.12 0.50 5.43

3 15 3.36 0.81 1.47 0.58 2.13 1.12 1.62 5.10

Diversion 1 15 5.97 0.78 0.04 0.58 -1.88 1.12 4.30 7.64

2 15 0.79 0.54 2.41 0.58 4.25 1.12 0.36 1.94

3 15 2.99 0.77 1.55 0.58 2.27 1.12 1.34 4.64

Procedural

Behaviour

1

15

11.64

1.60

0.56

0.58

0.05

1.12

8.20

15.08

2 15 11.71 2.35 0.52 0.58 0.01 1.12 6.66 16.76

3 15 9.37 1.27 0.14 0.58 -0.53 1.12 6.65 12.09

Website

Features

1

15

10.15

1.26

0.24

0.58

-1.13

1.12

7.45

12.84

2 15 3.49 1.25 1.24 .058 1.12 1.12 0.82 6.17

3 15 3.56

0.91 1.98 0.58 3.81 1.12 1.60 5.52

Reading 1 15 22.03 3.30 0.27 0.58 -0.38 1.12 14.95 29.12

2 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

3 15 0.29 0.16 1.97 0.58 2.50 1.12 0.06 0.65

Word Count

1

15

1676.13

101.96

0.42

0.58

0.93

1.12

1457.45

1894.82

2 15 658.80 59.13 0.23 0.58 -0.38 1.12 531.98 785.62

3 15 2637.87 92.94 -0.98 0.58 0.56 1.12 2438.54 2837.20

196

Number of

Thought Units

1 15 147.40 9.25 0.56 0.58 0.77 1.12 127.56 167.24

2 15 30.73 3.40 0.60 0.58 -0.14 1.12 23.45 38.02

3 15 123.27 5.31 0.90 0.58 0.64 1.12 111.87 134.66

Time on Each

Page

1

15

36.07

2.35

0.32

0.58

0.02

1.12

31.04

41.10

2 15 52.53 3.90 0.08 0.58 -0.84 1.12 44.17 60.90

3 15 47.80 3.07 0.42 0.58 -0.99 1.12 41.23 54.37

Number of

Pages Visited

1

15

26.93

2.33

0.36

0.58

0.03

1.12

21.94

31.93

2 15 16.13 1.08 0.32 0.58 -0.65 1.12 13.81 18.45

3 15 18.60 1.37 -0.40 0.58 0.77 1.12 15.66 21.54

Food Group

Pages

1

15

6.40

1.06

0.95

0.58

0.61

1.12

4.12

8.68

2 15 2.80 0.31 0.44 0.58 -0.07 1.12 2.13 3.47

3 15 2.93 0.49 0.47 0.58 -0.18 1.12 1.88 3.99

Content Pages 1 15 1.60 0.49 0.95 0.58 -1.29 1.12 0.56 2.64

2 15 0.87 0.22 1.13 0.58 1.96 1.12 0.40 1.33

3 15 0.40 0.13 0.46 0.58 -2.09 1.12 0.12 0.68

Visits to

Homepage

1

15

2.13

0.41

0.96

0.58

1.20

1.12

1.25

3.02

2 15 0.60 0.13 -0.46 0.58 -2.09 1.12 0.32 0.88

3 15 0.67 0.23 1.46 0.58 2.01 1.12 0.17 1.16

Time Viewing

Videos

1

15

150.67

32.30

1.08

0.58

-0.05

1.12

81.39

219.94

2 15 334.00 60.68 0.21 0.58 -1.61 1.12 203.86 464.14

3 15 355.33 51.72 -0.83 0.58 -1.08 1.12 244.40 466.26

Number of

Videos

Stopped

1

15

4.27

0.53

0.27

0.58

-0.62

1.12

3.13

5.40

2 15 1.20 0.33 0.55 0.58 -1.42 1.12 0.50 1.90

3 15 2.53 0.77 2.47 0.58 7.78 1.12 0.89 4.18

Enters Search

Term

1

15

1.13

0.51

2.22

.058

5.36

1.12

0.05

2.22

2 15 0.40 0.21 2.54 0.58 7.07 1.12 0.06 0.86

3 15 0.40 0.19 1.63 0.58 1.32 1.12 0.01 0.81

Maneuvers

Virtual Tour

1

15

0.93

0.44

1.60

0.58 1.24 1.12 0.01 1.88

2 15 0.80 0.60 3.54 0.58 12.94 1.12 0.49 2.09

3 15 0.67 0.39 2.29 0.58 4.74 1.12 0.16 1.50

Visits How-To

Videos

1

15

1.00

0.22

.00

0.58

-1.62

-1.20

0.63

1.37

2 15 0.47 0.13 0.15 0.58 -2.31 1.12 0.09 0.84

3 15 0.53 0.19 1.07 0.58 -0.11 2.20 0.16 0.91

Visits Outside

Link

1

15

2.13

0.76

2.18 0.58 5.51 1.12 0.50 3.77

2 15 0.80 0.30 1.76 0.58 3.37 1.12 0.17 1.43

3 15 1.13 0.41 1.57 0.58 1.54 1.12 0.25 2.02

Views Recipe

Page

1

15

6.80

0.82

0.04

0.58

-0.43

1.12

5.05

8.55

2 15 6.13 1.03 1.57 0.58 2.93 1.12 3.93 8.34

3 15 6.27 0.74 -0.65 0.58 0.18 1.12 4.68 7.85

Views Recipe

Finder Page

1

15

4.28

0.75

0.31 0.58 -1.32 1.12 2.67 5.87

2 15 3.20 0.99 2.32 0.58 6.36 1.12 1.07 5.33

197

3 15 5.20 0.73 -0.28 0.58 -0.78 1.12 3.63 6.77

Visits Virtual

Tour

1

15

1.67

0.37

0.68

0.58

-0.85

1.12

0.87

2.47

2 15 1.20 0.24 1.92 0.58 5.54 1.12 0.68 1.72

3 15 1.33 0.16 -0.31 0.58 -0.40 1.12 0.99 1.68

Selects Filter

Option

1

15

1.20

0.43

1.26 0.58 0.58 1.12 0.28 2.12

2 15 0.87 0.55 2.49 0.58 5.32 1.12 0.31 2.05

3 15 1.13 0.42 2.10 0.58 5.09 1.12 0.22 2.04

Takes a Note 1 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

2 15 0.60 0.53 3.78 0.58 14.42 1.12 0.54 1.74

3 15 1.53 0.82 2.15 0.58 3.68 1.12 0.23 3.30

Virtual Tour

Arrow

1

15

2.00

0.81 2.01 0.58 4.34 1.12 0.27 3.73

2 15 1.53 0.62 1.16 0.58 -0.44 1.12 0.20 2.87

3 15 2.40 1.05 1.69 0.58 1.49 1.12 0.15 4.65

Views Photos

1

15

5.73

1.41

1.07

0.58

2.14

1.12

2.72

8.75

2 15 8.07 2.63 1.76 0.58 3.75 1.12 2.42 13.71

3 14 6.64 2.40 1.18 0.60 0.08 1.15 1.47 11.82

Comfort Level 1 15 5.07 0.32 -0.14 0.58 -0.36 1.12 4.39 5.74

2 15 6.07 0.25 -0.71 0.58 -0.33 1.12 5.53 6.60

3 15 6.27 0.23 -1.32 0.58 1.82 1.12 5.78 6.76

*Note: Group 1 = Concurrent, 2 = Retrospective, 3 = Virtual Revisit

198

Tests of Normality

Kolmogorov-Smirnov(a) Shapiro-Wilk

Group Statistic df Sig. Statistic df Sig.

Planning 1 0.39 15 .000 0.66 15 .000

2 0.18 15 .191 0.88 15 .044

3 0.30 15 .001 0.80 15 .004

Connecting 1 0.20 15 .101 0.94 15 .392

2 0.12 15 .200* 0.96 15 .744

3 0.15 15 .200* 0.95 15 .513

Reasoning 1 0.31 15 .000 0.80 15 .004

2 0.14 15 .200* 0.94 15 .328

3 0.14 15 .200* 0.96 15 .722

Reflecting 1 0.13 15 .200* 0.96 15 .628

2 0.12 15 .200* 0.98 15 .950

3 0.15 15 .200* 0.89 15 .063

Website

Content

1

0.10

15

.200*

0.95

15

.504

2 0.12 15 .200* 0.93 15 .287

3 0.13 15 .200* 0.95 15 .545

User

Experience

1

0.16

15

.200*

0.89

15

.065

2 0.17 15 .200* 0.88 15 .050

3 0.14 15 .200* 0.96 15 .670

Understanding 1 0.20 15 .119 0.92 15 .219

2 0.28 15 .002 0.73 15 .000

3 0.20 15 .097 0.86 15 .024

Diversion 1 0.24 15 .023 0.86 15 .025

2 0.51 15 .000 0.41 15 .000

3 0.23 15 .029 0.84 15 .011

Procedural

Behaviour

1

0.17

15

.200*

0.96

15

.607

2 0.13 15 .200* 0.94 15 .381

3 0.14 15 .200* 0.97 15 .875

Website

Features

1

0.15

15

.200*

.946

15

.463

2 0.30 15 .001 .768 15 .001

3 0.27 15 .004 .739 15 .001

Reading 1 0.14 15 .200* .964 15 .762

2 - - - - - -

3 0.48 15 .000 .520 15 .000

Word Count 1 0.13 15 .200* 0.95 15 0.48

2 0.17 15 .200* 0.95 15 0.50

3 0.18 15 .200* 0.93 15 0.23

Number of

Thought Units

1

0.12

15

.200*

.971

15

.878

2 0.15 15 .200* .939 15 .366

3 0.17 15 .200* .934 15 .311

Time on Each

Page

1

0.17

15

.200*

.971

15

.869

2 0.09 15 .200* .969 15 .842

3 0.19 15 .137 .935 15 .326

Number of

Pages Visited

1

0.09

15

.200*

.983

15

.987

2 0.15 15 .200* .924 15 .219

199

3 0.13 15 .200* .967 15 .818

Food Group

Pages

1

0.15

15

.200*

.903

15

.106

2 0.23 15 .026 .903 15 .107

3 0.15 15 .200* .956 15 .627

Content Pages 1 0.27 15 .005 .793 15 .003

2 0.30 15 .001 .794 15 .003

3 0.39 15 .000 .630 15 .000

Visits to

Homepage

1

0.27

15 .005 .900 15 .096

2 0.39 15 .000 .630 15 .000

3 .304 15 .001 .748 15 .001

Time Viewing

Videos

1

0.23

15

.029

.851

15

.018

2 0.17 15 .200* .896 15 .466

3 0.13 15 .200* .946 15 .466

Number of

Videos

Stopped

1

0.22

15

.053

.927

15

.246

2 0.23 15 .033 .790 15 .003

3 0.24 15 .016 .717 15 .000

Enters Search

Term

1

0.32

15 .000 .657 15 .000

2 0.42 15 .000 .557 15 .000

3 0.44 15 .000 .596 15 .000

Maneuvers

Virtual Tour

1

0.44 15 .000 .616 15 .000

2 0.43 15 .000 .403 15 .000

3 0.47 15 .000 0.53 15 .000

Visits How-

To Videos

1

0.22

15 .061 .805 15 .004

2 0.35 15 .000 .643 15 .000

3 0.36 15 .000 .716 15 .000

Visits Outside

Link

1

0.25 15 .012 .730 15 .001

2 0.29 15 .001 .734 15 .001

3 .333 15 .000 .728 15 .001

Views Recipe

Page

1

0.13

15

.200*

.962

15

.723

2 0.25 15 .015 .850 15 .018

3 0.14 15 .200* .937 15 .343

Views Recipe

Finder Page

1

0.18 15 .186 .920 15 .190

2 0.32 15 .000 .735 15 .001

3 0.17 15 .200* .934 15 .312

Visits Virtual

Tour

1

0.28

15

.003

.845

15

.015

2 0.31 15 .001 .712 15 .001

3 0.31 15 .001 .766 15 .001

Selects Filter

Option

1

0.30 15 .001 .766 15 .001

2 0.46 15 .000 .476 15 .000

3 0.27 15 .005 .723 15 .000

200

Takes a Note 1 - - - - - -

2 0.48 15 .000 .332 15 .000

3 0.42 15 .000 .561 15 .000

Virtual Tour

Arrow

1

0.27 15 .004 .711 15 .000

2 0.40 15 .000 .660 15 .000

3 0.34 15 .000 .649 15 .000

Views Photos

1

0.21

15

.081

.842

15

.013

2 0.21 15 .062 .790 15 .003

3 0.27 14 .006 .757 14 .002

Comfort

Level

1

0.21

15 .069 .922 15 .205

2 0.23 15 .026 .844 15 .014

3 0.26 15 .006 .775 15 .002

* This is a lower bound of the true significance.

(a) Lilliefors Significance Correction

*Note: Group 1 = Concurrent, 2 = Retrospective, 3 = Virtual Revisit

201

Test of Homogeneity of Variance

Levene Statistic df1 df2 Sig.

Planning 14.36 2 42 .000

Connecting 3.44 2 42 .041

Reasoning 6.04 2 42 .005

Reflecting 2.11 2 42 .134

Website Content 5.02 2 42 .011

User Experience 2.43 2 42 .101

Diversion 2.98 2 42 .062

Procedural Behaviour 2.20 2 42 .124

Website Features 1.86 2 42 .169

Reading 21.04 2 42 .000

Word Count 0.97 2 42 .389

Number of Thought Units 4.99 2 42 .011

Time on Each Page 2.59 2 42 .087

Number of Pages Visited 3.870 2 42 .029

Food Groups Pages 7.46 2 42 .002

Content Pages 19.83 2 42 .000

Visits to Homepage 3.32 2 42 .046

Time Viewing Videos 4.61 2 42 .015

Number of Videos Stopped 0.96 2 42 .393

Enters Search Term 4.38 2 42 .019

Maneuvers Virtual Tour 0.19 2 42 .826

Visits How-To Videos 1.05 2 42 .358

Visits Outside Link 3.08 2 42 .057

Views Recipe Page .382 2 42 .685

Views Recipe Finder Page .053 2 42 .948

Visits Virtual Tour 5.39 2 42 .008

Selects Filter Option 0.18 2 42 .837

Takes a Note 7.31 2 42 .002

Virtual Tour Arrow 1.01 2 42 .374

Views Photos 2.02 2 42 .146

Comfort Level 0.39 2 42 .677

202

Appendix C-1

Initial List of Categories for Study 2

Initial List of Categories with Frequency Counts

Category

Frequency

Count

Absorbing or mentally taking in information 20

Accessing information with ease 1

Attending to the spatial organization of the classroom 6

Being curious about website features or content 9

Comparing content to own teaching practice 49

Connecting content to special ed. classroom contexts 4

Connecting navigation and website content to current grade 46

Connecting navigational experience to professional learning 23

Connecting to teacher's college 1

Connecting website content to future planning and

classroom set up

39

Connecting website content to literacy knowledge and

understanding

24

Connecting website content to prior knowledge 6

Connecting website to other resources 5

Constructing new ideas for future practice-constructing

knowledge

21

Continuing navigation 1

Critiquing website content and organization 15

Deciding to explore information in-depth 2

Describing challenges in implementing literacy program 5

Describing multitasking behaviour 4

Describing website content being viewed 3

Discontinuing an action or behaviour 20

Encoding information for future retrieval 31

Explaining online behaviour 83

Explaining what and why content was read 33

Exploring information 14

Expressing awareness of study task 17

Expressing opinion about website content 71

Expressing opinion about website features and organization 34

Feeling confident during navigation 4

Feeling confused 16

Feeling frustrated about navigation 20

Feeling inspired excited motivated 13

203

Feeling more advanced or superior than website content 2

Feeling overwhelmed 7

Focusing on PD initiatives 1

Hoping for more information 24

Identifying goals of instructional practice or best practices

in literacy

71

Identifying new knowledge new information 19

Identifying or recognizing familiar information 27

Identifying or referring to own learning style 35

Identifying professional goals 41

Identifying self as a learner 9

Learning by observing, watching 5

Looking for representation 2

Looking for resources for specific literacy concept 1

Maintaining current knowledge and understanding of

literacy practices

4

Making a navigational decision 52

Making a navigational decision based on own children's

struggles

2

Narrowing the focus down 6

Navigating with ease 2

Noticing appealing and eye catching features 24

Noticing new information 2

Noticing website errors 1

Observing and reflecting on own actions and non-actions 31

Offering improvements to website content 6

Planning navigation-thinking ahead about options,

navigational strategies

7

Reading to find new or interesting information 2

Realizing or discovering how to use certain website features 9

Referring to future use of website 6

Referring to limitations in own professional development

OR how technology can be used to off set PD limitations

5

Referring to literacy terms and or practices 4

Reflecting on general use of websites-online learning style 33

Reflecting on program 3

Reflecting on students' needs and goals 13

Relating content to a stage or level of teaching 27

Relating content to real-life or authentic experiences of

classroom practice

12

Relating or associating website content and pages to each

other

4

204

Revisiting information to refresh professional knowledge 15

Sharing a learning experience of visiting colleagues'

classrooms

5

Tailoring information to teaching practice 10

Troubleshooting strategies 6

Trying to understand and use website features and

organization

31

Using background information and prior knowledge to

understand content

3

Wanting to understand the credibility of the website 8

Willing to move outside comfort zone 1

Wondering about activity extensions 2

Wondering about classroom contexts 4

205

Appendix C-2

Sample of Memos Written During Analysis

06/15/2014

Initial Coding:

Coding first five transcripts (V01-V05) without looking back and regrouping or reorganizing

codes. All codes are being laid out on the table and new codes are being added as they emerge

from the transcripts. Using verbs to begin codes (comparing, connecting, constructing)-what

Charmaz refers to as "activating coding"-coding with words that reflect action

06/30/2014

Categories:

Categories may include broad areas, information directly relating to website content and

organization and information relating to cognition and learning

07/05/2014

Teacher Knowledge and Understanding of Instructional Practice:

Learning about teachers' understandings of what they view as best practices in literacy. They are

often expressing research based practices in relation to what they see on the website and their

current grade. Also relating these best practices to their professional goals and school wide

goals.

08/23/2014

Themes/Model:

Analysis of think aloud transcripts + interviews + questionnaire data = understanding of

elementary teachers' motivations and beliefs in literacy planning and practice

09/20/2014

Triangulation:

Use initial codes from think aloud analysis to code/analyze interviews and questionnaire

(triangulation)-while coding keep an open mind about new categories

09/29/2014

Use of Internet for Professional Practice:

From Questionnaire: All teachers in VR group reported that they use the Internet for their

professional practice--> motivation to seek out new and relevant material and also to incorporate

technology into their daily practice, with their students.

206

Appendix C-3

Themes and Descriptions Sent to Participants Involved in Member Check

Potential Themes

THEMES IDENTIFIED:

1. Focusing on student needs and goals

Regardless of experience and background, teachers approached the website with the

common goal of supporting their students

2. Analyzing and evaluating information in relation to their own practice

Connecting, analyzing Analysing and deconstructing content in relation to experience and

background

Reasoning-most common reason for making a decision-connecting to practice Approaching

information from various backgrounds and perspectives (spec. ed., music, drama, grade

levels)

3. Encoding information for future retrieval

Use of memory aids and strategies for future retrieval (note taking, connecting to practice)

4. Intentions for practice and professional learning goals

Intentions for practice planning, connecting

Professional development, learning goals, professional knowledge

5. Reflecting on their learning and use of website

Identifying learning styles (e.g., visual learner)

Resolving confusion with self-questioning, troubleshooting strategies

Monitoring own cognitive processes during task

Individual learning styles (individual differences in web seeking behaviours)

6. Valuing best practices in literacy and teaching

Attitudes towards teaching and learning, knowledge about literacy instruction

Beliefs about literacy instruction and overall best teaching practices

7. Feeling motivated for learning and applying new knowledge Intrinsic motivation-being curious, interested, intrigued, motivated, driven by interest

Self-motivated, self-determined

207

Appendix C-4

Participants’ ‘Member Check’ Comments

Potential Themes

THEMES IDENTIFIED:

Focusing on student needs and goals

Regardless of experience and background, teachers approached the website with the

common goal of supporting their students

Agreed! I think this relates particularly to student engagement and also having a classroom

set-up that encourages the types of learning and interactions that we would like to

encourage.

Agree. I want to create the best program possible for my students. Yes, thinking, “what does this website have to offer me and my students? Is there anything

here that I don’t already do that I could/should be doing to help support my students in their

learning and development as readers and writers?”

Yes

Focusing on student needs AND specific curriculum expectations

I would definitely agree with this – when I search, the most important thing I would initially

be thinking about is, is this relevant and applicable to my students (age/grade/level) and will

they benefit from it?

This makes sense to me as teacher’s first goal is to support their students. It’s the first check

of how I evaluate any and all new resources.

I would hope and assume that this is a given for teachers. Do we do it without even thinking?

Analyzing and evaluating information in relation to their own practice

Connecting, analyzing yes

Analyzing and deconstructing content in relation to experience and background I especially

like seeing how teachers in the videos online introduce a topic (what language they use, how

they help the students make personal connections and engage them in the lessons)

Reasoning-most common reason for making a decision-connecting to practice I am not quite

sure that I understand this

Approaching information from various backgrounds and perspectives (spec. ed., music,

drama, grade levels)

Yes! I noticed in my transcript that I mentioned many times how I found things that I already

do. Of course I was looking for new ideas, but I think that coming across things that I

already know/do makes me think that the web site (or book or whatever) is a good fit for me

and for my teaching philosophy.

Agree. I definitely thought about how I could use some of the ideas from the site.

And deciding whether I agree or disagree with an approach. Deciding whether I think a

particular approach would work for me with my current group of students.

Yes – I was very much looking to connect to my grade level.

YES!

208

Agree – I’m always looking for things to enhance my own practice as well – so looking for

ways and ideas that might be ‘better than’ what I am already doing

Again, this makes sense to me as something must be practical and applicable to the class

you are teaching. You are first and foremost responsible for your students. Your students

and their particular understanding and needs are what drives you to explore new resources.

Content that makes sense in terms of your current practice is the easiest to utilize. It’s sort

of like ZPD. You’re most likely to move your practice to the next logical step - not jump to a

huge new overall of practice from one new resource.

Information that can be used across the curriculum and integrated into more than one

area/theme.

Encoding information for future retrieval

Use of memory aids and strategies for future retrieval (note taking, connecting to practice)

Yes. I wonder if it would be a good idea for the site to have something like a “Favourites”?

Something that lets you save or bookmark things you’d like to return to later?

Agree. I took notes.

And thinking if I have the professional knowledge, experience, time, skills and/or resources

to utilize a particular approach or to structure lessons in a new way.

Yes – I was copying links to a word document

yes – specifically ways to display information so that it is student friendly (on charts or

graphic organizers, bulletin board, in their notebooks)

Lol. This sounds like a great idea. I’m terrible at remembering where I’ve written notes.

If something really catches my eye or attention on a website, that will help me remember it

and go back to it.

Intentions for practice and professional learning goals

Intentions for practice planning, connecting organizing the ideas or concepts in a logical

order

Professional development, learning goals, professional knowledge

Always feels good to be adding to these things!

Agree. I intend to use this site to support my professional practice.

Yes, I think about future professional development, my annual learning plan/goals for

growth.

Yes

Similar to above – I agree and feel that part of my motivation is always looking for ways to

improve upon my practice and see what ideas and aids I can gather that might be better

than what I am currently doing

I think that second to student need your own professional learning goals connect to how and

when you’ll approach new resources. You have to have the impetus to want to change your

practice before you are in a position to learn / grow as an educator. Professional knowledge

connects to ZPD.

Pushing myself out of my comfort zone. Trying a new idea/method I’m not so familiar with.

Visualize myself as my students being asked to try something new/scary. That’s how growth

occurs.

209

Reflecting on their learning and use of website

Identifying learning styles (e.g., visual learner) yes

Resolving confusion with self-questioning, troubleshooting strategies yes – videos especially

show possible questions or confusions students might have with a topic

Monitoring own cognitive processes during task yes – helpful with breaking down what to

me is a simple concept but still has to be explained to students (ie: sharing what I am

thinking at various points either while reading a book or doing an experiment)

Individual learning styles (individual differences in web seeking behaviours)

Not totally sure about this one. Is this just in the sense of metacognition while using the web

site? Like thinking about how it works and how I’m using it?

Agree. This was largely a factor of the self-talk involved in the study.

Noticing my frustration or sense of calm when a sight is easy to navigate. When I click on

something and it opens instantly and I can begin to read/scan right away and not waste time

waiting. I am thinking about how the information appears visually, what I notice first, if

there are visual distractions, if it is easy to read, etc.

Yes

I think that the recorded think aloud really reinforced this.

Site was successful if return to it. Must be easy to troubleshoot.

Valuing best practices in literacy and teaching

Attitudes towards teaching and learning, knowledge about literacy instruction – especially

key literacy concepts and styles and how to teach them

Beliefs about literacy instruction and overall best teaching practices yes

Again, I think we gravitate toward things that support our existing beliefs, but it’s also great

to have something of a challenge of a piece of new learning embedded, too, so that we’re

always growing.

Agree. I told a number of colleagues about the website.

This is a big one. I spend very little time reading about instructional strategies or

approaches that I already know about. I like to quickly confirm that, yes, I am doing that in

my classroom and then read on to find out how I can do it even better or go further with it.

To me, this is similar to themes 2 and 4 again in the sense that I am motivated to improve

upon my practice and am seeking information that might help me to do so

I think your own beliefs colour everything that you do. Most people work from a positive

place of wanting the best for their students. People evaluate their understanding in terms of

what they think is best for their students, fits best with their own understanding.

Integrated into everything every day. Providing high quality materials to students.

Feeling motivated for learning and applying new knowledge

Intrinsic motivation-being curious, interested, intrigued, motivated, driven by interest also

helpful to be introduced to new material/books to teach expectations or concepts

Self-motivated, self-determined yes

Yes. I was definitely interested in exploring this site and curious about learning more!

Agree. I was excited to learn about a useful site like this.

210

Yes – I agree, curiosity, intrigue, and constantly wanting to keep my program fresh and

learn new strategies is a huge motivator.

Definitely!

Interest would certainly drive me to explore a new resource. The beauty of this website is

content is categorized / organized in so many different ways that you can explore on your

own terms.

If I’m not interested, will my students be?

General comments:

I also compare (in all honesty) my thinking around how best to teach a particular concept

or, more broadly, plan for a unit of instruction to how others are doing it. I think about how I would have categorized the information or grouped the posts to be more

in line with my sense of organization or long range plans. I am looking for posts that relate directly to overall curriculum expectations that I am

planning to teach. I am looking for ideas about how to integrate literacy across the content areas. I am looking for shortcuts so that I can work smarter, not harder. I am looking for current best practice and innovative ways of teaching concepts.

211

Appendix C-5

Frequency of Themes across Participants

Frequency Distribution of Themes Following Coding of Think Aloud Transcripts

Participant

Perception of

Professional

Learning

Focusing on

Students

Needs and

Instructional

Goals

Individual

Differences

Navigating a

Professional

Development

Website

Evaluating

Information

Encoding

Information

for Future

Retrieval

Continued

Professional

Learning

Intentions

for

Practice

Reflecting

on

Learning

V01 11 7 1 13 12 3 6 3 16

V02 3 6 8 2 10 12 1 18 10

V03 12 13 4 4 14 9 5 3 19

V04 23 4 4 7 8 1 5 7 17

V05 11 8

11 5 1 2 5 13

V06 26 4 1 2 12 1 9 1 6

V07 21 5 1 12 7 11 7 14 17

V08 20 5 4 15 12 1 5 4 26

V09 23 8 2 7 16 4 9 15 11

V10 11 5 3 11 18 5 6 2 13

V11 18 9 4 4 9 4 10 16 10

V12 17 13 4 11 12 9 11 5 23

V13 18 11 7 7 22 4 5 5 14

V14 8 9 9 8 11 1 8 5 13

V15 15 16 5 3 19 9 9 8 13

Total 237 123 57 117 187 75 98 111 221

212

Frequency Distribution of Themes Following Coding of All Three Data Sources

Participant

Perception

of

Professional

Learning

Focusing on

Students

Needs and

Instructional

Goals

Individual

Differences

Navigating a

Professional

Development

Website

Evaluating

Information

Encoding

Information

for Future

Retrieval

Continued

Professional

Learning

Intentions for

Practice

Reflecting on

Learning

V01 14 7 4 13 15 3 7 3 17

V02 6 7 12 3 12 12 3 22 10

V03 13 13 6 5 15 11 10 7 21

V04 31 4 5 7 10 1 6 7 18

V05 13 9 4 11 7 1 7 7 14

V06 30 8 5 2 13 1 9 1 8

V07 25 5 6 15 8 11 8 15 19

V08 24 5 7 15 13 1 6 4 29

V09 25 9 5 7 16 5 10 16 13

V10 14 5 7 13 18 5 7 3 17

V11 21 10 7 4 9 4 11 16 11

V12 26 14 5 11 13 10 13 6 25

V13 21 13 9 8 22 4 8 6 15

V14 11 10 9 9 11 1 10 6 16

V15 18 16 6 4 20 9 10 8 15

Total 292 135 97 127 202 79 125 127 248

213

Appendix C-6

Samples from Audit Trail

Raw Data

Think Aloud Transcript (Sample)

V07

January 8, 2014

Transcript

Okay, the first thing I was looking at was who the author of the website was, and so I

clicked on the link at the bottom and I saw that it was OISE and since I’m familiar with that

institution I didn’t feel like I needed to explore that website anymore. And then I went up

and started to read about the Melissa Institute website, and then I wanted to learn a little bit

more about it by reading what was on the main page.

And then I was interested in who was sort of behind the institution so I clicked on the about

us button and read a little bit about the history about why the institution was set up. And

then I scrolled down to read about the members of the board. And at this point what I was

also looking at, I was hoping to find something about how the institute was funded, but

given my time constrain I didn’t want to spend too much time doing that. So I just quickly

looked at who was on the board of directors and so I wanted to see if there was sort of a

balance of educators and I was noticing that there were lawyers and people from the

medical scientific community and just wanted to keep that in the back of my mind in terms

of the credibility of the website...

214

Interview Transcript

V09

March 12, 2014

Interview

I: What were your general feelings while you explored the Balanced Literacy Diet website?

V09: I think it was really well developed. I liked all the visuals, the fact that there’s some video

included for how to present concepts to students. But also the bulletins and how to put it on

display for students and what came out of their work. That really grabbed my attention and

images always seems to work rather than looking at a whole bunch of writing. For me it was

really broken down really well too, but the image really had to capture me to look at the reading.

I: On a scale of 1 to 7, 1 indicating very uncomfortable and 7 indicating very comfortable, how

did you feel about thinking aloud while you watched a screen recording of your navigation?

V09: Probably a 6, I was pretty comfortable. At certain times I was trying to figure out where I

was going. In my mind I was trying to figure out the website initially and I was all over the place

so I wasn’t really sure what I was thinking. I was just trying to get the concept and get to an

image and get to something new. And sometimes I would go back and it would be messy as to

where to click next but I mean I’m not the smoothest when it comes to transitioning on the

computer with something new. But overall I think it was all right cause I did get through a lot of

it. I did do a little repeating of it though.

I: What did you find challenging while you explored the Balanced Literacy Diet website?

V09: Well like I said initially food groups to me, that part, that title, it just kept bringing me back

to health, but then when I clicked on it, it was very clear that they…a couple times I looked at it

and I wasn’t sure what the tab was about and so I went back to it but overall it was well

organized and it made sense and there were lots of drop down menus to go to.

I: What features of the Balanced Literacy Diet website did you find most appealing?

V09: The ideas and the display ideas and the presentation ideas for the students. The talking

through videos up to a point but I don’t spend as much time on that as I do normally. I do like

sometimes if it’s a lesson that I’m really interested in doing it’s nice to see it modeled. It’s also

nice to see it explained out in writing but the modeling actually gives it a good idea of what the

idea would look like… and I think normally when I look at sites like this, if I see something that

interests me I would be jotting down little notes to remind myself so poster-wise, key concepts

and how it was laid out so I can copy that at some point.

I: What other features would you have liked to view/explore in the Balanced Literacy Diet

website?

V09: I guess something that might have been nice is to have some sort of activities linked to it.

So I don’t know that this site had it but if there’s a concept that’s trying to be taught to try and

even have an interactive game that students can then use on the smart board. I didn’t get the

sense that there was a lot of that or any on this site. Or even the links, I didn’t look at any of the

links cause I didn’t want to leave the website, but even if it could link up to other smart board

activities I think.

I: Is there anything else you would like to share about online professional resources?

V09: I guess the one frustration with using the internet is the computer system in the school

system are pretty slow and not terribly well designed. We have a lot of problems at our school

with technology so, it’s not the most reliable of getting information to the students. You can plan

a whole lesson and then the smart board might be taken out for a staff meeting or it might

disappear somewhere else and it becomes a bit of a problem and so what we did have previously

is one smart board traveling to all the grade classes. So we have sixteen grade two classes and

215

one smart board would rotate around to the different classrooms and you would get it for a

couple weeks. But just being ready for it and having the lessons and timing it with the program

and trying to get through especially at the beginning of the year and report card time, people just

want to get through material. Most teachers don’t want the smart board in their room taking up

space…it’s not always the most effective way of teaching. And the speed at school makes it

frustrating. So I do a lot more research on websites at home in my own time.

I: Is there anything else you would like to share about your experience of thinking aloud while

exploring The Balanced Literacy Diet website?

V09: I tend to think aloud a lot even when there’s no one around so it’s not that difficult I think.

216

Screen Shot of Participant Screen Recording

217

NVivo Screen Shots

Sample of Nine Major Themes and Sub-Themes

Sample of Coding Process

218

Theory Construction

Model 1

Model 2

219

Model 3

Causal Conditions

Beliefs about

professional learning

and literacy

instruction

Focusing on Student

Needs and Goals

Individual Differences

Central Phenomenon

Navigating a Professional

Development Website

Strategies

Analyzing and

Evaluating

Encoding Information

for Future Retrieval

Potential Outcomes

Reflecting on Learning

Continued Professional

Learning

Intentions for Practice

affect leads to

brings about

Elementary Teachers’ Learning Experiences as they

Navigated a Professional Development Literacy Website

220

Final Model

Elementary Teachers’ Learning Experiences as they Navigate a

Professional Development Website

Conditions Affecting Central Phenomenon Strategies for Navigating

Navigation a Professional Development

Website

Theme 1:

Perception of

Professional Learning

Theme 4:

Navigating a

Professional

Development Website

Theme 5:

Evaluating

Information

Theme 2:

Focusing on Student

Needs and Instructional

Goals

Theme 6:

Encoding

Information for

Future Retrieval

Theme 3:

Individual

Differences

Potential Outcomes of Navigating a Professional Development Website

Theme 7:

Reflecting on Learning

Theme 8:

Continued Professional

Learning

Theme 9:

Intentions for Practice