Reliability and Preliminary Use of a Rubric to AssessPre-Kindergarten Teachers’ Video Uses

Bridget A. Walsh • Heidi Cromer • Daniel J. Weigel •

Leah Sanders

Published online: 6 November 2012

� Springer Science+Business Media New York 2012

Abstract The presence and use of new technologies in

early childhood settings are rapidly increasing. One tech-

nology tool used in early childhood settings is monthly

DVD classroom newsletters, yet there is a lack of assess-

ments to support pre-kindergarten teachers’ uses of such

DVD newsletter technology—in general and in specific.

The present study helps to fill this gap by developing and

testing a revised rubric to evaluate the quality of monthly

DVD classroom newsletters. Results indicate that the

revised Monthly DVD Classroom Newsletter-Rubric

exhibited good overall reliability. We suggest that the use

of a rubric to assess pre-kindergarten teacher-created

monthly DVD classroom newsletters supports teachers’

decision making about technology uses and professional

development.

Keywords Pre-kindergarten teachers � Rubric �Technology � Video

Introduction

A variety of methods have been used by teachers to

encourage parent involvement, including traditional paper

newsletters and parent-teacher conferences (Epstein 1995).

One nontraditional and promising approach is monthly

DVD classroom newsletters. Monthly DVD classroom

newsletters help parents interact with their child’s class-

room by using video technology to record segments from

the classroom and allow teachers to verbally provide tea-

cher messages and announcements and information about

children’s learning, and more (Sanchez et al. 2009; San-

chez and Walsh 2010). These clips are copied onto a DVD

and sent home to communicate information to parents

regarding their child’s schooling (Sanchez et al. 2009)

typically with each family getting one copy of a monthly

DVD classroom newsletter. However, there is no evalua-

tion of the quality of these DVDs. The purpose of this

paper is to examine a researcher-developed rubric with

teacher input or the Monthly DVD Classroom Newsletter

Rubric (MDCN-R), in terms of content validity and inter-

rater reliability. Furthermore, this paper compares the dif-

ferences of the reported MDCN-R quality results between

monthly DVD classroom newsletters produced during the

first year (Year 1) and second year (Year 2) at participating

pre-kindergarten sites.

Technology in early childhood classrooms is supported

by professional groups (e.g., National Association for the

Education of Young Children & the Fred Rogers Center for

Early Learning and Children’s Media 2012) and DVD

newsletters have the potential to be a very effective form of

communication for families (Sanchez et al. 2009, 2011;

Walsh et al. 2008). Some other newer technological tools

presently integrated into early childhood settings run the

gamut from smartphones to record video for daily blogs

and online portfolios for parents to view (Parnell and

Bartlett 2012). Using video to capture children’s learning

in the classroom can be challenging and requires skill but it

is possible for teachers to get outstanding footage to cap-

ture teaching and learning in the classroom (Luckenbill

2012) and to use the video to communicate with parents to

nourish children’s development and learning (Gimbert and

Cristol 2004). The present work is anchored in using DVD

classroom newsletters as a mode to promote classroom-

to-home communication (Sanchez and Walsh 2010; San-

chez et al. 2011).

B. A. Walsh (&) � H. Cromer � D. J. Weigel � L. Sanders

University of Nevada, Reno, Reno, NV, USA

e-mail: bridgetw@unr.edu

123

Early Childhood Educ J (2013) 41:325–337

DOI 10.1007/s10643-012-0559-0

The next section of this paper will provide a brief

overview of the use of a specific technology tool: monthly

DVD classroom newsletters. This includes discussion of

monthly DVD classroom newsletters as they apply to early

childhood settings. The final section focuses on an

assessment method for examining the use of these DVD

newsletters and draws support from existing assessments.

Monthly DVD Classroom Newsletters in Early

Childhood Settings

For 2 years, three to four pre-kindergarten teachers in a

Nevada school district with the support of a teacher coach,

teacher coordinator, and a University faculty member

created monthly DVD classroom newsletters to promote

classroom-to-home communication and reinforce chil-

dren’s classroom learning. The video clips show children

and teachers engaged in teaching and learning scenes that

range from solving math problems to talking about feel-

ings. For example, a teacher showcased parts of an inter-

active storybook reading about feelings. Children were

shown drawing faces that represented how they were

feeling. The class created a graph in which the children put

their pictures under a feeling (e.g., excited, sad, happy,

scared, shy). Then, the children counted how many faces

were under each feeling and concluded which feelings had

the most or least children. Finally, children wrote out the

feeling of the face they drew. There were pictures and

labels at each table to facilitate the activity. For example,

there was a picture of a happy face with the word happy

written under the picture. The teacher was captured posing

individual questions to children about what they drew and

what they were writing. The segment was edited to

approximately 4 min and the editing included the addition

of soft background music. See Sanchez and Walsh (2010),

Walsh et al. (2008) for a description of other possible DVD

scenes.

While almost all of the DVD content was spoken, they

also included bilingual (Spanish and English) written

content in the form of sentences on the bottom of the screen

that occasionally appeared to underscore messages during

teaching and learning segments (Sanchez et al. 2011). The

DVDs also featured classroom schedules and announce-

ments about upcoming events. The bilingual menu allowed

the families to view the DVDs in English, Spanish, or both.

The content length of each DVDs is approximately 15 min

(see Calabrese 2006; Sanchez and Walsh 2010; Sanchez

et al. 2011; Walsh et al. 2008).

Critics of this project state that it is essentially one-way

communication between classroom and home. While we

concur with that statement, we add that it gives all parents

opportunities to be involved, ideas to help their child learn

at home, and a window into their child’s school day. Many

parents have yet to become involved in their children’s

education due to barriers, such as linguistic and cultural

obstacles (Sanchez et al. 2009). For example, one challenge

that English Learning (EL) parents face is that they des-

perately want to learn English (Good et al. 2011). Under-

standing and acknowledging challenges, such as EL

parents’ desire to want to learn English, is critical to

encouraging parent involvement (Panferove 2010). Many

existing parent involvement strategies do not fit all parents

because of the language and literacy levels required. DVDs

remove the need for parents to read.

Rubric to Assess Quality of Monthly DVD Classroom

Newsletters

Many assessments surrounding technology have been

developed with K-12 teachers (Christensen and Knezek

2001). The extant literature indicates some existing

assessments and evaluation tools regarding classroom

technology designed for early childhood teachers. These

include: Teacher Technology Assessment (Bewick and

Kostelnick 2004), the Early Childhood Educational Tech-

nology Evaluation Toolkit (McManis and Parks 2011), and

the Self-Reflection Framework (Snider and Hirschy 2009).

Please see Table 1 for more information about these

existing assessments, their designed usage, and descrip-

tions of their purposes. However, the availability of tech-

nology assessments for early childhood professionals is

limited. Because of this lack of technology assessments

with pre-kindergarten teachers, a rubric was developed,

validated, and then revised for the Monthly DVD Class-

room Newsletter project.

Monthly DVD Classroom Newsletter-Rubric (MDCN-R)

Research regarding the utilization of a rubric to evaluate

the quality of monthly DVD classroom newsletters does

not exist, thus the present study seeks to help close that

gap. We developed and validated a rubric, the Monthly

DVD Classroom Newsletter-Rubric, to assess develop-

mentally appropriate practice (Copple and Bredekamp

2009) and technical components of the DVDs.

Rubrics have been well-documented in the extant liter-

ature to assess a variety of topics that incorporate the

National Association for the Education of Young Children

(NAEYC) standards. These guidelines or standards are

aimed to promote developmentally appropriate practice,

which fosters young children’s learning and development

(Copple and Bredekamp 2009). Rubrics have examined

video recordings in instruction to determine the extent of

the alignment with the NAEYC standards (Thornton and

Broughton 2010), pre-service teachers’ design of the

environment and reflection with the incorporation of

326 Early Childhood Educ J (2013) 41:325–337

123

NAEYC standards (Critchosin 2009), and early childhood

candidates NAEYC standards-based assignments and

activities (Autry et al. 2009). The inclusion of NAEYC

guidelines is important for the assessment of early child-

hood candidates or in-service teachers’ effective practice

and growth.

Rubrics that have focused on aspects of technology have

included video recordings of early childhood pre-service

teaching instruction (Thornton and Broughton 2010) and

pre-service and in-service video teaching projects (Swee-

der 2007). Hall and Hudson (2006) designed a rubric to

assess undergraduate elementary education students’ video

production. The assessment included technical items, such

as shot composition, sound quality, transitions, camera

angles, editing, and text (Hall and Hudson 2006). When

assessing teachers on a technology project, the technical

aspect of a rubric is clearly important.

The authors developed the MDCN-R to assess the

quality of teacher-created DVDs in terms of technical

aspects and developmentally appropriate practice. The

rubric was developed through an iterative process between

school personnel, teachers, and university researchers. The

rubric was designed for researchers, project directors, or

outside evaluators to assess the quality of the DVDs. Initial

discussions revealed that a technically sound DVD and

developmentally appropriate teaching and learning scenes

in the DVD were most important to capture in the rubric.

The items within each of the two sections were developed

over time through discussions about existing monthly

DVDs. The general structure of the MDCN-R involves a

scoring scale in which each item in the two sections can

be scored as the following: insufficient (1 point), emergent

(2 points), proficient (3 points), or distinguished (4 points).

The aforementioned scoring labels and points have been

adopted from Ledoux and McHenry’s (2006) rubric. The

two sections of the MDCN-R are described in more detail

in the below sections.

Technical Section of MDCN-R

The first section of the original MDCN-R contained eight

items related to the visual and audio quality of the DVD

itself including length of DVD, written language conven-

tions (grammar, spelling, and/or vocabulary errors), audio,

visual definition (resolution), visual definition (camera

techniques), lighting, and menu functionality. Each of these

items focuses on the technical functionality of the DVD;

for example, the audio item focuses on the clarity of the

audio in terms of audio levels and balance with background

noise. In essence, the Technical Section focuses on the

mechanical aspects of the DVD and does not address

content of the DVD beyond consideration of the visual

content itself in terms of use of scenes to clearly show an

intended message and purpose.

Developmentally Appropriate Practice Section of MDCN-R

For the Developmentally Appropriate Practice (DAP)

Section we adopted three NAEYC Standards detailed by

Copple and Bredekamp (2009). One adopted standard

employed in the MDCN-R is Copple and Bredekamp’s

(2009) ‘‘planning curriculum to achieve important goals’’

(Guideline 3A, 1, p. 20), which focuses on the teacher’s

display of knowledge and understanding of DAP. A dis-

tinguished ranking on this item would indicate that exam-

ples of this knowledge are seen through DVD content

depicting teachers considering the domains of development

(physical, social/emotional, and cognitive) as well as more

than two such disciplines as language, literacy, mathe-

matics, social studies, science, art, music, physical, edu-

cation, and health.

The second adopted standard in the MDCN-R is Copple

and Bredekamp’s (2009) is ‘‘planning curriculum to

achieve important goals’’ (Guideline 3C, 2, p. 20). In the

rubric, DVD content is analyzed in terms of the variety of

Table 1 Existing assessments of technology for early childhood teachers

Assessment Authors Designed for use by Purpose of assessment

Teacher technology

assessment

Bewick and

Kostelnick

(2004)

Teachers Teacher self-assessment checklist of his/her own

knowledge of computer operations, for choosing

appropriate software, and for using computers for

instruction and instructional support

Early childhood

educational

technology evaluation

toolkit

McManis

and Parks

(2011)

Teachers and other early childhood

team members (i.e., parents,

administrators, directors, etc.)

Assist early childhood team members in selecting

appropriate technology that encourages development,

meets child needs and standards, and monitors child

progress

Self-reflection

framework

Snider and

Hirschy

(2009)

Teachers Assist classroom teachers in making appropriate decisions

about what types of technology to integrate into the early

learning environment

Early Childhood Educ J (2013) 41:325–337 327

123

experiences the teacher presents to support children

reaching the goals outlined in the curriculum. A distin-

guished ranking on this item would indicate that the teacher

always underscores many varied experiences in the DVD

content.

The third MDCN-R item in this section considers Copple

and Bredekamp’s (2009) standard, ‘‘creating a caring

community of learners’’ (Guideline, 1B, p. 16). This stan-

dard considers the extent to which the DVD displays that

relationships appear to be an important context through

which children learn and develop. A distinguished ranking

on this item would indicate that relationships are seen as

valued in the DVD content through scenes depicting

opportunities for play, collaboration on investigations, and

time to talk with peers and adults.

Purpose

To our knowledge, the extant literature has not developed a

rubric to provide an assessment of in-service pre-kinder-

garten teachers’ specialized technology use in the form of

creating videos for parents. There is a great need for such

instruments for a reason such as to help teachers make

data-driven decisions specific to technology use in their

classrooms (Britten and Cassady 2006). The first purpose

of this project was to examine the MDCN-R and revised

MDCN-R in terms of content validity and inter-rater reli-

ability. The second purpose was to report the results of use

of the MDCN-R to compare differences between Year 1

and Year 2. We suggest that teachers’ active involvement

in technology (i.e., such as creating DVD classroom

newsletters), and peer feedback in the form of the MCDN-

R may support teachers’ professional growth.

Method

Data Sources

The data sources for this study consisted of a total of 24

monthly DVD classroom newsletters from two consecutive

school years during the winter/spring months. Year 1 DVDs,

from the 2010 to 2011 school year, consisted of 13 DVDs

from three pre-kindergarten sites. The Year 2 (2011–2012)

DVDs consisted of 11 DVDs from three of the same schools

and one new school site participant. Each school site located

in a public school district in northern Nevada had between 2

and 5 monthly DVDs per school year.

Participants

During Year 1 there were three participating Pre-K teach-

ers from a large school district in the Western United

States. Year 2 teachers included the same three partici-

pating teachers and one new participating teacher.

Information gathered during Year 2 reported that the par-

ticipating teachers, all female, held bachelor’s degrees

(n = 3) and master’s degrees (n = 1). The years of expe-

rience teaching Pre-K ranged from approximately 5 years

to 18 years. Families receiving the monthly DVD class-

room newsletters were primarily lower income and from

Spanish-speaking households as indicated by classroom

data.

Content Validity

The MDCN-R was evaluated to determine its content valid-

ity, i.e., the extent to which the rubric adequately assessed the

content domain. Specifically, the MDCN-R was evaluated by

an assistant professor from an outside institution with

expertise in children’s media and early childhood develop-

ment and education. In their rubric design study Roblyer and

Wiencke (2003) adopted criteria from previous work in order

to permit evaluators to review and rate the rubric. The present

study also adopted Jonassen, Peck, and Wilson’s criteria (as

cited in Roblyer and Wiencke 2003), this included:

Had elements that are comprehensive in describing

performance and are ‘‘unidimensional,’’ or not able to

be broken down further into component behaviors;

had ratings that represent clearly different categories

that do not overlap and were comprehensive in cov-

ering the full range of performance; was stated so that

it communicated elements and ratings clearly and

unambiguously. (p. 90)

Overall, the evaluator indicated that the rubric met the

aforementioned criteria. The minor suggestions received

from the reviewer led to small changes in rating descrip-

tions to enhance the clarity of the rubric.

First Round of Reliability: MDCN-R

An author hosted a 20-min information session and training

phase with two master-level graduate students with some

knowledge of Spanish. During this time, the graduate

students participated in a practice phase using the MDCN-R that

included a consensus discussion for items of disagreement.

After the first round of reliability, there were two

important revisions to the rubric. First, the language con-

gruency (LC) item in the rubric was dropped to allow

teachers more flexibility in the Spanish and English scenes

in Year 2. Also, the Menu Functionality (MF) item was

rewritten for clarity. This means that the revised MDCN-

R included a total of 10 items. The highest possible overall

328 Early Childhood Educ J (2013) 41:325–337

123

rubric score is 40 points. In the current rubric, the Tech-

nical Section contains seven items for a possible 28 points.

The Developmentally Appropriate Section contains three

items for a possible 12 points. See the Appendix for the

revised rubric.

Second Round of Reliability: Revised MDCN-R

Recall that Year 1 DVDs, from the 2010–2011 school year,

consisted of 13 DVDs from three pre-kindergarten sites.

The Year 2 (2011–2012) DVDs consisted of 11 DVDs

from three of the same schools and one new school site

participant. Each school site had between 2 and 5 monthly

DVDs per school year.

Two different coders participated in the second round of

reliability. Specifically, two bilingual (English and Span-

ish) doctoral students participated in a 90-min DVD

newsletter rubric training session conducted by another

author. At this training session, the two raters were given

an overview of the Monthly DVD Classroom Newsletter

project and shown some clips to familiarize them with the

layout and purpose of the monthly DVD newsletters. Next,

the revised MDCN-R was reviewed item-by-item and

thoroughly discussed. The scoring process was also dis-

cussed and explained in detail. The raters were not familiar

with the concept of developmentally appropriate practice

so this aspect of the rubric was reviewed in more depth.

Following the comprehensive review of the rubric, the

raters practiced the scoring process by independently

scoring some sample DVDs (both whole DVDs and DVD

clips) using the revised MDCN-R. After each DVD or clip,

the raters shared how they scored each item and a discus-

sion regarding the scoring of each item commenced until

the raters felt comfortable and experienced with the scoring

process.

After the practice/training phase, the raters indepen-

dently scored each of the monthly DVD newsletters from

Year 1 and Year 2. They used a computerized form to

record the scores for each school site’s monthly DVDs,

and noted any relevant comments that arose during the

scoring process. In addition to scoring each item, there

were total scores for the technical and developmentally

appropriate practice sections, and an overall score sum-

med for each of the DVDs. At the completion of the

individual scoring, the scoring sheets of each rater were

evaluated to assess inter-rater reliability (Scott’s Pi,

Cohen’s Kappa, Krippendorff’s Alpha, and percent

agreement were calculated). Following this process, the

two raters discussed any disagreements and a consensus

was reached to compose a final master score sheet. Using

the master score sheet, the scores of Year 1 and Year 2

DVDs were compared.

Results

Reliability for Original MDCN-R

The rubric required coder judgment, meaning that coding

was not a purely straightforward task with a perfect or

extremely high level of reliability expected (Wimmer and

Dominick 2006). According to Landis and Koch (1977) a

Cohen’s Kappa can be interpreted as follows:\.20 is poor,

.21–.40 is fair, .41–.60 is moderate, .61–.80 is good, and

.81–1.0 is very good. The minimum reliability of Scott’s Pi

and Krippendorf’s Alpha of .75 or above is needed for inter-

rater reliability (Wimmer and Dominick 2006). As shown in

Table 2, four of the variables (i.e., Length, LC, 3A1, 3C2)

are very good according to Cohen’s Kappa and above the rule

of thumb for Scott’s Pi and Krippendorf’s Alpha. Four of the

variables are good or close to moderate agreement according

to Cohen’s Kappa. The variables of audio, visual definition

(camera technique), and Copple and Bredekamp’s (2009)

standard of creating a caring community of learners (1B)

needed further exploration due to the low reliability. Overall,

the reliability calculations after the first reliability round

indicated that some aspects of the rubric were promising.

Reliability for Revised MDCN-R

See Table 3. Inter-coder reliability for four of the items

(i.e., Length, VDCT, Light, and 3C2) had perfect agree-

ment. Three items (i.e., MF, 3A1, 1B) had very good

agreement according to Cohen’s Kappa and were above the

rule of thumb for Scott’s Pi and Krippendorf’s Alpha.

Agreement for the remaining three items (i.e., WLC,

Audio, VDR) was fair according to Cohen’s Kappa.

Secondary Findings

Change scores were calculated in order to determine

whether ratings of the DVDs improved between Year 1 and

Year 2, see Table 4, where positive scores indicate higher

ratings at Year 2. Due to missing data across either Year 1

or Year 2, change scores could only be calculated for 8

pairs of Year 1 and Year 2 data. There were no changes in

Overall Total ratings between Year 1 and Year 2. However,

as shown in Table 4, there were areas that showed

increases in ratings for particular items for some DVDs, as

well as areas that showed decreases in ratings for particular

items for some DVDs.

Discussion

Assessment systems comprised of widely accepted and

reliable rubric formats (e.g., Ledoux and McHenry 2006)

Early Childhood Educ J (2013) 41:325–337 329

123

Table 2 Reliability results for first round of DVD coding (Year 1) for MDCN-R

Variable Scott’s Pi Cohen’s Kappa Krippendorff’s Alpha Agreements Disagreements Percent agreement (%)

Length 1 1 1 13 0 100.0

WLC .49 .519 .51 10 3 76.9

Audio .17 .22 .202 7 6 53.8

VDR .592 .598 .607 10 3 76.9

VDCT .137 .161 .17 5 8 38.5

Light .69 .698 .702 11 2 84.6

LC 1 1 1 13 0 100.0

MF .592 .598 .607 10 3 76.9

3A1 .869 .87 .874 12 1 92.3

3C2 .869 .87 .874 12 1 92.3

1B .353 .386 .378 7 6 53.8

Length Length of DVD, WLC written language conventions, Audio audio, VDR visual definition (resolution), VDCT visual definition (camera

technique), Light lighting, LC language congruency, MF menu functionality, 3A1 planning curriculum to achieve important goals (3A, 1), 3C2

planning curriculum to achieve important goals (3C, 2), 1B creating a caring community of learners (1B)

Table 3 Reliability results for second round of DVD coding (Year 1 and Year 2) for revised MDCN-R

Variable Scott’s Pi Cohen’s Kappa Krippendorff’s Alpha Agreements Disagreements Percent agreement

Length 1 1 1 24 0 100.0

WLC .436 .436 .448 17 7 70.8

Audio .289 .294 .304 16 8 66.7

VDR .333 .339 .347 18 6 75.0

VDCT 1 1 1 24 0 100.0

Light 1 1 1 24 0 100.0

MF .917 .912 .918 23 1 95.8

3A1 .829 .830 .832 22 2 91.7

3C2 1 1 1 24 0 100.0

1B .917 .912 .918 23 1 95.8

Length Length of DVD, WLC written language conventions, Audio audio, VDR visual definition (resolution), VDCT visual definition (camera

technique), Light lighting, LC language congruency, MF menu functionality, 3A1 planning curriculum to achieve important goals (3A, 1), 3C2

planning curriculum to achieve important goals (3C, 2), 1B creating a caring community of learners (1B). In Year 2, LC or language congruency

was dropped to allow teachers more flexibility in English and Spanish scenes than the previous year. In Year 2, the MF or menu functionality was

rewritten for clarity

Table 4 Change Scores for Year 1 and Year 2 ratings

Length WLC Audio VDCT Light MF Total T 3A1 3C2 Total D Overall total

School A—Feb. 0 0 1 0 0 0 0 1 0 0 0

School A—Apr. 0 0 -1 0 0 0 0 -1 1 0 0

School A—May 0 0 0 0 0 0 0 0 0 0 0

School B—Feb. -3 2 1 0 0 0 0 0 0 0 0

School B—Mar. -3 0 1 0 0 0 0 -2 0 0 0

School B—May 0 2 0 0 0 0 0 2 0 0 0

School C—Feb. -3 0 0 1 0 0 0 -2 0 0 0

School C—Mar. 0 -1 0 0 0 0 0 -1 0 0 0

Length Length of DVD, WLC written language conventions, Audio audio, VDCT visual definition (camera technique), Light lighting, MF menu

functionality, Total T total for the technical section, 3A1 planning curriculum to achieve important goals (3A, 1), 3C2 planning curriculum to

achieve important goals (3C, 2), Total D total for the developmentally appropriate practice (DAP) section, Overall Total overall total scores for

both technical and DAP sections. In Year 2, LC or language congruency was dropped to allow teachers more flexibility in English and Spanish

scenes than the previous year. In Year 2, the MF or menu functionality was rewritten for clarity

330 Early Childhood Educ J (2013) 41:325–337

123

are needed to promote teacher growth. The primary pur-

pose of this paper was to report on the reliability of a rubric

that assesses early childhood educators’ use of video. The

secondary purpose of this paper was to explore actual

scores or uses of the rubric. To our knowledge, this is the

first study that focused on the development of a rubric to

assess in-service teachers’ use of technology to promote:

(1) developmentally appropriate practice (Copple and

Bredekamp 2009), and, (2) the technical aspects of creating

parent involvement DVDs.

Results from this study suggest there is initial evidence of the

MDCN-R’s validity and reliability to assess bilingual (Spanish

and English) video. Overall, the revised MDCN-R displayed

good reliability. Length of the DVDs was perfect in both

rounds of coding, probably given that it was the most

straightforward item to code. More coder judgment was

needed for the other items in the rubric meaning that

coding was not mechanical for most items (Wimmer and

Dominick 2006). We speculate that because rater judgment

was warranted, overall the reliability improved in the

second round due to the detailed training session (Staus-

berg et al. 2008) compared to the less extensive training

session in the first round. There were four coders in the

present study. Two bilingual coders in the second round

compared to coders with some knowledge of Spanish in the

first round could explain the higher overall reliability of the

bilingual video in the second round. Reliability for audio

was low in the first and second round of coding. This could

be a limitation and possibly be due to a weakness in the

categories for this item (Stausberg et al. 2008). Alterna-

tively, coders may have differences in hearing health or

there could have been variation in the devices they used to

view or code the videos. The audio needs to be further

examined to determine if it is a weak item or if other

differences help to explain the low reliability of it in two

rounds.

The revised MDCN-R may be a useful assessment to

evaluate efforts in early childhood settings that use video,

specifically assessment of the appropriate teaching and

learning practices (Copple and Bredekamp 2009) and

technical aspects of the video, especially length, visual

definition (camera technique), and lighting. Early child-

hood programs creating video in the form of monthly DVD

classroom newsletters (Sanchez and Walsh 2010; Sanchez

et al. 2011) may find the revised MDCN-R particularly

useful. In other words, there have been other efforts that

have used video in the existing literature. For example,

other efforts have included the use of smartphones to

record video to share with parents on blogs (Parnell and

Bartlett 2012) or uploading video of classroom activities

for parents to blogs (Liang et al. 2011). These types of

initiatives may also benefit from the use of the revised

MDCN-R to evaluate early childhood program uses of

video. Future research may wish to expand upon the

MDCN-R to inform professional development efforts cen-

tered on technological tools that use video in early child-

hood settings. In addition, the revised MDCN-R may be

adapted to future technologies and school district initia-

tives, such as posting video to web-based informational and

monitoring programs. Future studies will need to explore

adaptations of the MDCN-R to fit these other technologies.

For early childhood programs using video, it may be

helpful to set aside time during professional development

activities to have teachers independently or as a group rate

their videos using the rubric and compare those to ratings

done by outside evaluators such as university researchers.

Alternatively, existing teacher assessments (Bewick and

Kostelnick 2004; McManis and Parks 2011; Snider and

Hirschy 2009) or creating a new teacher self-assessment to

complement the revised MDCN-R may be useful assess-

ment method to promote teachers growth with technology.

Researchers use of these assessments as part of an iterative

process, that aims to develop a technologically based

communication effort between classroom and parents,

needs to be fully explored. Teachers’ self-assessment can

promote professional growth when coupled with other

professional growth strategies such as feedback from

external change agents like university researchers (Ross

and Bruce 2007). It is imperative that teachers are sup-

ported with self-assessment and reflection tools to help

them identify characteristics and practices that need to

change or evolve in order to promote the quality of

teaching and learning in their classroom (McCombs 1997;

Ross and Bruce 2007). Use of a self-assessment or self-

assessments to complement use of the rubric and use of the

revised rubric as a form of teacher self-assessment are

areas that warrant attention.

The secondary findings that explore the uses of the

revised MDCN-R will be shared with the teachers at the

second Professional Learning Community (PLC) meeting

of the 2012–2013 school year. There were no changes in

overall scores on the MDCN-R from Year 1 to Year 2. At

teacher meetings, researchers and teachers will discuss

reasons for areas that showed increases and decreases from

year-to-year in a way that values teachers’ theories about

the changes in some rubric items. The preliminary appli-

cations of the rubric ratings illustrate the potential use of

this instrument to support pre-kindergarten teachers’

growth with technological tools.

Limitations

Although this study has shown the promise of the MDCN-R,

some limitations need to be acknowledged. First, our sample

of 24 DVDs was relatively small. We recommend studies in

Early Childhood Educ J (2013) 41:325–337 331

123

the future that include more classrooms and use the rubric

over time to inform practice with the video.

Second, the videos or monthly DVD newsletters were

produced by teachers who volunteered to participate and

had the resources to do so (e.g., Internet, camera, computer

software program, DVDs, access to DVD Burner). It is

possible that not all schools have access to the equipment

to create DVD newsletters. Nonetheless, video technolo-

gies, such as DVDs offer key capabilities that text cannot

accomplish for communicating with diverse families, such

as Spanish-speaking parents with low-literacy levels, for

giving a window into their child’s classroom, and for

promoting home and educational practices. Teachers

without access to these materials may consider contacting

office and technology companies for donations or form a

cooperative effort with stakeholders to share resources to

produce the DVDs. Another approach is to use video fea-

tures on phones and post the video online with password

protection. This strategy reduces barriers associated with

burning/copying DVDs. For readers desiring something

similar to the project featured in this article, pooling

resources across a school or within a community to create a

school-wide DVD newsletter, may be a useful starting

point.

Finally, using video to capture classroom activities can

be time-consuming and requires skill but it is possible and

rewarding (Luckenbill 2012). Anecdotally, the researchers

observed that the teachers were engaged in this project. We

speculate it was because they volunteered to participate,

had ownership of the project, and supported each other. As

a result, requiring teachers to participate or requiring them

to produce more videos may not be an appropriate avenue

to alleviate the issue of a small sample.

Conclusion

The revised MDCN-R has good overall reliability. It could

be an assessment tool for pre-kindergartner teachers’ video

production and uses, especially those that seek to inform

and communicate to parents/families about classroom

activities in early childhood settings. Many assessments

surrounding technology have been developed with K-12

teachers (Christensen and Knezek 2001). Because the

revised MDCN-R was developed for use in pre-kindergar-

ten classrooms, it makes an important contribution to the

extant literature. Sharing the results of the revised rubric

with teachers and discussing the results may help them

make decisions about their uses of technological tools in

early childhood settings. The researchers have plans in

place to fully develop a self-assessment to complement the

revised MDCN-R. Future research is planned to use the

rubric as a fidelity measure in a study that develops the

video intervention and examines it in relation to early

childhood student outcomes as well as parent and teacher

experiences with the video.

332 Early Childhood Educ J (2013) 41:325–337

123

Appendix

Early Childhood Educ J (2013) 41:325–337 333

123

334 Early Childhood Educ J (2013) 41:325–337

123

Early Childhood Educ J (2013) 41:325–337 335

123

References

Autry, M. M., Lee, J., & Fox, J. (2009). Developing a data-driven

assessment for early childhood candidates. Journal of Early

Childhood Teacher Education, 30(2), 138–149. doi:10.1080/

10901020902885695.

Bewick, C. J., & Kostelnick, M. (2004). Teacher technology

assessment. Beyond the Journal: Young Children on the Web.

Retrieved from http://www.naeyc.org/yc/pastissues/2004/may.

Britten, J. S., & Cassady, J. C. (2006). The technology integration

assessment instrument. Computers in the Schools, 22, 49–61.

doi:10.1300/J025v22n03_05.

Calabrese, N. M. (2006). Video technology: A vehicle for educators

to enhance relationship with families. Education, 127,

155–160.

Christensen, R., & Knezek, G. (2001). Instruments for assessing the

impact of technology in education. Computers in the Schools, 18,

5–25. doi:10.1300/J025v18n02_02.

Copple, C., & Bredekamp, S. (2009). Developmentally appropriate

practice in early childhood programs serving children from birth

through age 8 (3rd ed.). Washington, DC: National Association

for the Education of Young Children.

Critchosin, H. (2009). Creating a professional learning community

through the use of technology: A link between a community

school, pre-service teachers, and higher education. In I. Gibson,

et al. (Eds.), Proceedings of Society for Information Technology

& Teacher Education International Conference 2009 (pp.

3255–3260). Chesapeake, VA: Association for the Advancement

of Computing in Education.

Epstein, J. L. (1995). School/family/community partnerships: Caring

for the children we share. Phi Delta Kappan, 76, 701–712.

Gimbert, B., & Cristol, D. (2004). Teaching curriculum with

technology: Enhancing children’s technological competence

during early childhood. Early Childhood Education Journal,

31, 207–216.

Good, M. E., Masewicz, S., & Vogel, L. (2011). Latino English

language learners: Bridging achievement and cultural gaps

between schools and families. Journal of Latinos and Education,

9, 321–339. doi:10(1080/15348431),2010,491048.

Hall, L., & Hudson, R. (2006). Cross curricular connections: Video

production in a K-8 teacher preparation program. Contemporary

Issues in Technology and Teacher Education, 6(3), 328–341.

Landis, J., & Koch, G. (1977). The measure of observer agreement for

categorical data. Biometrics, 33, 159–174.

Ledoux, M. W., & McHenry, N. (2006). Electronic portfolio adoption

for teacher education candidates. Early Childhood Education

Journal, 34(2), 103–116. doi:10.1007/s10643-006-0111-1.

Liang, P. H., Wang, Y. S., Wu, H. J., & Liao, Y. C. (2011).

Constructing a class blog for parent- teacher communication:

The case of a private kindergarten in Taiwan. In S. Barton et al.

(Ed.), Proceedings of global learn Asia Pacific, Australia,

1827–1832. Retrieved from http://www.editlib.org/f/37408.

Luckenbill, J. (2012). Getting the picture: Using the digital camera as

a tool to support reflective practice and responsive care. Young

Children, 67(2), 28–36.

McCombs, B. L. (1997). Self-assessment and reflection: Tools for

promoting teacher changes toward learner-center practices.

NASSP Bulletin, 81, 1–14. doi:10.1177/019263659708158702.

336 Early Childhood Educ J (2013) 41:325–337

123

McManis, L.D., & Parks, J. (2011). Evaluating technology for early

learners [E-book and toolkit].Winston-Salem, NC: Hatch Early

Learning. Retrieved from www.hatchearlychildhood.com/toolkit.

National Association for the Education of Young Children & the Fred

Rogers Center for Early Learning and Children’s Media. (2012).

Technology and interactive media as tools in early childhood

programs serving children from birth through age 8. Retrieved from

http://www.naeyc.org/files/naeyc/file/positions/PS_technology_

WEB2.

Panferove, S. (2010). Increasing ELL parental involvement in our

schools: Learning from the parents. Theory into Practice, 49,

106–112. doi:10.1080/00405841003626551.

Parnell, W., & Bartlett, J. (2012). iDocument: How smartphones and

tablets are changing documentation in preschool and primary

classrooms. Young Children, 67(3), 50–57.

Roblyer, M. D., & Wiencke, W. R. (2003). Design and use of a rubric

to assess and encourage interactive qualities in distance courses.

The American Journal of Distance Education, 17(2), 77–98.

Ross, J. A., & Bruce, C. D. (2007). Teacher self-assessment: A

mechanism for facilitating professional growth. Teaching and

Teacher Education, 23(2), 146–159.

Sanchez, C., & Walsh, B. A. (2010). Teachers’ perceptions of the use

of monthly DVD classroom newsletters. International Journal of

Technology in Teaching and Learning, 6(2), 157–171.

Sanchez, C., Walsh, B. A., & Rose, K. K. (2009). How to encourage

Hispanic family involvement. Texas Child Care: The Quarterly

Journal for Caregivers Everywhere, 33, 24–29.

Sanchez, C., Walsh, B. A., & Rose, K. K. (2011). DVD newsletters:

New ways to encourage communication with families. Dimen-

sions of Early Childhood, 39(2), 20–26.

Snider, S., & Hirschy, S. (2009). A self-reflection framework for

technology use by classroom teachers of young children. He Kupu,

2, 30–34. Retrieved from http://www.hekupu.ac.nz/Journal%20

files/Issue1%20June%202009/A%20Self-Reflection%20Frame

work%20for%20Technology%20Use%20by%20Classroom%20

Teachers%20of%20Young%20Learners.pdf.

Stausberg, J., Lehmann, N., Kaczmarek, D., & Stein, M. (2008).

Reliability of diagnoses coding with ICD-10. International

Journal of Medical Information, 77, 50–57. doi:10.1016/

j.ijmedinf.2006.11.005.

Sweeder, J. (2007). Digital video in the classroom: Integrating theory

and practice. Contemporary Issues in Technology and Teacher

Education, 7(2), 107–128.

Thornton, M. W., & Broughton, A. (2010, November). Video

analysis: A key assessment tool in early childhood teacher

education. In Session presented at the meeting of the National

Association for the Education of Young Children, Anaheim, CA.

Walsh, B. A., Buckley, R. R., Rose, K. K., Sanchez, C., & Gillum, N.

L. (2008). Innovative school-family communication: Parent’s

perceptions of the use of monthly classroom DVD newsletters.

In P. G. Grotewell & Y. R. Burton (Eds.), Early childhood

education: Issues and developments (pp. 211–221). Hauppauge,

NY: Nova Science.

Wimmer, R. D., & Dominick, J. R. (2006). Mass media research: An

introduction (8th ed.). Belmont, CA: Thomson-Wadsworth.

Early Childhood Educ J (2013) 41:325–337 337

123