Arthur L. Petterway, PhD Proposal Dissertation Defense, Dr. William Allan Kritsonis, Committee Member

MAY 2006 1

A Mixed-Methods Analysis of the Impact of High Stakes Testing on

English Language Learners in Major Urban High Schools in

Texas

A Dissertation Proposalby

Arthur L. Petterway

William Allan Kritsonis, PhDDissertation Committee Member

MAY 2006 2

Proposal Defense Format

I. Purpose of StudyII. Research Questions(4)III. Hypotheses(2)III. Significance of the StudyIV. Review of LiteratureV. Research Design

MAY 2006 3

Purpose of the Study

The purpose of the study is two-fold:

1. Determine whether there is a significant relationship between the ratio of ESL students taking the TAKS test, relative proportion of the school’s faculty who are certified to teach English as a first language and the Grade 10 TAKS in Reading/English Language Arts and Mathematics.

MAY 2006 4

Purpose of the Study

2. Explore what certified ESL teachers, non – certified ESL teachers who teach ELLs, campus administrators, and district ESL personnel view as the impact that high stakes standardized assessments have on ELLs, ELL curriculum and instruction.

MAY 2006 5

Research Questions(1): What is the effect of high

stakes standardized assessments and their impact on ELLs’ motivation, state of mind or behavior / view of the test?

MAY 2006 6

Research Questions(2):• What is the influence of

teacher certification status on high stakes standardized assessments on ELLs?

MAY 2006 7

Research Questions(3):

• What is the impact of high stakes standardized assessments on ELLs?

MAY 2006 8

Research Questions(4): Is there a relationship between

TAKS performance in Reading/English Language Arts and Mathematics of 10th Graders and designation of ESL or non-ESL, type of teacher (certified or non-certified to teach ESL) and % of students who do not speak English as a first language ?

MAY 2006 9

Hypotheses(1):

Ho1: There is no statistically significant relationship between the ratio of ESL students taking the TAKS test, relative proportion of the school’s faculty who are certified to teach ESL students and the % of students who do not speak English as a first language and the school’s performance in the Grade 10 TAKS test in English/Language Arts.

MAY 2006 10

Hypotheses(2):

Ho2: There is no statistically significant relationship between the of ratio of ESL students taking the TAKS test, relative proportion of the school’s faculty who are certified to teach ESL students and the % of students who do not speak English as a first language and the school’s performance in the Grade 10 TAKS test in Mathematics.

MAY 2006 11

Significance of the Study:• Expected outcome of the study

will be to provide additional data for standardized assessment writers in regards to biases and to school districts in developing assessments that truly measure learning without the nullifying effects of linguistic and cultural biases.

MAY 2006 12

Significance of the Study

•It will also help to enhance the reliability of standardized assessments as a tool to determine accountability for student performance of English language learners.

MAY 2006 13

Review of Literature

• No Child Left Behind (NCLB)• Adequate Yearly Progress

(AYP)• Limited English Proficient

(LEP) Students• High Stakes/Statewide

Testing

MAY 2006 14


• No Child Left Behind (NCLB)–Historical Note–Description of the Key Factors

–Expectations for Parents–Response to NCLB

MAY 2006 15


• No Child Left Behind Act of 2002 (NCLB)–Historical Note: In 2003, the Center of Educational Policy clarified why accountability was not a part of ESEA in 1965:

MAY 2006 16

Review of Literature “At that time the federal role in education was marginal, most state education agencies had very limited authority and capabilities, and local people were extremely wary that more federal aid would bring federal control”(p.iv).

MAY 2006 17


• No Child Left Behind Act of 2002 (NCLB)-Description of the Key Factors (Rosenbusch, 2005)(4):

1. Accountability2. Testing3. Teacher Quality4. Scientifically-Based Research

MAY 2006 18


• Accountability to Parents– Beginning in 2005, grades 3 – 8 must be tested

in Math and English– By the end of SY 2005 – 2006, teachers must

be “highly qualified”– Number of students achieving state standards

must increase each year until reaching 100% at the end of 12 years

– Schools must notify parents if their child’s school is targeted for improvement

MAY 2006 19


• Expectations for Parents (cont.)– If a school targeted for improvement fails after

two years, parents may choose to transfer their child to another school or enroll in free tutoring.

Source: collegeboard.com

MAY 2006 20


• Response to NCLB– Controversy (Rosenbusch, 2005)– Majority of Americans believe in local control of

schools (Rose & Gallup, 2003)– Many believe Math and English testing not

sufficient to give accurate picture of the school (Rose & Gallup, 2003)

– Could result in narrowing of the curriculum and “sorting of students” (Marshak, 2003, p.229)

– NEA says focus on punishment is an obstacle (National Education Association, n.d.)

MAY 2006 21


• Adequate Yearly Progress (AYP)

–Purpose and Support to NCLB

–Changes and Updates

MAY 2006 22


• Limited English Proficient (LEP) Students–Definition of English Language Learners (ELLs)

– Issues and Other Considerations of LEP

MAY 2006 23


• High Stakes/Statewide Testing– Principles of Testing Programs– Accountability in Testing– Effects of High Stakes Testing

on Student Motivation– Other Considerations of

Assessment on Testing

MAY 2006 24

Review of Literature• Identification of ELLs

– Home Language Survey– Proficiency tests

(Abedi, 2004b)

– Assessment instruments may not be sufficient

– Leads to inconsistency

(Zehler, Hopstock, Fleischman & Greniuk, 1994)

MAY 2006 25


• Strategies to improve LEP instruction– Improve classification methods– Improve monitoring– Improve teacher quality– Include redesignated LEP students as

part of the LEP subgroup

(Abedi, 2003)

MAY 2006 26


• High Stakes Statewide Testing– States required to administer Reading and

Math tests annually in grades 3 – 8 and during one year in high school starting in 2005 – 2006 (National Center for Education Statistics, 2002)

– States must meet AYP goals (Abrams & Madaus, 2003)

– Use of testing to change pedagogical priorities has a long history (Madaus & O’Dwyer, 1999)

MAY 2006 27

Principles of Testing Programs

MAY 2006 28


• Principle 1–The power of tests is a

perceptual phenomenon

–All parties must believe the results are important

(Airasian, 1988)

MAY 2006 29


• Principle 2– The more a quantitative social indicator

is used for social decision making, the more likely it will be to distort and corrupt the social process it is intended to monitor(Amrein & Berliner, 2002; Haladyna, Nolen & Haas, 1991; Klein, Hamilton, McCaffrey & Stecher, 2000; Linn, 1998)

MAY 2006 30

Review of Literature• Principle 3

– If important decisions are based on test results, then teachers will teach to the test.

(Jones et al., 1999; Madaus, 1991; Madaus, 1991; McMillan, Myran & Workman, 1999; Pedulla et al., 2003; Stecher, Barron, Chun & Ross, 2000)

MAY 2006 31


• Principle 4– In every setting where high-stakes

tests operate, the exam content eventually defines the curriculum

(Herman and Golan, n.d.; Hoffman, Assaf, & Paris, 2001; Jones et al., 1999; Pedula et al., 2003)

MAY 2006 32


• Principle 5– Teachers pay attention to the form

of the questions of high-stakes tests and adjust their instruction accordingly.

(Taylor, Shepard, Kinner & Rosenthal, 2003; Kortz, Barron, Mitchell, & Keith, 1996a)

MAY 2006 33


• Principle 6– When test results are the arbiter of

future education or life choices, society treats test results as the major goal of schooling.

(Holmes, 1911, p.128; Edwards, 2003)

MAY 2006 34


• Principle 7–A high-stakes test transfers

control over the curriculum to the agency that sets or controls the exam.

(Myers, 2003)

MAY 2006 35


• Accountability in Testing– Cut-off scores are arbitrary (Horn,

Ramos, Blumer & Madaus, 2000)

– Test scores are fallible (Rhoades & Madaus, 2003)

– No test can be truly comprehensive (Harlow & Jones, 2003)

MAY 2006 36


• Effects of High-Stakes Testing on Student Motivation and Learning– Evidence shows that such tests actually

decrease student motivation and increase the number of students who leave school early.

(Arein & Berliner, 2003)– Attaching high stakes to tests apparently

obstructs students’ paths to becoming lifelong, self-directed learners and alienates students.

(Sheldon & Biddle, 1998)

MAY 2006 37


• Effects of High-Stakes Testing on Student Motivation and Learning (cont.)– Many researchers hold high-stakes testing at

least partly to blame for climbing dropout rates.

(Rothstein, 2002; Jacob, 2001; FairTest & Massachusetts CARE, 2000)

– More teenagers are exiting school early to earn a General Educational Development (GED) credential

(Murnane, Willett, & Tyler, 2000)

MAY 2006 38


• Effects of High-Stakes Testing on Student Motivation and Learning (cont.)– May increase drop-out rate when promotion to

next grade hinges on passing state exams

(Goldschmidt & Wang, 1999)– Results in a high percentage of minority and

low socio-economic background students being retained (McNeil, 2000; Haney, 2000, 2001; Klein, Hamilton, McCaffey & Stecher, 2000; Yardley, 2000; Fisher, 2000)

MAY 2006 39


• Other Considerations of Assessment and Testing– Can affect the number of students, especially

black and Hispanic students, who are classified as Special Education

(Thurlow, Neilson, Tellucksingh,& Ysseldyke, 2000; Haney, 2000; D’Emilio, 2003, June; Zehler, Fleischman, Hopstock, Pendzick, & Stepherson, 2003)

MAY 2006 40


• Other Considerations of Assessment and Testing (continued)– Can negatively affect teacher and student

morale (Anderson, 2004; Flores & Clark, 2003)– Can decrease student motivation (Lane &

Stone, 2002)– May increase drop-out rate for English

Language Learners ( Hood, 2003; Anderson, 2004; Barro & Kolstad, 1987; Kaufman, alt & Chapman, 2001)

MAY 2006 41


• Other Considerations of Assessment and Testing (continued)– May contribute to teacher burnout (Hinde,

2003)– Vitally connected to socio-cultural, economic

and psychological issues (solano-Flores & Trumbull, 2003

– Can change the way in which teachers and students interact (Cheng, 1999)

MAY 2006 42


• Other Considerations of Assessment and Testing (continued)– Requires administrators, teachers

and students to be motivated and invested in demonstrating achievement in order to be successful (Lane and Stone, 2002)

MAY 2006 43

Research Design

• Research Methodology– Descriptive Comparative Research

– Triangulation (open-ended questionnaire, focus groups, and interviews)

– Explanatory Design

– Correlational Research

MAY 2006 44

A Mixed Methods Analysis of Impact of High Stakes Testing on English Language Learners in Major Urban High Schools in Texas

QUANTITATIVE

QUALITATIVE

Explanatory Design

Predictors% ELLs passing

% Non-ELLspassing

% Non-Certified passing

Views/Opinions

Administrators

TeachersDistrict

Personnel

Dependent Variable

Grade 10

TAKS

Reading/ELA &Math

Impact of

Statewide

Testing

On

ELLs

Student

Performance

ESL

Curriculum

And

Instruction

Multiple Regression

MAY 2006 45

Research Design

• Correlational Research– Independent Variables:

1. The ratio of ESL students taking the TAKS test2. The relative proportion of the school’s faculty who are certified to teach ELL3. The percentage of students who do not

speak English as a first language– Dependent Variable:

10th grade Reading/English Language Arts and Mathematics TAKS scores

MAY 2006 46

Research Design• Subjects of the StudyQuantitative:

All major high schools in selected major urban school districts in Texas

MAY 2006 47

Research Design

• Subjects of the StudyQualitative:

Per SchoolTotal

1. ESL Teachers 3 302. Non-Certified ESL 3 30

Teachers1. Principals 1 102. Assistant Principals 2 203. District ESL Personnel 8

Total 98

MAY 2006 48

Research Design• Pilot Study - Qualitative

– 2 HISD schools

Three basic considerations:1. Administer pre-test under conditions

comparable to those anticipated in the final study.

2. Analyze results to assess the effectiveness of the trial questionnaire.

3. Make additions, deletions, and/or modifications to the questionnaire.

(Isaac and Michael, 1995)

MAY 2006 49

Research Design• Instrumentation

Quantitative:

The records section of TEA

Qualitative:

• Open-ended questionnaire www.apetterway.speedsurvey.com

• Focus groups

• Interviews

http://www.apetterway.speedsurvey.com/

MAY 2006 50

Research Design

• Reliability and Validity – Qualitative

The triangulation method will involve (1) the analysis of the quantitative data, (2) collation of data from the on-line questionnaire, and (3) interviews/focus groups.

MAY 2006 51

Research Design• Data Analysis – Quantitative

Descriptive and Comparative Statistics will be used to calculate the means and standard deviations between the variables.

MAY 2006 52

Research Design

• Data Analysis – QuantitativeCorrelational Research

Two separate multiple regressions will be computed:

y=a+b1x1+b2x2+b3x3

– Reading/English Language Arts – Mathematics

MAY 2006 53

Research Design• Data Analysis – Quantitative

Correlational Research

SPSS computations:

1. A Pearson r correlation coefficient will be calculated

2. Multiple R and R squared

3. Degree of freedom, the F-value, and the level of significance

4. Regression formula for predictability

MAY 2006 54

Research Design

Data Analysis – Qualitative

The information gathered from the qualitative portion will be organized under different categories in a frequency table. Percentages will be calculated and listed in descending order.

MAY 2006 55

A Mixed Methods Analysis of Impact of High Stakes Testing on English Language Learners in Major Urban High Schools in Texas

QUANTITATIVE

QUALITATIVE

Explanatory Design

Predictors% ELLs passing

% Non-ELLspassing% Non-

Certified passing

Views/Opinions

Administrators

TeachersDistrict

Personnel

Dependent Variable

Grade 10

TAKS

Reading/ELA &Math

Impact of

Statewide

Testing

On

ELLs

Student

Performance

ESL

Curriculum

And

Instruction

Multiple Regression