Standard Setting Report - MaineStandard Setting Report ... The computational programming used to carry out all analyses during the standard setting meeting was completed and thoroughly

Standard Setting Report

JUNE 28–30, 2010

PORTLAND, ME

Prepared for the Maine Department of Education by:

100 Education Way, Dover, NH 03820 (800) 431-8901

TABLE OF CONTENTS

CHAPTER 1. OVERVIEW OF STANDARD SETTING ..........................................................................................................2

CHAPTER 2. TASKS COMPLETED PRIOR TO STANDARD SETTING ..................................................................................3

2.1 Creation of Achievement Level Descriptors .................................................................................................3

2.2 Selection of Student Portfolios (Bodies of Work)..........................................................................................3

2.3 Preparation of Materials for Panelists .........................................................................................................3

2.4 Preparation of Presentation Materials .........................................................................................................4

2.5 Preparation of Instructions for Facilitators Document ................................................................................4 2.6 Preparation of Systems and Materials for Analysis During the Meeting .....................................................4

2.7 Selection of Panelists .............................................................................................................. ......................4 CHAPTER 3. TASKS COMPLETED DURING STANDARD SETTING ....................................................................................6

3.2 Orientation................................................................................................................... .................................6

3.3 Review of Assessment Materials ...................................................................................................................6 3.4 Review of Achievement Level Definitions .....................................................................................................7

3.5 Training Evaluation ............................................................................................................................. .........7

3.6 Round 1 Judgments .......................................................................................................................................7 3.7 Tabulation of Round 1 Results ......................................................................................................................7

3.8 Round 2 Judgments ............................................................................................................................. ........10

3.9 Tabulation of Round 2 Results ....................................................................................................................10 3.10 Round 3 Judgments .................................................................................................................. ...................13

3.11 Tabulation of Round 3 Results ....................................................................................................................13

3.12 Repeat Process for Second Grade and Content Area (Mathematics and Reading Only) ........................... 21 3.13 Evaluation............................................................................................................................. ......................21

CHAPTER 4. TASKS COMPLETED AFTER STANDARD SETTING ....................................................................................22

4.1 Analysis and Review of Panelists’ Feedback ..............................................................................................22

4.2 Calculation of Recommended Cutpoints.....................................................................................................22

4.3 Preparation of this Standard Setting Report...............................................................................................27

REFERENCES .................................................................................................................................................................28

APPENDICES...................................................................................................................................................................29

APPENDIX A. ACHIEVEMENT LEVEL DEFINITIONS

APPENDIX B. AGENDAS

APPENDIX C. NONDISCLOSURE FORM

APPENDIX D. SAMPLE RATING FORM

APPENDIX E. SAMPLE EVALUATION

APPENDIX F. OPENING SESSION POWERPOINT

APPENDIX G. FACILITATORS’ SCRIPTS

APPENDIX H. PANELIST AFFILIATIONS

APPENDIX I. EVALUATION RESULTS

APPENDIX J. POLICY ADJUSTMENTS

Overview of Standard Setting Process 1 2010 Maine PAAP Standard Setting Report

Chapter 1. OVERVIEW OF STANDARD SETTING

The purpose of this report is to summarize the activities of the standard setting meeting for Maine’s

Personalized Alternate Assessment Portfolio (PAAP) in reading and mathematics (grades 2–7, 10, and 11),

science (grades 5, 8, and 11) and writing (grades 4, 7, and 11). The PAAP standard setting meeting was held

between June 27 and 29, 2010. In all, there were 14 panels with 70 panelists participating in the process.

Eight panels met for two days, and each panel established cuts for two grade level combinations (either two

reading grades or two mathematics grades). The remaining six panels met for one day and established cuts for

a single grade and content area combination. The configuration of the panels is shown in Table 1-1. Note that

some panelists participated in multiple content areas.

Table 1-1. 2010 PAAP Standard Setting: Configuration of Standard Setting Panels

Panel Num

pan

ber of

elists June 28

June 29

June 30

Panel 1 8 Math 2 Math 3 Panel 2 8 Math 4 Math 5 Panel 3 1 0 Math 6 Math 7 Panel 4 9 Math 10 Math 11 Panel 5 1 1 Reading 2 Reading 3 Panel 6 9 Reading 4 Reading 5 Panel 7 8 Reading 6 Reading 7 Panel 8 7 Reading 10 Reading 11 Panel 9 7 Science 5 Panel 10 7 Science 8 Panel 11 7 Science 11 Panel 12 1 0 Writing 4 Panel 13 9 Writing 7 Panel 14 8 Writing 11

A modified version of the body of work method was used for setting standards for the PAAP. The

body of work standard setting method was developed specifically for use with assessments that are designed to

allow for a range of student responses, such as portfolio- or performance-based assessments (Kingston, Kahl,

Sweeney, & Bay, 2001). A modified version of the method has been in use for a number of years that

substantially reduces the logistical burden of the procedure and has been found to yield reasonable and

defensible cutpoints. In the body of work method, panelists are presented with samples of actual student work

(in this case, student portfolios) and make their judgments based on those samples. Specifically, panelists

examine each student portfolio and determine which achievement level best matches the particular

knowledge, skills, and abilities the student exhibits through his or her performance on the work sample. This

report is organized into three major sections, describing tasks completed prior to, during, and after the

standard setting meeting.

Overview of Standard Setting Process 2 2010 Maine PAAP Standard Setting Report

Chapter 2. TASKS COMPLETED PRIOR TO STANDARD SETTING

2.1 Creation of Achievement Level Descriptors

The Achievement Level Descriptors (ALDs) describe the set of knowledge, skills, and abilities that

students in each achievement level are expected to display. Staff at the Maine Department of Education

(MDOE) created these draft descriptors prior to the standard setting meeting, where they were presented to

the panelists. The draft ALDs are provided in Appendix A.

2.2 Selection of Student Portfolios (Bodies of Work)

The goal in selecting student portfolios to use for the standard setting was to select a total of 30 to 40

bodies of work, spread as evenly as possible across the range of possible total raw scores. For the PAAP,

teachers select from a series of standardized tasks. For future administrations, the teachers will be required to

sample tasks that match a predefined blueprint. Consequently, an additional goal was to sample portfolios that

matched the blueprint as best as possible. However, because the blueprint was not implemented in 2010, it

was not possible to find portfolios that represented the blueprint at each total score. A list of portfolios was

generated with approximately three times the target number of portfolios at each score point, and the

portfolios were reviewed and selected by Measured Progress special education staff. The final numbers of

portfolios selected ranged from 29 to 38.

2.3 Preparation of Materials for Panelists

The following materials were assembled for presentation to the panelists at the standard setting

meeting:

▪ Meeting agenda

▪ Nondisclosure agreement

▪ ALDs

▪ Samples of student portfolios

▪ Rating forms

▪ Evaluation forms

Copies of the ALDs, meeting agenda, nondisclosure form, a sample rating form, and evaluation forms

are included in Appendices A through E.

Tasks Prior to Standard Setting 3 2010 Maine PAAP Standard Setting Report

2.4 Preparation of Presentation Materials

The PowerPoint presentation used in the opening session was prepared prior to the meeting. The

presentation was designed to give panelists an overview of the assessment and how it is scored as well as a

preview of what to expect throughout the standard setting process. A copy of the presentation is included in

Appendix F.

2.5 Preparation of Instructions for Facilitators Document

A script was created for the group facilitators to refer to while working through each step of the

standard setting process. This document is included in Appendix G. The facilitators also attended a training

session, led by a Measured Progress psychometrician, approximately one week before the standard setting

meeting. The purpose of the training was to prepare the facilitators for the panel activities and to ensure

consistency in the implementation of procedures.

2.6 Preparation of Systems and Materials for Analysis During the

Meeting

The computational programming used to carry out all analyses during the standard setting meeting

was completed and thoroughly tested prior to the standard setting meeting.

2.7 Selection of Panelists

As was emphasized in Cizek and Bunch (2007), regardless of the method used, the selection of

panelists is an important factor in determining standard setting outcomes and maximizing the validity of the

standard setting process. The guidance provided by Standards for Educational and Psychological Testing

(1999) states “a sufficiently large and representative group of judges should be involved to provide reasonable

assurance that results would not vary greatly if the process were repeated.” Consistent with the above guidance,

as well as practical considerations regarding the maximum size of group that can be successfully managed by

group facilitators, the goal was to recruit standard setting panels of eight members representing different

stakeholder groups to set standards for each grade and content area. Targets for the size and composition of the

panels were also consistent with federal guidelines as described in Standards and Assessment Peer Review

Guidance: Information and Examples for Meeting Requirements of the No Child Left Behind Act of 2001 (U.S.

Department of Education, 2009).

Panelists were selected by the MDOE and Measured Progress prior to the standard setting meeting.

The goal was for each panel to consist of approximately eight participants: two or three special education

teachers experienced in working with students who have significant disabilities; two or three content area

teachers (representative of a range of grade level experiences); and one or two school administrators, higher

education personnel, general education teachers, or stakeholders from interest groups related to significant


disabilities. In addition, to the extent possible, panels were assembled so as to reflect a balance of gender,

race/ethnicity, and geographic location. A list of the panelists and their affiliations is included in Appendix H.


Chapter 3. TASKS COMPLETED DURING STANDARD SETTING

3.1 Overview of Body of Work Method

The body of work standard setting method was developed specifically for use with assessments that

are designed to allow for a range of student responses, such as portfolio- and performance-based assessments.

For a number of years a modified version of the method has been in use that substantially reduces the logistical

burden of the procedure and has been found to yield reasonable and defensible cutpoints. Panelists were asked

to evaluate each work sample from a holistic perspective before classifying it into a single achievement level.

3.2 Orientation

With regard to panelist training, Standards for Educational and Psychological Testing states the

following:

Care must be taken to assure that judges understand what they are to do. The

process must be such that well-qualified judges can apply their knowledge

and experience to reach meaningful and relevant judgments that accurately

reflect their understanding and intentions. (AERA/APA/NCME, 1999, p. 54)

The training of the panelists began with a general orientation at the start of the standard setting

meeting. The purpose of the orientation was to ensure that all panelists received the same information about

the need for and goals of standard setting and about their part in the process. First, the MDOE provided some

pertinent context about the PAAP program and an introduction to the issues of standard setting. Second, the

PAAP program manager provided an overview of the assessment, including alternate grade level expectations,

task bank selection, administration, and scoring. Next, a Measured Progress psychometrician presented a brief

overview of the body of work procedure and the activities that would occur during the standard setting

meeting. Once the general orientation was complete, each panel convened in a breakout room, where the

panelists received more detailed training and completed the standard setting activities.

3.3 Review of Assessment Materials

The first step after the opening session was for the panelists to become familiar with the PAAP. The

purpose of this step was to make sure the panelists thoroughly understood how the portfolio is administered

and scored. Panelists reviewed the Alternate Grade Level Indicators (AGLIs) and administration manuals. In

addition, panelists individually reviewed every fifth portfolio and discussed the knowledge, skills, and

abilities associated with each work sample as a group.

Tasks During Standard Setting 6 2010 Maine PAAP Standard Setting Report

3.4 Review of Achievement Level Definitions

The second step in the process, once the panelists convened into their content area and grade level

groups, was to discuss the Achievement Level Descriptors (ALDs). This important step was designed to ensure

that panelists thoroughly understood the knowledge, skills, and abilities needed for students to be classified

into achievement levels (Substantially Below Proficient, Partially Proficient, Proficient, and Proficient

With Distinction). Panelists first reviewed the ALDs on their own and then participated in group discussion

of the ALDs, clarifying the description for each achievement level. The discussions focused on the knowledge,

skills, and abilities that differentiated adjacent achievement levels. Bulleted lists of characteristics for each

level were generated based on the group discussion and were posted in the room for panelists to refer to during

the rounds of ratings.

3.5 Training Evaluation

Prior to beginning the Round 1 ratings, the panelists anonymously completed a training evaluation

form. The purpose of the evaluation was to ensure that panelists were comfortable with the process and ready

to move on to the rating task. Any issues or problems that came up in the training evaluations were addressed

before the facilitator proceeded to Round 1.

3.6 Round 1 Judgments

In the first round, panelists worked individually with the ALDs, the student portfolios, and the rating

form. The work samples consisted of 29 to 38 portfolios, with scores covering the full range of possible total

scores. For each portfolio, the panelists considered the skills and abilities demonstrated in the work sample,

and panelists decided which achievement level was the best match. The panelists worked their way through

the portfolios, making a rating for each one, and recorded their ratings on the rating form. While the portfolios

were presented in order of total score, panelists were not required to rate them strictly in increasing order.

Instead, panelists were encouraged to take a holistic look at the knowledge, skills, and abilities demonstrated

in the portfolio, rather than making a judgment based primarily on the total raw score.

3.7 Tabulation of Round 1 Results

After all panelists had completed their individual ratings, the Measured Progress data analysis team

calculated the average cutpoints for the group based on the Round 1 ratings. Cutpoints were calculated using

SAS statistical software. Logistic regression was used to determine each panelist’s individual cutpoints, and

then the cutpoints were averaged across the group. In addition, impact data were calculated, which reflected

the percentage of students who would fall into each achievement level based on the group average Round 1

ratings. The Round 1 results are outlined in Tables 3-1 through 3-4.


Achievement level Average

cut Standard

error Raw

Min

score

Max

Percent of students

PD 62.6 1.6 63 69 3.4 Proficient 45.9 2.1 46 62 21.0 Partially Proficient 24.9 2.5 25 45 50.0

PD 70.2 1.6 NA NA NA Proficient 52.9 2.8 53 69 35.6 Partially Proficient 25.8 1.9 26 52 41.5






PD 125.9 5.8 126 129 9.5

Proficient 83.2 4.3 84 125 34.4

Partially Proficient 24.1 11.9 25 83 42.3

SBP NA NA 0 24 13.7


cut Standard

error Raw

Min

score

Max

Percent of students



PD 46.0 1.3 NA NA NA Proficient 29.3 1.7 30 46 65.7

continued

Grade

Table 3-1. 2010 PAAP Standard Setting: Round 1 Mathematics

2

SBP NA NA 0 24 25.6

3

SBP NA NA 0 25 22.9

4

SBP NA NA 0 20 8.5

5

SBP NA NA 0 25 15.5

6

SBP NA NA 0 25 14.1

7

SBP NA NA 0 24 8.2

10

SBP NA NA 0 36 18.8

11

Grade

Table 3-2. 2010 PAAP Standard Setting: Round 1 Reading

2

SBP NA NA 0 11 2.5

3

SBP NA NA 0 18 25.5

4


Achievement level Average cut

Standard error

Raw

Min score

Max Percent of students






PD 87.1 2.1 NA NA NA Proficient 56.2 2.3 57 86 45.9 Partially Proficient 32.5 1.2 33 56 30.6 SBP NA NA 0 32 23.6


cut Standard

error Raw

Min

score

Max

Percent of students



PD 128.2 4.3 129 129 2.6 Proficient 88.6 6.9 89 128 30.7 Partially Proficient 46.6 3.4 47 88 43.9 SBP NA NA 0 46 22.8

Grade

4

SBP NA NA 0 14 8.4

5

SBP NA NA 0 12 3.4

6

SBP NA NA 0 31 30.3

7

SBP NA NA 0 17 8.7

10

SBP NA NA 0 21 17.0

11

Grade

Table 3-3. 2010 PAAP Standard Setting: Round 1 Science

5

SBP NA NA 0 23 13.8

8

SBP NA NA 0 29 8.7

11

Table 3-4. 2010 PAAP Standard Setting: Round 1 Writing

Grade


cut Standard

error Raw

Min

score

Max

Percent of students

PD 23.0 0.2 23 23 16.2 4 Proficient 16.2 0.7 17 22 29.9


continued


Grade Achievement level Average cut

Standard error

Raw

Min score


4 SBP NA NA 0 8 29.0

PD 32.8 0.1 33 33 11.7

7 Proficient 15.7 0.6 16 32 60.2

Partially Proficient 11.4 0.2 12 15 14.6 SBP NA NA 0 11 13.6

PD 42.8 2.2 43 43 4.3

11 Proficient 19.5 3.9 20 42 56.0 Partially Proficient 8.8 2.2 9 19 24.1 SBP NA NA 0 8 15.5


The purpose of Round 2 was for panelists to discuss their Round 1 judgments as a group and determine

whether any revisions were necessary. A psychometrician shared the average cutpoint locations with the

panelists to help inform their group discussion and Round 2 ratings. It is important to note that although the

impact data and raw score ranges are presented in Tables 3-1 through 3-4, they were not shared with the

panelists after Round 1. Prior to the group discussion, the facilitator asked for a show of hands to determine the

number of panelists who had placed each portfolio into each achievement level; the facilitator then recorded

the results on chart paper. Starting with the first portfolio they disagreed on, the panelists began discussing the

categorization of the portfolios according to their initial ratings in the context of the classifications made by

other members of the group. Panelists were encouraged to share their own points of view as well as listen to

the comments of their colleagues. Facilitators made sure the panelists knew that the purpose of the discussion

was not to reach consensus; at every point throughout the standard setting process, panelists were asked to

provide their own best judgment. Once the discussions were complete, the panelists completed the Round 2

rating form.


When Round 2 ratings were complete, the Measured Progress data analysis team calculated the

average cutpoints for the room and associated impact data. The results of the panelists’ Round 2 ratings are

outlined in Tables 3-5 through 3-8.


Grade Achievement level Average

cut

Standard Raw score

error Min Max

Percent of students

PD 62.6 1.2 63 69 3.4

2 Proficient 44.7 1.2 45 62 22.7 Partially Proficient 23.0 1.9 24 44 54.0

SBP NA NA 0 23 19.9

continued



Standard error

Raw

Min score


PD 71.6 1.9 NA NA NA Proficient 51.5 1.2 52 69 37.6 Partially Proficient 25.6 1.4 26 51 39.5








cut Standard

error Raw

Min

score

Max

Percent of students




Grade

3

SBP NA NA 0 25 22.9

4

SBP NA NA 0 21 9.0

5

SBP NA NA 0 26 16.6

6

SBP NA NA 0 25 14.1

7

SBP NA NA 0 24 8.2

10

SBP NA NA 0 37 19.4

11

Grade


2

SBP NA NA 0 18 22.2

3

SBP NA NA 0 18 25.5

4

SBP NA NA 0 11 2.4

5 PD 41.5 0.4 42 46 42.3 Proficient 23.9 0.3 24 41 41.3

continued



Standard error

Raw

Min score








Standard error

Raw

Min score





Grade

5

SBP NA NA 0 12 3.4

6

SBP NA NA 0 28 26.5

7

SBP NA NA 0 16 8.3

10

SBP NA NA 0 29 22.3

11

Grade


5

SBP NA NA 0 23 13.8

8

SBP NA NA 0 32 9.1

11



cut Standard

error Raw

Min

score

Max

Percent of students

PD 22.9 0.0 23 23 16.2


SBP NA NA 0 9 29.0

PD 32.9 0.1 33 33 11.7 7 Proficient 15.9 0.3 16 32 60.2


continued



cut

Standard Raw score error Min Max

Percent of students

7 SBP NA NA 0 11 13.6

PD 42.1 0.7 43 43 4.3


SBP NA NA 0 11 15.9


The purpose of Round 3 was for panelists to discuss their Round 2 ratings as a whole group and, if

necessary, to revise their judgments. Prior to the group discussion, the facilitator once again asked for a show

of hands to determine the number of panelists who had placed each portfolio into each achievement level; the

facilitator recorded the results on chart paper. The group average cuts based on the Round 2 results were

presented. In addition, in this round, the group was presented with the impact data. The psychometrician

presented the group average cuts and impact data to the group and explained how to use the information as

they completed their Round 3 discussions. Panelists were encouraged to discuss whether the percentage of

students classified in each performance level seemed reasonable, given their perceptions of the students and

the knowledge, skills, and abilities demonstrated in the portfolios. As in Round 2, starting with the first

portfolio for which there was disagreement, the panelists discussed their ratings, with the impact data

considered as additional context for the discussion. Finally, after the discussions were complete, panelists were

given a final opportunity to revise their ratings. Once again, the facilitator reminded the panelists that they

should use their individual best judgment and that it was not necessary for them to reach consensus.


When Round 3 ratings were complete, the Measured Progress data analysis team once again

calculated the average cutpoints for the room and associated impact data. The results of the panelists’ Round

3 ratings are outlined in Tables 3-9 through 3-12 and in Figures 3-1 through 3-4.



cut Standard

error Raw

Min

score

Max

Percent of students

PD 64.0 0.9 64 69 3.4


SBP NA NA 0 23 19.9

PD 70.3 1.9 NA NA NA 3 Proficient 51.1 1.2 52 69 37.6


continued


Grade Achievement level Average cut

Standard error

Raw

Min

score

Max

Percent of students

3 SBP NA NA 0 26 24.4

PD 61.1 0.4 62 69 20.5

4 Proficient 41.2 0.3 42 61 42.3

Partially Proficient 21.5 0.0 22 41 28.2 SBP NA NA 0 21 9.0

PD 66.9 0.6 67 69 33.2

5 Proficient 51.7 0.3 52 66 31.0


SBP NA NA 0 26 16.6

PD 84.8 1.2 85 99 10.1

6 Proficient 55.9 1.5 56 84 52.0


SBP NA NA 0 24 13.2

PD 91.3 0.9 92 99 19.8

7 Proficient 49.9 1.0 50 91 58.5


SBP NA NA 0 24 8.2

PD 120.1 1.3 121 129 7.9

10 Proficient 85.6 1.6 86 120 32.5


SBP NA NA 0 35 18.3

PD 122.4 1.8 123 129 10.8

11 Proficient 77.5 0.6 78 122 39.0 Partially Proficient 29.4 2.3 30 77 34.4 SBP NA NA 0 29 15.8



cut Standard

error Raw

Min

score

Max

Percent of students









Grade


2

SBP NA NA 0 17 22.2

3

SBP NA NA 0 17 25.0

4

SBP NA NA 0 11 2.4

5

SBP NA NA 0 13 4.3

6

SBP NA NA 0 18 13.7

7

SBP NA NA 0 17 8.7

10

SBP NA NA 0 29 22.3

11



cut Standard

error Raw

Min

score

Max

Percent of students





cut Standard

error Raw

Min

score

Max

Percent of students




Grade


5

SBP NA NA 0 23 13.8

8

SBP NA NA 0 32 9.1

11

Grade


4

SBP NA NA 0 9 29.0

7

SBP NA NA 0 11 13.6

11


ro 0

c

0

c

0

Partially Proficient

Figure 3-1. 2010 PAAP Standard Setting: Mathematics: Round 3 Results

0 0.....-

0co

C" 0 0) Q.)

+""'

u CD

c

CJ) +""'

Q.)

""C :::J

+""' CJ) "¢

'+- 0

+""'

Q.)

u I-

Q.)

a_ N

0

02 03 04 05 06 07 10 11 Proficient with Distinction

Proficient

Substantially Below Proficient Mathematics: Round 3 Results


0

u

0


Figure 3-2. 2010 PAAP Standard Setting: Reading: Round 3 Results

0 0 ........

0ro

0 0)

..Q....

)..

co u ID

·c-

..C...

J..).

c Q)

"0

..:..:.:.J.. 0 CJ) "¢

'+- 0

........ c Q)

Q)

a_ N

0

02 03 04 05 06 07 10 11 Proficient with Distinction

Proficient

Substantially Below Proficient Reading: Round 3 Results


Figure 3-3. 2010 PAAP Standard Setting: Science: Round 3 Results

0 0......

0co

0 0)

..Q....

)..

co 0 (.) CD

·c-

..C...

J..).

c Q)

"0

..:..:.:.J.. 0 CJ) "¢

'+- 0

........ c Q) (.)

Q) 0

11.. N

0

Proficient with Distinction Proficient

Partially Proficient Substantially Below Proficient

05 08 11

Science: Round 3 Results


c

0

c

0

Figure 3-4. 2010 PAAP Standard Setting: Writing: Round 3 Results

0 0) Q)

+

c"

o"'

(.)

c

0 0 .....--

0 CX)

0 <D

(/) +""'

Q)

""0 ::J

+""' (/) "¢

'+- 0

+""'

Q) (.) I-

Q)

a_ C'\1

0

Proficient with Distinction

Proficient Partially Proficient Substantially Below Proficient

04 07 11

Writing: Round 3 Results


3.12 Repeat Process for Second Grade and Content Area

(Mathematics and Reading Only)

As mentioned above, eight panels each recommended cutpoints for two grade levels in mathematics

and reading. For those panels, once they had completed the entire process for the first grade level, they (1)

completed a process evaluation, giving their perceptions of the standard setting process and results thus far,

and (2) repeated the entire process (except for the training evaluation) for the second grade level.

3.13 Evaluation

The measurement literature sometimes considers the evaluation process to be another product of the

standard setting process (e.g., Reckase, 2001). To provide evidence of the participants’ views of the standard

setting process, panelists were asked to complete questionnaires throughout the process. The results of the

evaluations are presented in Appendix I.


Chapter 4. TASKS COMPLETED AFTER STANDARD SETTING

Upon conclusion of the standard setting meeting, several important tasks were completed. These tasks

centered on reviewing the standard setting process and addressing anomalies that may have occurred in the

process or in the outcomes, presenting the results to MDOE and making any final revisions or adjustments.

4.1 Analysis and Review of Panelists’ Feedback

Upon completion of the evaluation forms, panelists’ responses were reviewed. In general, this review

did not reveal any anomalies in the standard setting process or indicate any reason that a particular panelist’s

data should not be included when the final cutpoints were calculated. The one exception was a panelist in

grade 10 mathematics who did not appear to adequately understand the standard setting task. The panelist

continually placed the Proficient cut at a higher achievement than the Proficient With Distinction cut.

Although, the panelist self-corrected this issue for grade 11 mathematics, the panelist’s ratings were removed

from the grade 10 mathematics calculations and are not reflected in the results presented in this report. It

appeared that all remaining panelists understood the rating task and attended to it appropriately.

4.2 Calculation of Recommended Cutpoints

At the end of the standard setting meeting, the Round 3 cuts were presented to the Department of

Education as the final results of the standard setting meeting. Following the standard setting, a few concerns

were raised about the cuts. Most important of these was that higher standards (from a raw score cut

perspective) were established by the panelists in grades 2 and 3 than in grades 4 and 5 despite the fact that

these grades are based on the same task banks, level of complexity, and raw score scale. In theory, the grade 2

and grade 3 cuts should represent similar achievement requirements as grades 4 and 5. In addition, because a

single task bank is used for High School and because of the population of interest, a single set of cuts for

these students seemed more appropriate. Finally, the Proficient cut was set at a higher level in grade 6

mathematics than in grade 7 mathematics. Given that these two grades are based on the same task banks, level

of complexity, and raw score scale, it was felt that proficiency in the two grades should represent similar

achievement requirements. Consequently, the following adjustments were made to the Round 3 raw score cuts

and are presented in Tables 4-1 through 4-2 and Figures 4-1 through 4-2.

Grade 4 raw score cuts were applied to grade 2 and grade 3.

A single set of cuts was established for High School by averaging the grade 10 and grade 11 raw

score cuts.

The grade 7 Proficient cut in mathematics was raised to equal the grade 6 Proficient cut in

mathematics.

Tasks After Standard Setting 22 2010 Maine PAAP Standard Setting Report


cut Standard

error Raw

Min

score

Max

Percent of students

PD 61.1 NA 62 69 3.4 Proficient 41.2 NA 42 61 30.7 Partially Proficient 21.5 NA 22 41 58

PD 61.1 NA 62 69 14.1 Proficient 41.2 NA 42 61 34.6 Partially Proficient 21.5 NA 22 41 40

PD 61.1 0.4 62 69 20.5 Proficient 41.2 0.3 42 61 42.3 Partially Proficient 21.5 0 22 41 28.2

PD 66.9 0.6 67 69 33.2 Proficient 51.7 0.3 52 66 31 Partially Proficient 26.8 0.2 27 51 19.3


PD 91.3 0.9 92 99 19.8 Proficient 55.9 NA 56 91 53.1 Partially Proficient 24.9 0.6 25 55 18.8

PD 121.25 NA 122 129 9 Proficient 81.55 NA 82 121 35.4 Partially Proficient 32.45 NA 33 81 38.4 SBP NA NA 0 32 17.1

Grade

Table 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results

2

SBP NA NA 0 21 8

3

SBP NA NA 0 21 11.2

4

SBP NA NA 0 21 9

5

SBP NA NA 0 26 16.6

6

SBP NA NA 0 24 13.2

7

SBP NA NA 0 24 8.2

HS



cut Standard

error Raw

Min

score

Max

Percent of students

PD 42.2 NA 43 46 7.6 Proficient 24.4 NA 25 42 53 Partially Proficient 11.5 NA 12 24 36.9

PD 42.2 NA 43 46 17.3 Proficient 24.4 NA 25 42 45.2 Partially Proficient 11.5 NA 12 24 34.6


PD 41.6 0.4 42 46 42.3 Proficient 23.9 0.2 24 41 41.3 Partially Proficient 13 0.3 14 23 12

PD 59.1 1 60 66 8.5 Proficient 30.8 0.8 31 59 62.8 Partially Proficient 18.9 0.7 19 30 15

PD 64 0.5 64 66 24.8 Proficient 37.3 1.5 38 63 44 Partially Proficient 17.1 0.7 18 37 22.5

PD 85 NA 85 86 4.2 Proficient 56.2 NA 57 84 40.2 Partially Proficient 33 NA 33 56 30.9 SBP NA NA 0 32 24.7

Grade

Table 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results

2

SBP NA NA 0 11 2.5

3

SBP NA NA 0 11 2.9

4

SBP NA NA 0 11 2.4

5

SBP NA NA 0 13 4.3

6

SBP NA NA 0 18 13.7

7

SBP NA NA 0 17 8.7

HS


ro 0

0

0

Figure 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results

0 0......

0co

C" 0 0) Q.)

+""'

u c.o c

CJ) +""'

Q.)

""C :::J

+""' CJ) 'V

'+- 0

+""'

Q.)

u I-

Q.)

a_ N

0

2 3 4 5 6 7 HS Proficient with Distinction

Proficient


Mathematics: Adjusted Results


ro 0

c

0

c

0

Figure 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results

0 0.....-

0 CX)

C" 0 0) Q.)

+""'

u CD

c

CJ) +""'

Q.)

""C :::J

+""' CJ) "¢

'+- 0

+""'

Q.)

u I-

Q.)

a_ N

0


Proficient


Reading: Adjusted Results


After carefully considering the above information, the MDOE remained concerned about a few of the

cut scores in mathematics and writing. Consequently, the department decided to make a final policy adjustment

to some of the cut scores. The resulting DOE-approved operational cut scores, and a more detailed explanation

outlining how the final cut scores were established can be found in Appendix J.

4.3 Preparation of this Standard Setting Report

Following final compilation of standard setting results, Measured Progress prepared this report, which

documents the procedures and results of the 2010 standard setting meeting in order to establish performance

standards for the Maine PAAP in mathematics, reading, science, and writing.


REFERENCES

American Educational Research Association, American Psychological Association, & National Council on

Measurement in Education (1999). Standards for educational and psychological testing. Washington,

DC: American Educational Research Association.

Cizek, G. J., & Bunch, M. B. (2007). Standard setting: Establishing and evaluating performance standards

on tests. Thousand Oaks, CA: Sage Publications.

Kingston, N., Kahl, S., Sweeney, K., & Bay, L. (2001). Setting performance standards using the body of

work method. Setting performance standards: concepts, methods, and perspectives (pp. 219–

248). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

Reckase, M.D. (2001). Innovative methods for helping standard-setting participants to perform their task:

The role of feedback regarding consistency, accuracy, and impact. In G. J. Cizek (Ed.) Setting

performance standards: concepts, methods, and perspectives (pp. 159-173). Mahwah, NJ: Lawrence

Erlbaum.

U.S. Department of Education (2009). Standards and assessments peer review guidance: Information and

examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: U.S.

Department of Education Office of Elementary and Secondary Education. Retrieved June 10, 2010

from the World Wide Web: www2.ed.gov/policy/elsec/guid/saaprguidance.pdf.

References 28 2010 Maine PAAP Standard Setting Report

APPENDICES

Appendices 29 2010 Maine PAAP Standard Setting Report

Appendix A—ACHIEVEMENT LEVEL DEFINITIONS

Appendix A—Achievement Level Definitions 1 2010 Maine PAAP Standard Setting Report

Personalized Alternate Assessment Portfolio (PAAP)

Achievement Level Definitions for Reading ~ Grade 2

Reading achievement level definitions describe the quality of a student’s responses on state-level

alternate assessments in relation to the alternate reading standards. These definitions serve as the

foundation for achievement level definitions for the alternate grade level expectations in reading.

Skills associated with these achievement level definitions may include some of the following:

• student initiated communication

through signs, symbols, pictures,

gestures, and/or oral language

• relating of symbols to the

objects/ideas they represent

• using phonemic awareness skills

• using context clues to determine

meaning of words

• reading aloud sight words

• putting key events in correct

sequence

• identifying setting or characters

• comprehension of information

gained through listening or

viewing

• retelling events in a story

Appropriate performance at grade two is clarified by the level of complexity within the standard.

Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the

consistent ability to read, comprehend, and interpret literary OR informational texts appropriate

to the student’s instructional level.

Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also generally demonstrates the

ability to read, comprehend, and interpret literary OR informational texts appropriate to the

student’s instructional level.

Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also

demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary

OR informational texts appropriate to the student’s instructional level.

Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary and/or demonstrates incorrect understanding of word meaning. The work also

demonstrates minimal or limited ability to read, comprehend, and interpret literary OR

informational texts appropriate to the student’s instructional level.















meaning of words


• distinguishing parts of a book

(e.g., front, top, bottom, title,

author)

• using explicitly stated information

from the text to answer questions

• recognizing a central idea from

text when presented with pictures

Appropriate performance at grade three is clarified by the level of complexity within the

standard.



























meaning of words


• putting key events in correct

sequence

• identifying setting or characters

• comprehension of information

gained through listening or

viewing

• retelling events in a story

Appropriate performance at grade four is clarified by the level of complexity within the standard.



























meaning of words


• distinguishing parts of a book

(e.g., front, top, bottom, title,

author)


from the text to answer questions

• recognizing a central idea from

text when presented with pictures

Appropriate performance at grade five is clarified by the level of complexity within the standard.
















Reading achievement level definitions describe the quality of a student’s responses on state-level alternate

assessments in relation to the alternate reading standards. These definitions serve as the foundation for

achievement level definitions for the alternate grade level expectations in reading.


• relating of symbols to the objects/ideas

they represent • using phonemic awareness skills

• using context clues, a dictionary and/or

glossary to determine meaning of words

• using word parts, phonics, knowledge

of sounds, syllable types and/or word

patterns to decode

• identifying synonyms, antonyms and/or

categorizing words



from the text to answer questions • recognizing a central idea from text

when presented with pictures

• identifying or describing characters,

setting, problems, solutions, events

and/or plot • making logical predictions

• making basic inferences

• paraphrasing or summarizing

Appropriate performance at grade six is clarified by the level of complexity within the standard.


The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary

and/or demonstrate understanding of word meaning. The work also demonstrates the consistent ability to

read, comprehend, and interpret literary OR informational texts appropriate to the student’s instructional

level.

Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or

demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,

comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.

Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar

vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the

incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts

appropriate to the student’s instructional level.

Substantially Below Proficient

The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary

and/or demonstrates incorrect understanding of word meaning. The work also demonstrates minimal or

limited ability to read, comprehend, and interpret literary OR informational texts appropriate to the









• relating of symbols to the objects/ideas

they represent • using phonemic awareness skills

• using context clues, a dictionary and/or

glossary to determine meaning of words

• using word parts, phonics, knowledge

of sounds, syllable types and/or word

patterns to decode


categorizing words


• using explicitly stated information from

the text to answer questions

• recognizing a central idea from text

when presented with pictures • obtaining information from the table

of contents, glossary, table of contents

• connecting information within a text

• identifying parts of a book (e.g.,

author, title, beginning, end)

• Paraphrasing or summarizing

• Drawing conclusions or making

inferences

Appropriate performance at grade seven is clarified by the level of complexity within the standard.

Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary

and/or demonstrate understanding of word meaning. The work also demonstrates the consistent ability to


level.

Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or



Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the



Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary

and/or demonstrates incorrect understanding of word meaning. The work also demonstrates minimal or





Achievement Level Definitions for Reading ~ High School






• using context clues, dictionary, glossary

and/or thesaurus to determine meaning

of words

• using word parts or phonics to decode • reading aloud sight words


categorizing words

• using explicitly stated information from

the text to answer questions


when presented with pictures

• identifying or describing characters,

setting, problems, solutions and/or

events

• identifying author’s basic message

and/or purpose • identifying narrator

• paraphrasing, summarizing, and/or

comparing/contrasting

• making logical predictions

• making inferences


when presented with pictures • obtaining information from the table

of contents or glossary

• connecting information within a text • synthesizing information

Appropriate performance at high school level is clarified by the level of complexity within the standard.

Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the consistent ability to


level.

Proficient

The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or




The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar

vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the



Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary and/or demonstrates incorrect understanding of word meaning. The work also demonstrates minimal or





Achievement Level Definitions for Mathematics ~ Grade 2

Mathematics achievement level definitions describe the quality of a student’s responses on statelevel

alternate assessments in relation to the alternate mathematics standards. These definitions

serve as the foundation for achievement level definitions for the alternate grade level

expectations in mathematics.


• matching small collections of equivalent sets

• matching simple 2-D shapes

• comparing two items based on multiple attributes

• identifying measurement tools

• copying simple patterns

• collecting data

• identifying the outcome of an event

Appropriate performance at grade 2 is clarified by the level of complexity within the standard.

Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses

demonstrate the ability to utilize information and solve problems including implementing

strategies, accurately performing procedures and providing solutions.

Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections among central ideas. The student’s responses demonstrate basic ability to solve

problems, including performing procedures and providing solutions. The student’s work may

contain minor errors.

Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited

ability to solve problems, including performing procedures and providing solutions. Problem

solving strategies may be flawed and procedures preformed inaccurately.

Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate connections among central ideas. The student’s responses demonstrate minimal ability

to solve problems. Problem solving strategies may be flawed or inappropriate and there may be

many omissions.









• indicating or labeling collections of

equivalent sets

• reading, writing, and counting

numbers

• recognizing place value

• skip counting

• comparing two items based on

multiple attributes

• identifying measurement tools

• copying and/or extending patterns

• collecting data

• identifying the outcome of an

event

Appropriate performance at grade three is clarified by the level of complexity within the

standard.












many omissions.









• ordering and comparing numbers

• identifying two-digit numbers

• solving addition and subtraction

problems involving one-digit

numbers

• identifying 2-D shapes

• comparing two items based on

multiple attributes

• identifying congruent figures

• identifying and using measurement

tools

• copying and extending simple

patterns

• collecting, organizing, and/or

interpreting data

• identifying the outcome of an

event













many omissions.









• matching and/or identifying coins

• decimal notation

• comparing two items based on multiple attributes

• identifying and using measurement tools

• copying and extending patterns













many omissions.









• indicating, comparing, and/or ordering rational numbers limited to fractions with

denominators of 2, 3, 4, and/or 5

• identifying and/or classifying 2-D shapes and/or angles

• matching quantities that are equal

• finding the value that will make an open sentence true

Appropriate performance at grade six is clarified by the level of complexity within the standard.












many omissions.









• ordering and comparing whole numbers

• using measurement tools and estimating outcomes

• computing equivalencies

• using more, less, equal, and/or other comparisons to analyze data or solve problems

Appropriate performance at grade seven is clarified by the level of complexity within the

standard.

Proficient with Distinction: The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses











many omissions.



Achievement Level Definitions for Mathematics ~ High School

Mathematics achievement level definitions describe the quality of a student’s responses on state-level

alternate assessments in relation to the alternate mathematics standards. These definitions serve as the

foundation for achievement level definitions for the alternate grade level expectations in mathematics.


• solving multi-step addition, subtraction, multiplication and/or division problems

involving whole numbers, fractions,

decimals, percents and/or ratios

• describing or illustrating the

relationships between the four operations

• copying, extending, and describing

patterns • writing rules for finding specific cases

of a linear or nonlinear relationship

• finding the value that will make an open

sentence true

• representing unknown quantities with

letters

• simplifying and writing linear

algebraic expressions • collecting, arranging, interpreting,

and/or analyzing data to formulate or

justify conclusions, make predictions

or solve problems

• using more, less, equal, and/or other

comparisons to analyze data or solve

problems,

• using measures of central tendency or

range to analyze situations or solve problems

Appropriate performance at the high school level is clarified by the level of complexity within the

standard.

Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the

ability to make connections among central ideas. The student’s responses demonstrate the ability to utilize

information and solve problems including implementing strategies, accurately performing procedures and

providing solutions.

Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections

among central ideas. The student’s responses demonstrate basic ability to solve problems, including

performing procedures and providing solutions. The student’s work may contain minor errors.

Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited ability to solve

problems, including performing procedures and providing solutions. Problem solving strategies may be

flawed and procedures preformed inaccurately.

Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate

connections among central ideas. The student’s responses demonstrate minimal ability to solve problems.

Problem solving strategies may be flawed or inappropriate and there may be many omissions.



Achievement Level Definitions for Science ~ Grade 5

Science achievement level definitions describe the quality of a student’s responses on state-level alternate

assessments in relation to the alternate science standards. These definitions serve as the foundation for

achievement level definitions for the alternate grade level expectations in science.


• identifying night and day

• identifying the Sun

• identifying the Earth’s Moon

• identifying the position of the sun at

different times

• identifying or drawing different phases

of the Moon

• identifying weather through observation

• identifying different forms that water

can take in the weather

• matching weather to the effects it can

have on the Earth’s surface • indentifying pictures or descriptions

of given animals and plants

• indentifying plants and animals, and

components of the environments that

animals depend on for food and shelter


Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in science, including the

ability to make connections among central ideas. The student’s response demonstrates the ability

to utilize information and solve problems and explain central concepts with clarity and accuracy.

Proficient The student’s work demonstrates a general understanding of essential concepts in science and connections among central ideas. The student’s response demonstrates the ability to utilize

information and solve problems and explain central concepts. Student work may contain minor

errors.


The student’s work demonstrates incomplete understanding of essential concepts in science and

inconsistent connections among central ideas. The student’s response demonstrates some ability

to utilize information and solve problem. The quality of the responses is inconsistent. Explanation

of concepts may be incomplete or unclear.

Substantially Below Proficient The student’s work demonstrates limited understanding of essential concepts in science and infrequent or inaccurate connections among central ideas. The student’s responses demonstrate

minimal ability to solve problems. Explanations are illogical, incomplete, or missing. There are

many inaccuracies.



Achievement Level Definitions for Science ~ Grade 8





• identifying or demonstrating ways

objects can move • identifying that an object’s motion can

be changed by pushing or pulling

• identifying or describing wave motions,

earthquakes, vibrations, and/or water

waves • identifying human body parts

• matching animals and/or plants to their

parts

• identifying parts that allow living

things to meet basic needs • identifying that some living things are

made of one cell and some are made

of many cells

• identifying parents and offspring

• demonstrating an understanding of

life cycles, and/or identifying similar

and different characteristics of

offspring and parents

Appropriate performance at grade eight is clarified by the level of complexity within the standard.

Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in science, including the



Proficient The student’s work demonstrates a general understanding of essential concepts in science and connections among central ideas. The student’s response demonstrates the ability to utilize


errors.

Partially Proficient The student’s work demonstrates incomplete understanding of essential concepts in science and




Substantially Below Proficient The student’s work demonstrates limited understanding of essential concepts in science and infrequent or inaccurate connections among central ideas. The student’s responses demonstrate


many inaccuracies.



Achievement Level Definitions for Science ~ High School





• sorting objects into groups using

physical properties • describing physical properties of objects

and materials

• using observable characteristics to

describe physical changes

• identifying chemical and physical changes

• identifying organisms that are similar

and different based on external features

• describing how plants and/or animals

look

• describing ways in which the needs of

a plant and/or animal are met by its

environment

• sorting living things based on external

features

• matching organisms to the

environment in which they live

• identifying organisms that once lived

on Earth but no longer exist • identifying examples of fossils and/or

explaining how fossils are used to

help us understand the past

Appropriate performance at the high school level is clarified by the level of complexity within the

standard.


The student’s work demonstrates an understanding of essential concepts in science, including the



Proficient

The student’s work demonstrates a general understanding of essential concepts in science and

connections among central ideas. The student’s response demonstrates the ability to utilize


errors.

Partially Proficient The student’s work demonstrates incomplete understanding of essential concepts in science and




Substantially Below Proficient The student’s work demonstrates limited understanding of essential concepts in science and

infrequent or inaccurate connections among central ideas. The student’s responses demonstrate


many inaccuracies.



Achievement Level Definitions for Writing ~ Grade 4

Writing achievement level definitions describe the quality of a student’s responses on state-level

alternate assessments in relation to the alternate writing standards. These definitions also serve

as the foundation for achievement level definitions for the alternate grade level expectations in

writing.


• identifying pictures or symbols to relate an experience, event, or idea

• composing responses

• using pictures to create an understandable story line with a beginning and end

• using pictures to identify and/or create characters


Proficient with Distinction The student’s performance demonstrates understanding of essential concepts in writing. The student’s work demonstrates the ability to compose a response that is well-organized, accurate

and focused.

Proficient The student’s performance demonstrates an understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is organized, accurate and

focused. Some errors may occur but do not interfere with meaning.

Partially Proficient The student’s performance demonstrates an incomplete understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that may be

inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that

interfere with meaning.

Substantially Below Proficient The student’s performance demonstrates limited understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is unorganized, inaccurate

and unfocused. Frequent errors may occur that interfere with meaning.



Achievement Level Definitions for Writing ~ Grade 7




writing.


• composing and sharing related

responses to convey needs

• representing facts through pictures

• using pictures to illustrate details or

information related to topic

• sorting or grouping facts and/or

ideas within a given category

• using pictures and/or words to create

meaning

• including details or information

relevant to topic

• using a given organizational

structure for grouping facts and/or

ideas

• using sufficient details or pictures

to illustrate facts

• using basic transition words

• providing a concluding sentence

Appropriate performance at grade seven is clarified by the level of complexity within the

standard.


and focused.










Achievement Level Definitions for Writing ~ High School




writing.


• using phonemic awareness and letter

knowledge to represent initial or

final consonant sounds,

• writing recognizable phrases or short

sentences to show understanding of

text

• using prior knowledge or references

to text to respond to questions or

when reading

• using a beginning and an ending to

organize ideas

• applying basic capitalization and

punctuation rules

• correctly spelling high frequency

words

• writing a variety of simple

sentences

• recognizing or applying English

spelling rules

• stating and maintaining focus

when responding to questions

Appropriate performance at the high school level is clarified by the level of complexity within

the standard.


and focused.









Appendix B—AGENDAS

Appendix B—Agendas 1 2010 Maine PAAP Standard Setting Report

MAINE ALTERNATE ASSESSMENT STANDARD SETTING

June 28 & 29, 2010

AGENDA

Day 1: Monday, June 28th

1st

Grade

All panelists together 8:00 – 8:30 Registration & continental breakfast

8:30 – 8:45 Welcome from Maine’s Department of Education (MDOE)

Introduction of MDOE staff

Introduction of Measured Progress staff 8:45 – 10:00 Overview of the Maine Alternate Assessment

Overview of Standard Setting Process

Panelists break into their respective workgroup rooms 10:00 – 10:15 Break; panelists move to their grade level/content area workgroup rooms 10:15 – 12:00 Introductions of facilitator and panelists

Review of PAAP materials (AGLEs and student samples)

Review and discuss Achievement Level Descriptors

Training Evaluation

All panelists together 12:00 – 12:45 Lunch

Panelists return to their respective workgroup room 12:45 – 4:00 Rounds 1, 2 & 3

Procedural Evaluation Break as needed (approximately 2:00 pm or between rounds)

Day 2: Tuesday, June 29

th

2nd

Grade

All panelists together

7:30 – 8:00 Continental breakfast

Panelists break into their respective workgroup rooms 8:00 – 8:45 Review process, answer questions and check for understanding

8:45 – 12:00 Review of PAAP materials (AGLEs and student samples)


Round 1

Begin Round 2


Panelists return to their respective workgroup room 12:45 – 3:30 Continue with Rounds 2 & 3

Break as needed (approximately 2:00 pm or between rounds)

All panelists together 3:30 – 4:00 Cross grade panel

Final Evaluation


MAINE ALTERNATE ASSESSMENT STANDARD SETTING

June 30, 2010

AGENDA

7:30 – 8:00 Registration/room assignments & continental breakfast

Panelists break into their respective workgroup rooms

8:00 – 8:45 Introductions of facilitator and panelists

Review process, answer questions and check for understanding

8:45 – 12:00 Review of PAAP materials (AGLEs and student samples)


Round 1

Begin Round 2


Panelists return to their respective workgroup room

12:45 – 4:00 Continue with Rounds 2 & 3

Final evaluation

Break as needed (approximately 2:00 pm or between rounds)


Appendix C—NONDISCLOSURE FORM

Appendix C—Nondisclosure Form 1 2010 Maine PAAP Standard Setting Report

Maine Personalized Alternate Assessment Portfolio

Nondisclosure Agreement for Standard Setting Panelists

The student work associated with the Maine Personalized Alternate Assessment Portfolio (PAAP) is confidential material. As such, the student portfolio may not be copied, shared, or discussed for any reason other than to score the student work. It is the policy of the Maine Department of Education that student portfolios be treated as private and secure material.

The undersigned is a PAAP In-State Standard Setting participant authorized to view PAAP material and hereby agrees to be bound to the terms of this agreement restricting the disclosure of said materials.

Name (printed)

Name (signature)

Date

Appendix C—Nondisclosure Form 3 2010 Maine PAAP Standard Setting Report

Appendix D—SAMPLE RATING FORM

Appendix D—Sample Rating Form 1 2010 Maine PAAP Standard Setting Report

Round: ID Number:

Maine PAAP Reading, Grade 2

SBP PP P PWD

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

Appendix D—Sample Rating Form 3 2010 Maine PAAP Standard Setting Report

Appendix E—SAMPLE EVALUATION

Appendix E—Sample Evaluation 1 2010 Maine PAAP Standard Setting Report

Str

ongly

Dis

agre

e

Dis

agre

e

Undec

ided

Agre

e

Str

ongly

Agre

e

Content Area:

Grade:

Standard Setting Training Evaluation

The purpose of this evaluation form is to obtain your feedback about the training you have received.

Please complete the information below. Do not put your name on the form. We want your feedback

to be anonymous.

Please mark the appropriate box for each statement.

I understand the goals of the standard setting meeting. □ □ □ □ □

I understand the procedures we are using to set standards. □ □ □ □ □

I understand how to use the standard setting materials. □ □ □ □ □

I understand the differences between the achievement levels. □ □ □ □ □

I understand how to make the cut score judgment. □ □ □ □ □

I know what tasks to expect for the remainder of the meeting. □ □ □ □ □

I am confident in my understanding of the standard setting task. □ □ □ □ □

Please indicate any areas in which you would like more information before you continue.

Please indicate any questions you may have about the remainder of the standard setting meeting.


Not

at a

ll

infl

uen

tial

Str

ongly

Dis

agre

e

Dis

agre

e

Undec

ided

Agre

e

Extr

emel

y

infl

uen

tial

Str

ongly

Agre

e

Content Area:

Grade:

Standard Setting Procedural Evaluation


I understood how to make the cut score judgments. □ □ □ □ □

I understood how to use the materials provided. □ □ □ □ □

I understood how to record my judgments. □ □ □ □ □

I think the procedures make sense. □ □ □ □ □

I am sufficiently familiar with the assessment. □ □ □ □ □

I understand the differences between the achievement levels. □ □ □ □ □


Please rate the influence of the following when setting standards.

The achievement level descriptors. □ □ □ □ □

The state content standards. □ □ □ □ □

My perception of the difficulty level of the assessment. □ □ □ □ □

The student responses. □ □ □ □ □

My experience working with students. □ □ □ □ □

What materials, information, or procedures were most influential in your placement of the cut scores?

Why?


Str

ongly

Dis

agre

e

Dis

agre

e

Undec

ided

Agre

e

Str

ongly

Agre

e

Content Area:

Grade:

Standard Setting Final Evaluation

Please complete the information below. Your feedback will provide a basis for evaluating the training,

methods, and materials. Do not put your name on the form. We want your feedback to be anonymous.

Gender: Male □ Female □ Race/ethnicity: White □ Black □ Hispanic □ Asian □ Pacific Islander □ American Indian □ Years of experience in education: 0-5 □ 5-10 □ 10-15 □ More than 15 □ Area of Expertise (Check all that apply): Students with Disabilities □

Students with Limited English Proficiency □ Economically Disadvantaged Students □ Gifted and Talented Students □ General Education □


I understood the goals of the standard setting meeting. □ □ □ □ □

I understood the procedures we used to set standards. □ □ □ □ □

The facilitator helped me understand the process. □ □ □ □ □

The materials contained the information needed to set standards. □ □ □ □ □

I understood how to use the materials provided. □ □ □ □ □

The achievement level descriptors were clear. □ □ □ □ □

I understood how to make the cut score judgments. □ □ □ □ □

I understood how to use the feedback provided after each round. □ □ □ □ □

I understood how to use the impact data. □ □ □ □ □

I understood how the cut scores were calculated. □ □ □ □ □

The facilitator was able to get answers to my questions. □ □ □ □ □

Sufficient time was allotted for training on the standard setting tasks. □ □ □ □ □

Sufficient time was allotted to complete the standard setting tasks. □ □ □ □ □


Not

at a

ll

infl

uen

tial

Not

at a

ll

use

ful

Extr

emel

y

infl

uen

tial

Extr

emel

y

use

ful

Content Area:

Grade:

The facilitator helped the standard setting process run smoothly. □ □ □ □ □

Please rate the usefulness of each of the following:

The opening session. □ □ □ □ □

The small group activities. □ □ □ □ □

Becoming familiar with the assessment. □ □ □ □ □

Articulating the differences between the achievement levels. □ □ □ □ □

Discussions with other participants.

Please rate the influence of the following when setting standards.

□ □ □ □ □

The achievement level descriptors. □ □ □ □ □

My expectations of students. □ □ □ □ □

The difficulty of the test materials. □ □ □ □ □

The student responses. □ □ □ □ □

My experience in the field. □ □ □ □ □

Discussions with other participants. □ □ □ □ □

Cut scores of other participants. □ □ □ □ □

Impact data. □ □ □ □ □

Please provide any additional comments about the standard setting process or suggestions as to how the

training and process could be improved.


APPENDIX F—OPENING SESSION POWERPOINT

Appendix F—Opening Session PowerPoint 1 2010 Maine PAAP Standard Setting Report

Slide 1

Slide 2

Maine’s Personalized

Alternate Assessment

Portfolio (PAAP)

Standard Setting

Introductions

Maine’s Department of Education

- Susan Fossett, PAAP and Accommodations

Coordinator

- Peter Bernard, PAAP Assistant

Measured Progress (Program Mgt.)

- Susan Izard, Division Director

- Sharon Houle, PAAP Program Manager

- Stephanie Arroyo, PAAP Specialist

- Michelle Couture, PAAP Program Assistant

Slide 3

Data

Introductions (cont.)

- Liz Burton, Psychometrician

- Jennifer Dunn, Psychometrician

- Kevin Froton, Data Processing

Slide 4

Introductions (cont.)

Facilitators (Monday and Tuesday)

Grade 2/3 Reading: Amanda Breitmaier

Grade 2/3 Mathematics: Alicia Cuttle

Grade 4/5 Reading: Jaime Alford

Grade 4/5 Mathematics: Stephanie Arroyo

Grade 6/7 Reading: Tim Greenlaw

Grade 6/7 Mathematics: Susan Izard

Grade 10/11 Reading: Jake Goldsmith

Grade 10/11 Mathematics: Kristen Cole

Slide 5 Introductions (cont.)

Facilitators (Wednesday):

Science

Grade 5: Alicia Cuttle

Grade 8: Stephanie Arroyo

Grade 11: Kristen Cole

Writing

Grade 4: Amanda Breitmaier

Grade 7: Susan Izard

Grade 11: Jake Goldsmith

Slide 6

Agenda

8:30 – 10:00 am General Session

10:00 am – 12:00 pm Breakout Rooms

12:00 – 12:45 pm Lunch

12:45 – 4:00 pm Breakout Rooms

Breaks as needed in Breakout Rooms

Slide 7 The PAAP is

What is the PAAP?

- designed for students with significant cognitive

disabilities who meet participation criteria;

- administered during the “teaching year” at the

same grade levels as students in general

education are assessed in October via NECAP;

- a collection of student work;

- assessed during the PAAP assessment window

December 1st through April 30th;

Slide 8

The PAAP is

What is the PAAP?

- tasks that teachers download to assess

students at an instructionally appropriate time;

- tasks selected by teachers that best fit their

students (using the test blueprint as a guide);

- scored using the PAAP Scoring Rubric to

obtain student achievement levels which are

then used to determine reportable scores; and

- reflective of input from an instructional team.

Slide 9 2009-10 Test Blueprint

Slide 10

Visual Guide to PAAP Requirements

Slide 11 Steps to Administer a PAAP

STEP 1: The teacher/IEP team determines a student is

eligible to participate for each content area.

STEP 2: The teacher reviews the Alternate Grade Level Expectations (AGLE) document to determine which

AGLE/Indicator and Level of Complexity (LoC) is

appropriate from the eight LoCs.

STEP 3: The teacher selects/downloads tasks from the

PAAP Task Bank (including graphics/passages). STEP

4: The teacher administers the tasks to the student. STEP

5: The teacher corrects/grades the student work and

provides details regarding the level of assistance

provided to the student.

Slide 12

Slide 13

Slide 14

Slide 15

What does a Task look like?

1. Entry Slip

2. Task Description

3. Student Work (template)

and graphics

4. Task Summary

Slide 16

Slide 17

Slide 18

The Entry Slip • Teacher fills in Student

Name and Grade.

• The Content Area, AGLE, Level of Complexity (LoC), and the AGLE/Indicator is pre-populated.

• By design, text provided in gray shaded cells is information for the teacher and is pre- populated.

• Areas with white or no shading indicates the teacher should fill in the missing information.

• LoC 1 for ELA

Template Pages

Slide 19

Slide 20 Task Description Page

• The Task Description page repeats the Content Area, AGLE/Indicator, LoC information.

• The Task Title, AGLE Page #, Task # has been added to the top identifying information.

• Prior Knowledge, Description of Task, Directions for Task Administration, and Responses Expected are provided to ease teacher administration.

Slide 21 Work Template Page

• LoC 1 Work

are always a chart

similar to this one.

(Reading & Writing)

always includes

color images.

Option 2 – Always Available at LoC 1 • Option 2 allows for fewer

of the item sets to be used

multiple times when

necessary to match the level of student knowledge.

• The items must be selected from those already provided

in the Task.

• Option 2 allows for the use of

a combination of selected sets that total six items (e.g.,

two sets each used on 3

different occasions…or

they can use the same set

on 6 different occasions).

Slide 22

How the Item 1 is Presented to the

Student

Slide 23

Slide 24 Liam 12/11/10

Option 2

sleeping

smiling

sleeping

smiling

sleeping

smiling

eating

person laughing

eating

person laughing

eating

person laughing

sleeping

sleeping

sleeping

eating

eating

eating

sleCeping

smilXing

slCeeping

eatCing

Ceating

laughXing

Liam 12/11/10

67

The Completed Work Template 1. Teacher fills in Liam 12/11/10

student name, date,

and student

responses. C

X

2. Teacher corrects C

C student responses

C using the Responses

Expected from X

Student section on

the Task Description.

response

Hand-over-Hand

Slide 25

Slide 26 Scoring Level of Accuracy

1. Determine the Level of Accuracy based on

the corrected student work and Data Key. – Point values are predetermined. NO PARTIAL POINTS

unless noted.

– % are pre-calculated.

Task 1 Summary

Slide 27 Scoring Level of Assistance

Unscorable:

●

● Altering Items/Tasks (task no longer connects to

the AGLE.)

Level of Assistance Score of 1:

● Modeling

● Demonstrating a response similar to the desired

X

Liam required modeling of Item 1 to understand what was being

asked of her. He proceeded without further assistance.

Scoring Level of Assistance

2. Determine Level of Assistance based

Administration Handbook.

.



• Use of Option 2 (LoC 1 only) to use fewer of the

item sets multiple times to match student

knowledge

• Limiting a student’s response (outside of LoC 1 at

Option 2) by removing one response option

• Use of clarifying questions to stimulate student

thought to the specific task without providing clues

to specific answers



• Independent

• Providing encouragement

• Completing Task by using

augmentative/alternative means of

communication

• Repeating directions

• Reacting to a student

• Rereading a passage (except for required

reading)

• Reminding a student to stay focused

Slide 30

upon criteria outlined in the PAAP

Scoring Level of Assistance

3.The bottom of the Task Summary is a reminder to

the teacher if the Student Work, Level of Accuracy,

and Level of Assistance boxes are not completed

and submitted together by Task, the Task becomes

UNSCORABLE.

Slide 31

Slide 32 Tasks 2 and 3

• Usually at LoC 1, Tasks 2 and/or 3 are a repeat

of Task 1 with different pictures, with the same

concept being assessed.

• As you move to higher LoCs, Tasks 2 and 3

build off Task 1, getting progressively more

difficult/complex. The skills may be different

depending on what is required of the LoC.

• Reading and Writing require 3 Tasks per AGLE

Entry.

• Mathematics and Science require 2 Tasks per

AGLE Entry.

Slide 33

What does the PAAP Assess?

√ Level of Complexity

√ Level of Accuracy

√ Level of Assistance

A final reminder…

☺ PAAPs were scored by two scorers,

sometimes three.

☺ Do not attempt to rescore the samples

your reviewing!

☺ You may disagree about the order of the

PAAPs; that’s fine.

☺ You need to stay focused on the task at

hand!

Slide 34 Who scored the PAAPs?

● Measured Progress scorers scored 2,024

PAAPs in early May using the PAAP Task

Scoring Rubric. Each PAAP was scored twice

and sometimes a third time if scores between

Scorer #1 and #2 were not an exact match.

● Measured Progress provided training.

● All scorers passed a qualifying test.

● MDOE staff were present at scoring and

available to answer questions.

Slide 35 PAAP Task

Scoring Rubric

Slide 36

Slide 37

Slide 38

Now the process…

Today’s Training

In today’s session we will cover:

1. an overview of standard setting;

2. details of the Body of Work process as it

will be implemented for the PAAP; and

3. your role in this process.

Note:

This session is intended to be an overview.

Your facilitator will give you more details and will guide you through the process step by step.

Slide 39 Logistical Overview Monday/Tuesday

Reading 2/3

Reading 4/5

Reading 6/7

Reading 10/11

Math 2/3

Math 4/5

Math 6/7

Math 10/11

Wednesday

Science 5

Science 8

Science 11

Writing 4

Writing 7

Writing 11

Proficient

Slide 40

Overview of Standard Setting

Slide 41 Content

Standards

vs.

Achievement

Standards

Content standards (AGLE) = “What”

Describe the knowledge and skills students are expected to

demonstrate by content area and grade span

Achievement standards (e.g., Proficient) =

“How well”

Describe attributes of student performance based on Achievement Level Descriptors

Slide 42

What is Your Job?

To recommend cut scores for each of the

achievement levels that will be used to report

PAAP results:




Based on Achievement Level

Descriptors, you will recommend cut

scores…

Cut score Cut score Cut score needed needed needed

Substantially Partially Proficient Proficient Below Proficient with Proficient Distinction

Achievement Continuum

Slide 43 We are trying to determine What knowledge, skills, and abilities (KSAs)

need to be demonstrated to be classified in each achievement level?

How much is enough?

What portfolio evidence corresponds to:



Proficient


Slide 44

Achievement Continuum

Substantially

Below Proficient


Proficient Proficient

with Distinction

Slide 45

Slide 46

General Phases of Standard Setting

Data-collection

Policy-making/Decision-making

Slide 47

Final Recommendations

Your recommendations will be reviewed and

presented to the policy makers, who are

responsible for final adoption of the cut

scores.

The recommendations may be accepted or

modified by the Maine Department of Education including the Commissioner of

Education and the Technical Advisory Committee.

Slide 48

Overview of Standard Setting

Method

Slide 49 Cut Score Recommendations

Provide data to establish the

following cut scores:



Proficient


Cut Score

Cut Score

Cut Score

Slide 50 How: The Body of Work Method

Examine student work and make a judgment

regarding the achievement level to which the

student work most closely corresponds.

Student Work Samples (portfolios or PAAPs) 30 to 40 student PAAPs

Your job is to classify each portfolio into the

achievement level in which you feel it

belongs.

Slide 51 Why the Body of Work?

Allows panelists to use samples of actual student

work to make their determinations

Is especially useful for assessments that consist

primarily or entirely of performance-based

items

Has been used successfully for setting standards

on similar assessments in the past

Has resulted in defensible cut points

Slide 52 General Process

Classify each portfolio into

one of 4 achievement

levels based on the

following:

Achievement Level Descriptors

KSAs measured by the portfolios

How the students performed on the portfolios

Slide 53

Before you start classifying portfolios….

You will need to become familiar with:

√ Alternate Grade Level Expectations (AGLEs)

√ Achievement Level Descriptors

What each level means

Identify the knowledge, skills and abilities

necessary to be classified in each level

√ Student PAAPs/portfolios

Understand the knowledge, skills and

abilities demonstrated in the work samples

Slide 54 Achievement Level Descriptors Individual review of Achievement Level Descriptors

Group discussion of what performance in each

achievement level looks like

Create bulleted lists of

The knowledge, skills, and abilities a student must demonstrate to be classified in each achievement level

The knowledge, skills, and abilities that distinguish one achievement level from another

You must reach consensus as a group about

the KSAs that define student performance at each achievement level.

Slide 55

Student Portfolios You will classify 30 to 40 student portfolios.

The portfolios cover the range of possible total

scores and are presented in order from lowest

(e.g., Sample #1) to highest (e.g., Sample #35)

total raw score.

Each portfolio has been selected because it

shows typical types of evidence submitted for

students who received a given total score.

Slide 56 Your Task

Think about a student who demonstrates the

KSAs for each level.

Classify each portfolio into the level you feel it

belongs:



Proficient


Slide 57

Slide 58

Slide 59

Please Note: You may disagree about the order of the portfolios; that’s fine.

You will categorize the portfolios as you see fit, whether your ratings agree with the order or not.

However, it is not your job to rescore the portfolios; you need to stay focused on the task at hand.

Slide 60

Round 1 Working Individually:

Review each portfolio

Focus on the knowledge, skills and abilities being

demonstrated in the portfolio

Determine which Achievement Level Descriptor best

matches the knowledge, skills and abilities


Classify the portfolio into the appropriate

achievement level

Complete the rating form

of the discussions.

Slide 61 Round 2 Working as a Group:

Discuss your portfolio classifications in relation to

The average round 1 results

The other panelists

The knowledge, skills and abilities

Working Individually:

Determine which Achievement Level Descriptor best matches the knowledge, skills and abilities demonstrated in the portfolio

Classify the portfolio into the appropriate achievement level


Slide 62

Round 3 Working as a Group:

Discuss your portfolio classifications in relation to

The round 2 results & impact data

The other panelists

The knowledge, skills and abilities

Working Individually:

Determine which Achievement Level Descriptor best matches the knowledge, skills and abilities


Classify the portfolio into the appropriate achievement level


Slide 63 A few reminders

It is not necessary for panelists to reach

consensus as to how the portfolios should be

categorized.

You should be open-minded when listening to

your colleagues’ rationales for their ratings.

You may or may not change your mind as a result

We want each panelist to use his or her own best

judgment in each round of rating.

Slide 64 Cross-Grade Policy Forum

(Tuesday afternoon)

After all groups have completed Round 3 for the

second grade level, the groups for each

content area (reading and mathematics) will

meet together to look at results across grades

and provide feedback.

Slide 65

Evaluation

At several different points in the process, we

will ask you to complete an anonymous

evaluation of the standard setting procedures.

Your honest feedback is important for

improving future standard settings, and for

evaluating the results of this one.

Slide 66

Questions about the Body of Work

Method?

Slide 67

Before you break into groups…

Slide 68 Top 10 Misconceptions About

Standard Setting

10. Standard setting is a great opportunity to

review and revise the PAAP.

9. Standard setting is the same thing as scoring.

8. This is a good time to discuss PAAP

administration policy.

7. This is a good time to revise the content

standards.

6. This is a good time to revise the PAAP

Achievement Level Descriptors.

Slide 69

Top 10 Misconceptions About Standard

Setting 5. This is a good time to discuss effective

teaching strategies.

4. Only scholars and researchers are qualified

to do this work.

3. Only educators are qualified to do this work.

2. The process is rigged.

1. Disagreement is bad.

Slide 70

What Next?

☺ Some meeting logistics

☺ After this session, you will break into grade

groups and complete the standard setting

process! – First grade/content

• Review the portfolios

• Discuss the Achievement Level Descriptors

• Round 1, 2, 3

– Repeat for second grade/content

– Cross grade policy forum

– Evaluation

☺ Some lucky folks will – Repeat for third grade/content

Slide 71 Room Assignments

Please refer to your hotel map with

room assignments in your folder.

Appendix G—FACILITATORS’ SCRIPTS

Appendix G—Facilitators’ Scripts 1 2010 Maine PAAP Standard Setting Report

GENERAL INSTRUCTIONS FOR GROUP FACILITATORS MAINE PAAP STANDARD SETTING

READING AND MATHEMATICS

JUNE 28-30, 2010

Overview

The Reading and Mathematics groups will each be setting standards for two grade levels. The

panels will complete the standard setting activities for the first grade level – discussing the

Achievement Level Descriptors and completing the three rounds of ratings – then will repeat the

entire process for the second grade level. For the first grade level, the panelists will complete two

evaluation forms: a training evaluation before starting round 1 and a procedural evaluation after

round 3. For the second grade level, the panelists will not need to fill out either of these

evaluations; instead, the panelists will complete the final evaluation after the Cross Grade Policy

Forum, which occurs at the very end of the process.

Introductions

1) Welcome group, introduce yourself (name, affiliation, a little selected background

information).

2) Have each participant introduce him/herself.

3) Ask each participant to sign a nondisclosure form. Do not proceed until a signed

nondisclosure form has been collected from each participant.

Complete Standard Setting Activities for First Grade Level Review Materials

1) Have the panelists take a few minutes to briefly look through the AGLEs 2) Have the panelists briefly review about every fifth portfolio, noting the increasing level of

performance.

3) When they are done, allow a minute or two for comments or questions.

4) Familiarize the panelists with the rating sheet and explain how to complete it:

a. Place one (and only one) “X” in each row

b. They can place the X’s at the low or high end of the box, but they must clearly be

within one box: no straddling!


Discuss Achievement Level Descriptors

Overview: In order to establish a thorough understanding of the expected performance of students

on the test, panelists must have a clear understanding of:

1) The definitions of the four achievement levels, and

2) what the key characteristics are that distinguish students in adjacent achievement level

categories.

The purpose of this activity is for the panelists to come to consensus about what characterizes

students in each of the four achievement level categories. This activity is critical since the ratings

panelists will be making in Rounds 1 through 3 will be based on these understandings.

Activities:

1. Introduce task. In this activity they will:

a. Individually review the Achievement Level Descriptors that describe the main

characteristics that define students in each achievement level category; and

b. discuss Descriptors as a group.

2. Have panelists individually review the Achievement Level Descriptors for each level. They

can make notes if they like. The goal here is for the panelists to come to a common

understanding of what it means to be in each achievement level. It is not unusual for

panelists to disagree with the Descriptors they will see; almost certainly there will be some

panelists who will want to change them. However, the task at hand is for panelists to have a

common understanding of what knowledge, skills, and abilities are described by each

Achievement Level Descriptor.

3. After individually reviewing the Descriptors, have the panelists discuss each achievement

level as a group, starting with Partially Proficient. The panelists will discuss the

characteristics a student must demonstrate in order to be classified in the Partially Proficient

category. Or, put another way, the most important characteristics that distinguish a

Substantially Below Proficient student from a student in the Partially Proficient category. They

will then repeat this process for the Proficient and Proficient with Distinction categories. The

purpose of this step is to have a collegial discussion in which to bring up/clarify any issues

or questions that any individual may have and to reach consensus on an understanding of

the Descriptors.

4. Have the panelists identify the most important characteristics describing students at each

achievement level and record those as bulleted lists on chart paper. These should be posted

on the walls for panelists to refer to as they complete the three rounds of rating.

Training Evaluation (First Grade Level Only)

After completing the discussion of the Achievement Level Descriptors for the first grade level, have

panelists fill out the training evaluation form before proceeding to Round 1. Before you start the

Round 1 activities, scan the completed evaluations to see if there are any problems or concerns that

need to be addressed before proceeding. Return the completed evaluations to the data analysis

room at the next convenient opportunity. It is not necessary to complete the training evaluation

form for the second grade level.


Round 1 Ratings

Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial

determination as to which achievement level category each portfolio should be classified into. In

this round, panelists will be working individually, without discussion with their colleagues.

Activities:

1. Make sure panelists have the following materials:

a. Rating form

b. set of portfolios

c. Achievement Level Descriptors

2. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,

from lowest scoring to highest. Make sure panelists understand that, even though the

portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be

in strictly increasing order.

3. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are

reviewing the portfolios, the panelists should keep in mind the Achievement Level

Descriptors. They should consider the knowledge, skills, and abilities demonstrated by

each and how they relate to the definitions of the achievement levels. The purpose of this

step is for panelists to make their initial determinations as to how the portfolios should be

categorized into the four achievement levels. The panelists are free to make notes on the

portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep

track of their categorizations.

4. Panelists may want to take notes as they work if there are particular points they would like

to discuss with their colleagues in Round 2. a. Have panelists write their ID and round number on the rating form. The ID number

is on their name tags.

b. Briefly remind them how to fill in the rating form.

c. Answer questions the panelists may have about the work in Round 1.

d. Once everyone understands what they are to do in Round 1, tell them to begin.

5. As panelists complete the task, ask them to carefully inspect their rating forms to ensure

they are filled out properly.

a. The ID and round number must be filled in.

b. Each portfolio must be assigned to one and only one achievement level.

c. Reiterate that although the portfolios are presented in order from lowest- to highest-

scoring, the panelists’ category assignments do not need to be in strictly increasing

order.

6. Facilitators should bring all the completed rating forms together to the data analysis work

room for tabulation. Prior to submitting them, however, using a show of hands, indicate on

a piece of chart paper how many panelists assigned each portfolio to each achievement level

category. This chart will be used for the Round 2 discussions.

Tabulation of Round 1 Results Tabulation of Round 1 results will be completed as quickly as possible after receipt of the rating forms. While the tabulation occurs, the group may take a break.


Round 2 Ratings

Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1

placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round

2 discussions, the psychometrician will share the group average cut points based on the Round 1

ratings.

Focusing on any portfolios for which there was disagreement as to how they should be categorized,

the panelists will discuss why they categorized each portfolio as they did, making sure that all

different points of view are included in the discussion.

Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make

their Round 2 ratings.

Activities:


a. Rating form



2. Have panelists write their ID number and Round 2 on the rating form.

3. Provide an overview of Round 2. Paraphrase the following:

a. As in Round 1, the primary purpose is to categorize each portfolio into the

achievement level category where you believe it belongs.

b. Each panelist needs to base his/her judgments on his/her experience with the

content area, understanding of students, the definition of each achievement level

category, discussions with other panelists, and the knowledge, skills, and abilities

required to answer each item.

4. Show the panelists how the portfolios would be categorized based on the room average

Round 1 cut point placements.

5. Remind panelists that they will be discussing each portfolio with their colleagues, but that

they will be categorizing the portfolios individually. It is not necessary for the panelists to

reach consensus about how to categorize each portfolio.

6. Give panelists an opportunity to ask questions about the task for Round 2.

7. Beginning with the first portfolio for which there is disagreement as to how it should be

categorized, the panelists should begin discussing the categorization of the portfolios

according to the Round 1 ratings. a. Panelists only need to discuss those portfolios for which there was disagreement as

to how they should be categorized.

b. Panelists should be encouraged to listen to their colleagues as well as express their

own points of view.

c. If the panelists hear a logic/rationale/argument that they did not consider and that

they feel is compelling, then they may adjust their ratings to incorporate that

information.

d. On the basis of the discussions and the feedback presented, panelists should make

their round 2 ratings.


e. The group does not have to achieve consensus. If panelists honestly disagree, that is

fine. We are trying to get the best judgment of each panelist. Panelists should not feel

compelled or coerced into making a rating with which they disagree.

Encourage the panelists to use the discussion and feedback to assess how stringent or

lenient a judge they are. If a panelist is categorizing portfolios consistently higher or

lower than the group, he or she may have a different understanding of the Achievement

Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,

but that disagreement should be based on a common understanding of the


8. When the group has completed their Round 2 ratings, collect the rating forms. When you

collect the rating forms, carefully inspect them to ensure they are filled out properly.


b. Each portfolio must have one (and only one) rating.






Round 3 Ratings

Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their

Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning

the Round 3 discussions, the psychometrician will share the Round 2 results with the group,

including the group average cut points and impact data, i.e., the approximate percentage of

students who would be classified into each achievement level category based on the room average

cut points from Round 2.

Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final

ratings.

Activities:


a. Rating form










required to answer each item. 4. Review the feedback information with the panelists.

a. Show the panelists how the portfolios would be categorized based on the room

average Round 2 cut point placements.

b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set

the final cut points, these are the approximate percentages of students who would be

classified into each achievement level category.




6. Give panelists an opportunity to ask questions about the feedback information or about the

task for Round 3. 7. Beginning with the first portfolio for which there is disagreement as to how it should be


according to the Round 2 ratings.

a. Panelists should discuss the portfolios for which there was disagreement as to how

they should be categorized, focusing in particular on thoe portfolios around the cuts.


own points of view.



information.



their final ratings.










8. When the group has completed their final ratings, collect the rating forms. When you collect

the rating forms, carefully inspect them to ensure they are filled out properly.


b. Each portfolio for Round 3 must have one (and only one) rating.

c. Return the completed rating forms to the data analysis work room.

Complete Procedural Evaluation After the panelists have completed the standard setting activities for the first grade level, have them

complete the procedural evaluation. Submit the completed evaluations to the data analysis work

room at the earliest convenient opportunity.

Complete Standard Setting Activities for Second Grade Level

After the panelists have completed the three rounds of ratings and filled in the procedural

evaluation, they will then repeat the standard setting activities (except the training and procedural

evaluations) for the second grade level: discussing the Achievement Level Descriptors and the

three rounds of ratings.

Complete Final Evaluation Form Following the cross grade panel, have panelists fill out the final evaluation form. Emphasize that their honest feedback is important.


GENERAL INSTRUCTIONS FOR GROUP FACILITATORS MAINE PAAP STANDARD SETTING

SCIENCE AND WRITING

JUNE 30, 2010

Introductions

4) Welcome group, introduce yourself (name, affiliation, a little selected background

information).

5) Have each participant introduce him/herself.

6) Ask each participant to sign a nondisclosure form. Do not proceed until a signed

nondisclosure form has been collected from each participant.

Review Materials 5) Have the panelists take a few minutes to briefly look through the AGLEs

6) Have the panelists briefly review about every fifth portfolio, noting the increasing level of

performance.

7) When they are done, allow a minute or two for comments or questions.

Discuss Achievement Level Descriptors

Overview: In order to establish a thorough understanding of the expected performance of students

on the test, panelists must have a clear understanding of:

3) The definitions of the four achievement levels, and

4) what the key characteristics are that distinguish students in adjacent achievement level

categories.

The purpose of this activity is for the panelists to come to consensus about what characterizes

students in each of the four achievement level categories. This activity is critical since the ratings

panelists will be making in Rounds 1 through 3 will be based on these understandings.


Activities:

5. Introduce task. In this activity they will:

c. Individually review the Achievement Level Descriptors that describe the main

characteristics that define students in each achievement level category; and

d. discuss Descriptors as a group.

6. Have panelists individually review the Achievement Level Descriptors for each level. They

can make notes if they like. The goal here is for the panelists to come to a common

understanding of what it means to be in each achievement level. It is not unusual for

panelists to disagree with the Descriptors they will see; almost certainly there will be some

panelists who will want to change them. However, the task at hand is for panelists to have a

common understanding of what knowledge, skills, and abilities are described by each

Achievement Level Descriptor.

7. After individually reviewing the Descriptors, have the panelists discuss each achievement

level as a group, starting with Partially Proficient. The panelists will discuss the

characteristics a student must demonstrate in order to be classified in the Partially Proficient

category. Or, put another way, the most important characteristics that distinguish a

Substantially Below Proficient student from a student in the Partially Proficient category. They

will then repeat this process for the Proficient and Proficient with Distinction categories. The

purpose of this step is to have a collegial discussion in which to bring up/clarify any issues

or questions that any individual may have and to reach consensus on an understanding of

the Descriptors.

8. Have the panelists identify the most important characteristics describing students at each

achievement level and record those as bulleted lists on chart paper. These should be posted

on the walls for panelists to refer to as they complete the three rounds of rating.

Round 1 Ratings

Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial

determination as to which achievement level category each portfolio should be classified into. In

this round, panelists will be working individually, without discussion with their colleagues.

Activities:

7. Make sure panelists have the following materials: a. Rating form



8. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,

from lowest scoring to highest. Make sure panelists understand that, even though the

portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be

in strictly increasing order.

9. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are

reviewing the portfolios, the panelists should keep in mind the Achievement Level

Descriptors. They should consider the knowledge, skills, and abilities demonstrated by

each and how they relate to the definitions of the achievement levels. The purpose of this

step is for panelists to make their initial determinations as to how the portfolios should be


categorized into the four achievement levels. The panelists are free to make notes on the

portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep

track of their categorizations.

10. Panelists may want to take notes as they work if there are particular points they would like

to discuss with their colleagues in Round 2.

a. Have panelists write their ID and round number on the rating form. The ID number

is on their name tags.

b. Briefly remind them how to fill in the rating form.

c. Answer questions the panelists may have about the work in Round 1.

d. Once everyone understands what they are to do in Round 1, tell them to begin.

11. As panelists complete the task, ask them to carefully inspect their rating forms to ensure

they are filled out properly.


b. Each portfolio must be assigned to one and only one achievement level.

c. Reiterate that although the portfolios are presented in order from lowest- to highest-

scoring, the panelists’ category assignments do not need to be in strictly increasing

order.





Tabulation of Round 1 Results Tabulation of Round 1 results will be completed as quickly as possible after receipt of the rating

forms. While the tabulation occurs, the group may take a break.


Round 2 Ratings

Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1 placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round

2 discussions, the psychometrician will share the group average cut points based on the Round 1

ratings.

Focusing on any portfolios for which there was disagreement as to how they should be categorized,

the panelists will discuss why they categorized each portfolio as they did, making sure that all

different points of view are included in the discussion.

Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make

their Round 2 ratings.

Activities:


d. Rating form

e. set of portfolios

f. Achievement Level Descriptors









13. Show the panelists how the portfolios would be categorized based on the room average

Round 1 cut point placements.




15. Give panelists an opportunity to ask questions about the task for Round 2.




f. Panelists only need to discuss those portfolios for which there was disagreement as

to how they should be categorized.

g. Panelists should be encouraged to listen to their colleagues as well as express their

own points of view.

h. If the panelists hear a logic/rationale/argument that they did not consider and that


information.

i. On the basis of the discussions and the feedback presented, panelists should make

their round 2 ratings.


j. The group does not have to achieve consensus. If panelists honestly disagree, that is









17. When the group has completed their Round 2 ratings, collect the rating forms. When you

collect the rating forms, carefully inspect them to ensure they are filled out properly.

c. The ID and round number must be filled in.

d. Each portfolio must have one (and only one) rating.






Round 3 Ratings

Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their

Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning

the Round 3 discussions, the psychometrician will share the Round 2 results with the group,

including the group average cut points and impact data, i.e., the approximate percentage of

students who would be classified into each achievement level category based on the room average

cut points from Round 2.

Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final

ratings.

Activities:


a. Rating form b. set of portfolios










12. Review the feedback information with the panelists. a. Show the panelists how the portfolios would be categorized based on the room

average Round 2 cut point placements.

b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set

the final cut points, these are the approximate percentages of students who would be

classified into each achievement level category.




14. Give panelists an opportunity to ask questions about the feedback information or about the

task for Round 3.




a. Panelists should discuss the portfolios for which there was disagreement as to how

they should be categorized, focusing in particular on thoe portfolios around the cuts.


own points of view.



information.



their final ratings.










16. When the group has completed their final ratings, collect the rating forms. When you collect

the rating forms, carefully inspect them to ensure they are filled out properly.


b. Each portfolio for Round 3 must have one (and only one) rating.

c. Return the completed rating forms to the data analysis work room.

Complete Final Evaluation Form After they complete Round 3, have panelists fill out the final evaluation form. Emphasize that their

honest feedback is important.


APPENDIX H—PANELIST AFFILIATIONS

Appendix H—Panelist Affiliations 1 2010 Maine PAAP Standard Setting Report

Armstrong, Linda

Barron, Carla

Belanger, Amanda

Special Educator

Special Education Tchr

RSU 1/Fisher Mitchell

Old Orchard Beach/OOB High School

Union 107/Woodland Elementary School

Reading, 2/3

Math HS

Reading 4/5

Belisle, Mary Math Tchr/Content Specialist SAD 51/Greely Middle School Math 6/7

Boucher, Anne Special Education Tchr RSU 34/Southern Penobscot Regional Program Math 6/7

Butler, Frances


Brewer/Brewer High School

Math HS

Carr, Deb

English Teacher

SAD 55/Sacopee Valley High School

Reading HS

Clark, Jill Special Education Tchr RSU 2/Richmond Middle School Math 6/7

Clemons, Janet Teacher MSAD #15/Gray New Gloucester High School Reading 6/7

Cobb, Patricia

Cole, Emily

Cole, Sally


Elementary Teacher

Reading Interventionist

RSU #11/River View Elementary School

RSU 29/Wellington School

RSU 29/Houlton Elementary School

Reading 4/5

Math 2/3

Reading, 2/3

Coleman, B David English Teacher MSAD 15/Gray-New Gloucester High School Reading HS

Connolly, Kathy


RSU # 26/Glenburn School

Math 6/7

Connolly, Shelley


SAD #4/SAD #4 Elementary School

Math 4/5

Corbett, Terras

Ed Tech 3

Brewer/Brewer High

Math HS

Cornett, Marla Special Education Tchr Lewiston Public Schools/Farwell Math 2/3

Dawson, Daryl Ed Tech 111 Brewer/Brewer High Math HS

Dock, Heidi Special Education Tchr SAD #17/Oxford Elementary Reading, 2/3

Panelist Affiliations

Name Title District/school June 28-29 June 30

Adams, Lynne Asst SPED Director Augusta/Cony Math 2/3 Science 8

Writing 4

Writing 11

Writing 4

Science 8 Science 11 Writing 11

Writing 7

Writing 11

Writing 4

Writing 11 Science 8 Science 5 Science 11

Writing 4

Science 11

Writing 4

continued



Drysdale, Rebekah Special Education Tchr Auburn/Walton Reading 4/5 Writing 7

Dunn, Julie HS SPED Resource Rm MSAD#29/Houlton Reading HS Science 11

Earnhardt, Marge

Frati, Alice

Teacher


Gov. Baxter School f/t Deaf

Bangor School Department/Mary Snow School

Reading 6/7

Math 4/5

Genovese, Katie School Counselor MSAD 15/Gray-New Gloucester Reading HS Science 11

Granger, Sheree Special Education Tchr Sweetser Reading, 2/3

Hargrove, Jesse Special Education Tchr A.O.S. 92/Winslow High School Reading HS Science 11

Hartley, Julie

Hayes, Priscilla

Resource Room Teacher


RSU 11/River View Community School

Auburn/Fairview

Math 4/5

Reading, 2/3

Writing 7

Hayes, Steve Coordinator Special Srvcs Easton School Dept./Easton schools Math 6/7 Writing 11

Herrick, Janet

Hodgkins, Susan

General Education Tchr

Special Services Director

SAD 4/Carroll L. McKusick Elementary

RSU #37/MSAD #37/N/A

Reading, 2/3

Reading 6/7

Writing 7

Howard, Deborah Teacher of the Deaf ME Educational Center f/t Deaf & Hard of Hearing/Governor Baxter School for the Deaf

Reading 4/5 Science 5

Howard, Linda

MSAD 41/Milo Elementary

Math 2/3 Science 5

Inman, Penny Lisbon School Department/Lisbon Community School

Reading, 2/3

Kelley, Debbie


MSAD 37/Narraguagus High School

Math 6/7 Science 8

Lavalle-Rivera, Juan

RSU Science 11

Math 6/7 Science 8

Lessard, Robyn


RSU 24/Ellsworth High School

Reading HS Writing 11

Luginbuhl, Ann Resource Room Teacher Union 104/Charlotte School Math 4/5

Malone, Sean Grade 3-8 Teacher MSAD 14/Union 108/Vanceboro Elementary School

Reading 6/7 Writing 7

McCormick, Kelly Asst Professor of Math University of Southern Maine Math 6/7

continued



Mitchell, Barbara

Special Education Tchr Medway School Department/Medway Middle School

Math 4/5 Science 5

Moody, Lyndon John Calais School Dept - Union 106/Calais High School

Math HS

Mullis, Deborah

Math 2/3 Science 5

Oceipka, Gail

Reeds Brook Middle School

Reading 6/7 Science

8

O'Neill, Kathryn

Peaslee, Kimberly

Pelletier, Deborah

Speech Language Pathologist


Special Educator

RSU 1/ Fisher-Mitchell

MSAD 15/Gray and New Gloucester

Acton Elementary School

Reading 4/5

Reading HS

Reading 6/7

Writing 4

Writing 7

Writing 4

Penner, Nancy Ed Tech Brewer/Brewer Middle School Math HS Science 11

Perry, Heather Plant,

Narda Pomerleau,

Rosemarie Pulkkinen,

Kerri Randall, Lenora

Reed, Paula

Special Educator - Autism



First Grade Teacher


Title 1 Teacher

Lisbon Community School

SAD 20/Fort Fairfield Elem. School

Scarborough/Blue Point


SAD 4/PCHS

Portland/East End Community School

Reading, 2/3

Math 2/3

Reading, 2/3

Reading, 2/3

Math HS

Reading, 2/3

Writing 4

Writing 7

Writing 4

Rehill, Kathy Math 2/3 Science 5

Robbins, Barbara

District Evaluator - SPED

MSAD 74/Solon Elementary

Reading 6/7 Writing

11

Robert, Cheryl

Saponara, Diana

Sawyer, Jane

Seiler, Scott

Sewell, Jill


First Grade Teacher

Special Education Admin.



Lewiston/Farwell


NA/Spring Harbor Academy

MSAD #42/Central Aroostook Jr/Sr High

MSAD #70/Mill Pond School

Math 2/3

Reading 4/5

Math 4/5

Reading 6/7

Reading 4/5

Writing 4

Writing 4

Writing 7

Shardlow, Naomi Retired SPED Director Reading HS Writing

11

continued


Reading 4/5


Skillin, Sarah


Lewiston/Farwell

Math 4/5 Science 5

Stokes, Kelly

Special Educator RSU #15/MSAD #!5/Gray New Gloucester High School

Math HS

Thurber, Jacqueline

Totman, Alice

Tucker, Amy


SPED Tchr - Case Mgr

Technology Integration Specialist

RSU #24/Sumner Memorial High School

RSU 52/Tripp Middle School

RSU 16

Math HS

Math 6/7

Math 6/7

Science 8

Writing 7

Writing 7

Viere, Janet

Special Educator

Auburn School Department/Walton School

Reading 4/5 Writing 11

Vigneault, Rita

RSU #19/Nokomis High School

Math 4/5 Writing 11

Winslow, Susan Title 1 Teacher Portland Public Schools/East End Community

School


APPENDIX I—EVALUATION RESULTS

Appendix I—Evaluation Results 1 2010 Maine PAAP Standard Setting Report

Mathematics Grades 2 and 3

Mathematics Grade 2 Training Evaluation

N Mean % SD % D % U % A % SA I understand the goals of the standard

setting meeting. 8 4.38 0 0 0 62 38

I understand the procedures we are using

to set standards. 8 3.88 0 0 25 62 12

I understand how to use the standard

setting materials. 8 4.12 0 0 0 88 12

I understand the differences between the

performance levels. 7 3.71 0 0 29 71 0

I understand how to make the cut score

judgment. 7 3.43 0 0 57 43 0

I know what tasks to expect for the

remainder of the meeting. 7 3.29 0 14 43 43 0

I am confident in my understanding of the

standard setting task. 8 3.62 0 0 38 62 0

SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree


Cut Scores? Have we discussed that?


I think I understand – not sure

Mathematics Grade 2 Procedural Evaluation

N Mean % SD % D % U % A % SA I understood how to make the cut score

judgments. 8 4.12 0 0 0 88 12

I understood how to use the materials

provided. 8 4.25 0 0 0 75 25

I understood how to record my judgments. 8 4.25 0 0 0 75 25 I think the procedures make sense. 8 3.62 0 25 12 38 25 I am sufficiently familiar with the

assessment. 8 4.25 0 0 12 50 38


performance levels. 8 4 0 12 12 38 38



Very few of the tasks lend themselves to establishing extension to show PWD.

Still struggle with PAAPs incorrectly scored either inaccuracy or in levels of assistance.

N Mean %Low %High The performance level descriptors. 8 4 0 0 12 75 12 The state content standards. 8 4.25 0 0 0 75 25 My perception of the difficulty level of the

assessment. 7 4.14 0 0 14 57 29

The student responses. 8 4.62 0 0 0 38 62 My experience working with students. 8 4.5 0 0 12 25 62


What materials, information, or procedures were most influential in your placement of the cut scores?

Why?

Examples included PAAPs with 4 standards- 4th

was counted in standard setting and shouldn’t have

been.

Examples included PAAPs with only 1 task force for a standard- how can you determine

growth/knowledge with only 1 task?

Note: please check Level of Accuracy Data Key, many were incorrect which made the teacher %

incorrect. For instance no 2/3 choice.

The PAAPs themselves. My knowledge of PAAPs. Panel discussions.

Rubric created by our group.

Student work vs. Actual score listed as there were many with incorrectly scored items.


Mathematics Grades 2 and 3 Final Evaluation Panelist Demographics N = 8

Gender

Race/ethnicity

Years of experience in education

Area of Expertise (check all that apply)

Male 0 Female 8

White 8 Black 0 Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0

0–5 2 5–10 0 10–15 2 More than 15 4

Students with Disabilities 8

Students with Limited English

Proficiency 0

Economically Disadvantaged

Students 2

Gifted and Talented Students 0

General Education 2

N Mean % SD % D % U % A % SA I understood the goals of the standard


I understood the procedures we used to

set standards. 8 4 0 0 25 50 25

The facilitator helped me understand the

process. 8 4.25 0 0 12 50 38

The materials contained the information

needed to set standards. 8 3.38 0 38 12 25 25


provided. 8 4.12 0 0 12 62 25

The performance level descriptors were

clear. 8 4.25 0 0 12 50 38

I understood how to make the cut score

judgments. 8 3.88 0 12 25 25 38

I understood how to use the feedback

provided after each round. 8 4.12 0 12 0 50 38

I understood how to use the impact data. 7 4 0 0 14 71 14

I understood how the cut scores were

calculated. 8 3.62 12 12 0 50 25

The facilitator was able to get answers to

my questions. 8 4.38 0 0 12 38 50

Sufficient time was allotted for training on

the standard setting tasks. 8 4.25 0 0 12 50 38

Sufficient time was allotted to complete


The facilitator helped the standard setting

process run smoothly. 8 4.62 0 0 0 38 62



Influence of N Mean %Low %High The performance level descriptors. 8 4 0 0 12 75 12

Usefulness of N Mean %Low %High The opening session. 8 3.38 12 0 38 38 12 The small group activities. 8 4.38 0 0 0 62 38

Becoming familiar with the assessment. 8 4.12 0 0 12 62 25

Articulating the differences between the


Discussions with other participants. 8 4.62 0 0 0 38 62

My expectations of students. 8 3.75 0 0 25 75 0 The difficulty of the test materials. 8 4.12 0 0 12 62 25 The student responses. 8 4.25 0 0 0 75 25 My experience in the field. 8 4.12 0 0 25 38 38 Discussions with other participants. 8 4.12 0 12 12 25 50 Cut scores of other participants. 8 3.62 0 12 25 50 12 Impact data. 7 3.14 0 29 29 43 0

Please provide any additional comments about the standard setting process or suggestions as

to how the training and process could be improved.

Examples should include those with correctly scored items and correctly identified level of

accuracy marked because we were told they were corrected when Measured Progress scored

but we didn’t have that info.Examples shouldn’t have missing tasks or zeros ( at the upper

limits) makes more difficult to give accurate reading limits) makes more difficult to give

accurate reading. It was unique to see I know that no highest level will create the need for

policy level discussion.

Samples had numerous errors-scoring errors, level of assistance errors, task with scores

below 33 on LOC 4 tasks at “top” of samples (30+) at gr. 3

Better quality of examples. Felt pressured to raise some students’ standings for better cut

scores.

Use tests that are scored correctly. Provide raw scores.



Mathematics Grade 4 Training Evaluation




to set standards. 8 4.75 0 0 0 25 75






judgment. 8 4.12 0 0 0 88 12







Please indicate any questions you may have about the remainder of the standard setting

meeting.


N Mean %Low %High The performance level descriptors. 13 4.54 0 0 15 15 69

Mathematics Grade 4 Procedural Evaluation


judgments. 16 4.38 0 0 0 62 38


provided. 16 4.56 0 0 0 44 56

I understood how to record my judgments. 16 4.75 0 0 0 25 75

I think the procedures make sense. 16 4.44 0 0 6 44 50

I am sufficiently familiar with the

assessment. 16 4.62 0 0 0 38 62





meeting.

The state content standards. 13 3.85 0 0 46 23 31 My perception of the difficulty level of the assessment.

13 3.69 15 0 23 23 38


What materials, information, or procedures were most influential in your placement of the cut

scores? Why?

None were more influential than others

Level of complexity, Level of accuracy, level of assistance

Student responses +my experience

Student work & core skills assessed for grade level



Gender

Race/ethnicity



Male 0 Female 8


0–5 0 5–10 1 10–15 0 More than 15 7



Proficiency 1


Students 3


General Education 3

N Mean % SD % D % U % A % SA

I understood the goals of the standard



set standards. 8 4.62 0 0 0 38 62


process. 8 4.88 0 0 0 12 88




provided. 8 4.62 0 0 0 38 62


clear. 8 4.62 0 0 0 38 62


judgments. 8 4.62 0 0 0 38 62



I understood how to use the impact data. 8 4.5 0 0 0 50 50


calculated. 8 4.5 0 0 0 50 50


my questions. 8 4.88 0 0 0 12 88









Influence of N Mean %Low %High The performance level descriptors. 8 4.5 0 0 0 50 50









Sample sizes leading to accuracy was different and impacted decisions.

Super Facilitator



Mathematics Grades 6 and 7 Training Evaluation




to set standards. 10 4.8 0 0 0 20 80






judgment. 10 4.1 0 0 30 30 40







Cut score further explained

Just need to put process to work.


meeting.

I believe I will learn what further questions I have from doing the actual work.

None at this time.



Mathematics Grades 6 and 7 Procedural Evaluation


judgments. 10 4.6 0 0 10 20 70


provided. 10 4.8 0 0 0 20 80

I understood how to record my judgments. 10 5 0 0 0 0 100



assessment. 10 4.5 0 0 10 30 60





meeting.

The state content standards. 10 4 0 0 20 60 20 My perception of the difficulty level of assessment.

the 10

4.2

0

0

10

60

30

The student responses. 10 4.7 0 0 0 30 70 My experience working with students. 10 4 0 0 20 60 20


scores? Why?

The criteria the group set for the steps of the rubric. This help keep me focus when personal

bias came in to play

The combination of LOC & LoA w/ & of accuracy as well as overall & outcome.

Achievement level descriptors helped me to better see subtle difference between levels.

Descriptors

The student PAAP samples and seeing their work (Loc, level of accuracy and level of

assistance). Combining that info with the descriptors helped me place each PAAP.

Defining more specifically the levels of proficiency using the Maine PAAP definitions.



Gender

Race/ethnicity



Male 2 Female 8


0–5 0 5–10 3 10–15 0 More than 15 7



Proficiency 0


Students 4


General Education 6





set standards. 10 4.8 0 0 0 20 80


process. 10 4.9 0 0 0 10 90




provided. 10 4.8 0 0 0 20 80


clear. 10 4.6 0 0 0 40 60


judgments. 10 4.7 0 0 0 30 70





calculated. 10 4.6 0 0 0 40 60


my questions. 10 4.8 0 0 0 20 80










Usefulness of N Mean %Low %High The opening session. 10 3.9 0 10 20 40 30 The small group activities. 10 5 0 0 0 0 100





My expectations of students. 10 4.2 0 0 30 20 50 The difficulty of the test materials. 10 4.5 0 0 10 30 60 The student responses. 9 5 0 0 0 0 100 My experience in the field. 10 4.6 0 0 0 40 60 Discussions with other participants. 9 4.89 0 0 0 11 89 Cut scores of other participants. 10 4.1 0 10 10 40 40 Impact data. 10 4.1 0 10 10 40 40



Maybe the intro sessions with lots of repetition + “you’ll learn more about this in small

groups later” could be shorter

Enjoyed the process very much



Mathematics Grades 10 and 11 Training Evaluation


setting meeting. 9 4.22 0 0 11 56 33


to set standards. 9 4.22 0 0 11 56 33






judgment. 9 4 0 0 22 56 22







Not a question, but a comment, I would have began the achievement levels discussion with

the proficient level instead of the substantially below level proficiency.

How to apply these cuts to the tasks

A bit more time with the materials and I’ll have it I’m almost there.

Let’s go through one together.

Have learned much so far


meeting.

I don’t know if I will be able to do this successfully. It seems a little confusing.

I think if we did the first one together, everyone would be on the same page about what we’re

looking for.

Am visual – will feel more confident as one is actually done.



Mathematics Grades 10 and 11 Procedural Evaluation


judgments. 9 4.11 0 0 11 67 22


provided. 9 4.22 0 0 0 78 22




assessment. 9 4.22 0 0 0 78 22





meeting.

Would have liked to do practice task first.

The state content standards. 9 4.11 0 0 22 44 33 My perception of the difficulty level of t assessment.

he 9

4.33

0

0

0

67

33

The student responses. 9 4 0 0 22 56 22 My experience working with students. 9 3.78 11 0 22 33 33


scores? Why?

Level descriptions

The descriptions of skills (prior knowledge) in the task descriptions. How specific do the ach.

level descriptions need to be followed when determining the standard?

Difficulty level and content standards

The grade level complexities and how the students performed on those tasks were most

influential to me.

LOC mainly as it loosely corresponded to the skills in the 4 levels.



Gender

Race/ethnicity



Male 2 Female 7


0–5 0 5–10 2 10–15 3 More than 15 4



Proficiency 2


Students 3


General Education 2



setting meeting. 9 4.11 0 11 0 56 33


set standards. 9 4 0 11 0 67 22


process. 9 4.22 0 11 0 44 44




provided. 9 4.44 0 0 0 56 44


clear. 9 3.89 0 0 33 44 22


judgments. 9 4.33 0 0 0 67 33





calculated. 9 3.78 0 11 11 67 11


my questions. 9 4.33 0 0 0 67 33


















I didn’t like the SBP PP P PD skills we listed. I found it was easier to go by LOC’s and

student scores along with the need for assistance.

I think a sample task would be very helpful to go over before starting the evaluations within

your own smaller group.

Process reversed to have open discussion for example of standard setting expectations.


Reading Grades 2 and 3

Reading Grades 2 and 3 Training Evaluation


setting meeting. 18 4.17 0 0 11 61 28


to set standards. 18 3.94 0 0 22 61 17






judgment. 18 3.5 0 0 61 28 11







Is Proficiency based on how well a student did on this particular assessment or upon where

he should be?

Difference between achievement levels. More about cut scores

Not sure how the indicators we established are helpful when scoring level ½ assessments.


meeting.

I just wanted to have the conversation with group to see if we are seeing it in a similar

manner.

Note: It is helpful to have done standard setting before.

How does substantially below proficient and partially proficient affect AYP?

Can a student be considered proficient if they received 100/83 at locs?


N Mean %Low %High The performance level descriptors. 11 4 0 0 9 82 9

Reading Grades 2 and 3 Procedural Evaluation


judgments. 11 4.18 0 0 0 82 18


provided. 11 4.36 0 0 0 64 36




assessment. 11 3.91 0 9 18 45 27





meeting.

The state content standards. 11 3.45 0 0 55 45 0 My perception of the difficulty level of assessment.

the 11

4.18

0

0

9

64

27



scores? Why?

Understanding of the student’s body of work, level of complexities. I know how difficult it

is for students to complete these tasks and try to give them the benefit of the doubt.

Level of complexity

Level of assistance and achievement level descriptors.

Great facilitators- questions were answered.


Reading Grades 2 and 3 Final Evaluation Panelist Demographics N = 11

Gender

Race/ethnicity



Male 0 Female 11


0–5 0 5–10 1 10–15 4 More than 15 6



Proficiency 1


Students 9


General Education 5

N Mean % SD % D % U % A % SA I understood the goals of the standard

setting meeting. 11 4.36 0 0 0 64 36


set standards. 11 4.45 0 0 0 55 45


process. 11 4.36 0 0 9 45 45




provided. 11 4.36 0 0 0 64 36


clear. 11 4.36 0 0 0 64 36


judgments. 11 4.27 0 9 0 45 45





calculated. 11 4 0 9 18 36 36


my questions. 11 4.55 0 0 0 45 55















My expectations of students. 11 4.27 0 0 9 55 36 The difficulty of the test materials. 11 4.36 0 0 9 45 45 The student responses. 11 4.27 0 0 0 73 27 My experience in the field. 11 4 0 0 27 45 27 Discussions with other participants. 11 4.45 0 0 9 36 55 Cut scores of other participants. 11 3.82 0 0 45 27 27 Impact data. 11 4 0 0 18 64 18







setting meeting. 9 4.22 0 0 11 56 33


to set standards. 9 4.11 0 0 11 67 22


setting materials. 9 4 0 0 11 78 11




judgment. 9 3.33 0 0 78 11 11







As a reg. ed Teacher I wish I had prepared myself more on PAAPs because I have realized

that I really knew very little. Thank you for the brief introduction.

The training provided was sequential and with a clearly stated outcome. At this point, I need

to get my feet wet with the process. Maybe ?’s will arise as I am completing the task.

Example of how to score demonstrated would be helpful on first task as a group.


meeting.


N Mean %Low %High The performance level descriptors. 9 4 0 0 11 78 11



judgments. 9 4.11 0 0 11 67 22


provided. 9 4.33 0 0 0 67 33




assessment. 9 3.89 0 0 33 44 22





meeting.

The procedures make sense but I feel like we could have had more time. Explanations of

different ability levels would have been helpful for Regular Ed. Teachers.


he 9

3.44

0

11

33

56

0



scores? Why?

Discussion- as a reg. ed teacher I expected the students to be able to read for the reading

section but they didn’t have to. Hard not to compare to the NECAP.

Discussion among facilitator and group; more time should be allowed for this process- felt

very rushed.

The more info we have about student achievement, the better decisions we can make.

Discussions were helpful; did feel rushed at times to get through the stack!

LOC; level of assistance; for successive rounds, listening to others’ feedback

LOCs and the student work with score sheets; more info (specific info) would have helped

with regards to the level of assistance and actual students responses, not just correct or

incorrect.


Having the handouts which defined the Achievement Levels as well as the charts with

additional descriptors were useful because I didn’t have to rely on my memory for the info- I

had a reference(s) to go back to.

I would have appreciated more level of assistance descriptions from teachers. As a teacher

who has given PAAPs I was never made aware of the formula in scoring the different levels

of the LOC and the weight given to the higher levels in scoring.

The LOC Indicators were very helpful with this process. I also would have like to have more

info for LO Assistance from teachers.



Gender

Race/ethnicity



Male 0 Female 9


0–5 1 5–10 2 10–15 1 More than 15 5



Proficiency 2


Students 3


General Education 5





set standards. 9 4.22 0 0 11 56 33


process. 9 4.11 0 0 11 67 22


needed to set standards. 9 4 0 0 11 78 11


provided. 8 4.25 0 0 0 75 25


clear. 9 4 0 0 11 78 11


judgments. 9 3.67 0 0 33 67 0





calculated. 8 3.12 0 25 38 38 0


my questions. 9 4.44 0 0 0 56 44















My expectations of students. 9 4.11 0 0 11 67 22 The difficulty of the test materials. 9 4 0 0 11 78 11 The student responses. 9 4.22 0 0 0 78 22 My experience in the field. 9 4.33 0 0 0 67 33 Discussions with other participants. 9 4.56 0 0 0 44 56 Cut scores of other participants. 9 3.56 0 0 44 56 0 Impact data. 9 3 11 0 67 22 0



Devise some sort of graphic organizer to record notes as we revised the portfolios (to keep

track of our reactions + inputs)







to set standards. 8 4.12 0 12 0 50 38






judgment. 8 4.12 0 0 0 88 12







Cut score


meeting.





judgments. 8 3.62 0 0 38 62 0


provided. 8 3.5 0 12 25 62 0

I understood how to record my judgments. 8 4 0 0 12 75 12



assessment. 7 4.29 0 0 0 71 29




meeting.

The state content standards. 8 3 0 25 62 0 12 My perception of the difficulty level of the assessment.

8 3.12 0 0 88 12 0



scores? Why?

Using the LOCs and TLC evidence of growth influences my cut score rating.

The LOC levels

It appears as if the Loc’s do not match the tasks. Ex an A1 LOC the student must be using

phoenic awareness/word parts and context clues, yet the task does not require the student

read. I feel this needs to be examined more closely.

It became obvious during/prior to Round 3 what Measured Progress was “looking for” this

influenced the decisions made causing me to question the validity of this entire process.

Difficulty of the task along with student responses.



Gender

Race/ethnicity



Male 2 Female 6


0–5 1 5–10 1 10–15 0 More than 15 6



Proficiency 2


Students 5


General Education 4



setting meeting. 8 4 0 0 0 100 0


set standards. 8 3.75 0 0 25 75 0


process. 8 3.88 0 0 12 88 0




provided. 8 3.88 0 0 12 88 0


clear. 8 3.38 0 0 62 38 0


judgments. 7 3.71 0 0 29 71 0





calculated. 8 3.75 0 0 25 75 0


my questions. 8 3.75 0 0 25 75 0









Usefulness of N Mean %Low %High The opening session. 8 3.38 0 12 50 25 12


The small group activities. 8 3.88 0 0 25 62 12 Becoming familiar with the assessment. 8 3.88 0 0 25 62 12 Articulating the differences between the performance levels.

8 3.75 0 0 25 75 0

Discussions with other participants. 8 4.38 0 0 12 38 50 Providing additional details to the performance level descriptors.

2 4 0 0 0 100 0

My expectations of students. 8 3.75 0 0 25 75 0 The difficulty of the test materials. 8 3.62 0 0 38 62 0 The student responses. 8 3.88 0 0 25 62 12 My experience in the field. 8 3.88 0 0 25 62 12 Discussions with other participants. 8 4 0 0 12 75 12 Cut scores of other participants. 8 3.38 0 0 62 38 0 Impact data. 8 3.62 0 12 12 75 0



The achievement level definitions made this process difficult. They do not match the tasks,

therefore made this process mute, in my opinion.







to set standards. 7 3.86 0 0 14 86 0


setting materials. 6 4 0 0 0 100 0




judgment. 7 3.71 0 0 29 71 0








meeting.





judgments. 7 4.43 0 0 0 57 43


provided. 7 4.71 0 0 0 29 71




assessment. 7 4.57 0 0 0 43 57





meeting.

(referring to “I think the procedures make sense”; subject checked “agree”): This seems

counter!

Subjectivity?

The state content standards. 7 4 0 14 0 57 29 My perception of the difficulty level of the assessment.

7 4.71 0 0 0 29 71



scores? Why?

Group discussion about student work. Initial setting of achievement indicators.

Fantastic process.

Group discussions and skills at each level.

Achievement Level Descriptors, definitions of the four levels- handout



Reading Grades 10 and 11 Final Evaluation


judgments. 7 4.57 0 0 0 43 57


provided. 7 4.71 0 0 0 29 71




assessment. 7 4.43 0 0 0 57 43





meeting.


he 7

4.57

0

0

0

43

57



scores? Why?

Excellent Process

Description of 4 levels SBP,PP,P,PWD + achievement level descriptors

The discussions and standards

Student work and discussion about it. Achievement indicators


Science Grade 5

Gender

Science Grade 5 Final Evaluation Panelist Demographics N = 7

Male 0 Female 7

White 7

Black 0

Race/ethnicity



Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0

0–5 0 5–10 1 10–15 2 More than 15 4



Proficiency 1


Students 1


General Education 3





set standards. 7 4.43 0 0 14 29 57


process. 7 4.57 0 0 0 43 57




provided. 7 4.57 0 0 0 43 57


clear. 7 4.57 0 0 0 43 57


judgments. 7 4.43 0 0 14 29 57





calculated. 7 4.14 0 0 29 29 43


my questions. 7 4.71 0 0 0 29 71


















Many teacher still are not following the process-fair amount of teacher error rather than

student error

It was more difficult than need to due to lack of higher LOC’s at some indicators

Much better samples than for math!

( Referring to “I understood how to make cut score judgments.”) Seriously, I don’t feel I

make cut score judgments

If this were to be done again, it would be helpful to have a summary sheet for each students

with task , LOL,LOA for all tasks on 1 sheet. This would save a lot of time, also ever

consider going paperless. Some of this could definitely be paperless


Science Grade 8

Gender


Male 1 Female 6

White 6

Black 0

Race/ethnicity




0–5 0 5–10 0 10–15 0 More than 15 7



Proficiency 0


Students 2


General Education 2





set standards. 7 4.71 0 0 0 29 71


process. 7 4.43 0 0 14 29 57




provided. 7 4.86 0 0 0 14 86


clear. 7 4.57 0 0 14 14 71


judgments. 7 4.57 0 0 0 43 57





calculated. 7 4.43 0 0 14 29 57


my questions. 7 4.86 0 0 0 14 86










Usefulness of N Mean %Low %High The opening session. 7 4 14 0 0 43 43 The small group activities. 7 4.57 0 0 0 43 57




Discussions with other participants. 7 5 0 0 0 0 100

Providing additional details to the

performance level descriptors. 4 5 0 0 0 0 100

My expectations of students. 7 4.14 0 0 14 57 29 The difficulty of the test materials. 7 3.86 0 0 29 57 14 The student responses. 7 4.71 0 0 14 0 86 My experience in the field. 7 4 0 14 0 57 29 Discussions with other participants. 7 5 0 0 0 0 100 Cut scores of other participants. 7 3.86 14 0 0 57 29 Impact data. 7 4 0 0 14 71 14



I enjoyed the process very much.

Hope the blue print and fall trainings can give clarity to next years grade level expectations

and that a full task bank will provide more evaluation options.


Science Grade 11

Gender


Male 2 Female 5

White 7

Black 0

Race/ethnicity




0–5 1 5–10 1 10–15 4 More than 15 1



Proficiency 1


Students 1


General Education 2





set standards. 7 4 0 0 14 71 14


process. 7 4.14 0 0 0 86 14




provided. 7 4.14 0 0 14 57 29


clear. 7 3.71 0 0 29 71 0


judgments. 7 4 0 0 14 71 14





calculated. 7 4.29 0 0 0 71 29


my questions. 7 4.14 0 0 0 86 14











Becoming familiar with the assessment. 7 4 0 0 29 43 29






My expectations of students. 7 4 0 0 29 43 29 The difficulty of the test materials. 7 4 0 0 14 71 14 The student responses. 7 4.14 0 0 14 57 29 My experience in the field. 7 4.14 0 0 29 29 43 Discussions with other participants. 7 4.14 0 0 29 29 43 Cut scores of other participants. 7 3.71 0 0 43 43 14 Impact data. 7 3.86 0 0 29 57 14



Process between facilitators seemed at times different. Possible that this is due to different

groups? Content areas.


Writing Grade 4

Gender

Writing Grade 4 Final Evaluation Panelist Demographics N = 10

Male 0 Female 10

White 10

Black 0

Race/ethnicity




0–5 1 5–10 3 10–15 2 More than 15 4



Proficiency 0


Students 6


General Education 3





set standards. 9 4.67 0 0 0 33 67


process. 9 4.56 0 0 0 44 56




provided. 9 4.67 0 0 0 33 67


clear. 9 4.67 0 0 0 33 67


judgments. 9 4.67 0 0 0 33 67





calculated. 10 4.2 0 0 20 40 40


my questions. 9 4.67 0 0 0 33 67














Discussions with other participants. 9 5 0 0 0 0 100






Since I was unfamiliar with what the student was expected to do, it would have been more

helpful to see the tasks before we designed our rubrics.


Writing Grade 7

Gender


Male 1 Female 8

White 9

Black 0

Race/ethnicity




0–5 1 5–10 2 10–15 0 More than 15 5



Proficiency 0


Students 3


General Education 5





set standards. 9 4.67 0 0 0 33 67


process. 9 4.67 0 0 0 33 67




provided. 9 4.67 0 0 0 33 67


clear. 9 4.56 0 0 0 44 56


judgments. 9 4.56 0 0 0 44 56





calculated. 9 4.56 0 0 0 44 56


my questions. 9 4.67 0 0 0 33 67

















My expectations of students. 9 4.11 0 0 22 44 33 The difficulty of the test materials. 9 4.11 0 0 22 44 33 The student responses. 8 4.62 0 0 0 38 62 My experience in the field. 9 4.11 0 0 22 44 33 Discussions with other participants. 9 4.67 0 0 0 33 67 Cut scores of other participants. 9 4 0 11 11 44 33 Impact data. 8 4.25 0 0 12 50 38



Awesome 3 days! This is the best I have been to in 16 yrs!


Writing Grade 11

Gender


Male 2 Female 6

White 8

Black 0

Race/ethnicity




0–5 0 5–10 0 10–15 1 More than 15 7



Proficiency 1


Students 6


General Education 4





set standards. 8 4.25 0 0 0 75 25


process. 8 4.38 0 0 0 62 38




provided. 8 4.25 0 0 0 75 25


clear. 8 4.12 0 0 12 62 25


judgments. 8 4.25 0 0 0 75 25



I understood how to use the impact data. 8 4 0 0 25 50 25


calculated. 8 4 0 12 0 62 25


my questions. 8 4.38 0 0 0 62 38




the standard setting tasks. 8 4 0 12 0 62 25







Becoming familiar with the assessment. 8 4 0 0 12 75 12









Would like guidelines of LOA to be present in future.

Efficiently done; leadership/guidance great help and support

Jake was great to work with!

Need more clarity on level of assistance- Scribe/- more info provide on how each teacher did

it.

I believe that we needed more information regarding the level of assistance. Some teachers

were not clear. Training for this may need to be more specific.


APPENDIX J—POLICY ADJUSTMENTS

Appendix J—Policy Adjustments 1 2010 Maine PAAP Standard Setting Report


cut Standard

error Raw

Min

score


PD 61.1 NA 62 69 3.4 Proficient 35 NA 35 61 51.7 Partially Proficient 21.5 NA 22 34 36.9

PD 61.1 NA 62 69 14.1 Proficient 39 NA 39 61 41.5 Partially Proficient 21.5 NA 22 38 33.2

PD 61.1 0.4 62 69 20.5 Proficient 41.2 0.3 42 61 42.3 Partially Proficient 21.5 0 22 41 28.2



PD 91.3 0.9 92 99 19.8 Proficient 55.9 NA 56 91 53.1 Partially Proficient 24.9 0.6 25 55 18.8

PD 121.25 NA 122 129 9 Proficient 81.55 NA 82 121 35.4 Partially Proficient 32.45 NA 33 81 38.4 SBP NA NA 0 32 17.1

MDOE Policy:

Students in grades 2 through 5 have the opportunity to access tasks in levels of complexity 1–4.

Adjusted cuts were made based on the grade 4 cuts as stated above. However, when looking at the adjusted

cuts for grades 2 and 3, it was felt that the adjusted cuts at Proficient were too high.

MDOE reviewed 4–6 student samples around the Partially Proficient /Proficient cut points along with

participant rater sheets and determined that cut scores between Partially Proficient and Proficient for students

in grade 2 should be minimally lowered. The same process occurred for students in grade 3 keeping in mind

that grade 3 should be slightly higher than grade 2 but still minimally lower than grade 4. These new cuts are

reflected in the chart titled “Mathematics Policy Results.”

Writing grade 7 cut scores at Proficient was minimally lowered following the same procedures as

outlined above for mathematics.

Grade

Mathematics: Policy Results

2

SBP NA NA 0 21 8.0

3

SBP NA NA 0 21 11.2

4

SBP NA NA 0 21 9

5

SBP NA NA 0 26 16.6

6

SBP NA NA 0 24 13.2

7

SBP NA NA 0 24 8.2

HS



cut Standard

error Ra

Min w score


PD 22.0 NA 22 23 19.9 Proficient 14.4 0.1 15 21 28.6 Partially Proficient 9.6 0.1 10 14 22.4

PD 32.0 NA 32 33 15.0 Proficient 23.0 NA 23 31 31.6 Partially Proficient 11.6 0.1 12 22 39.8


Grade

Writing: Policy Results

4

SBP NA NA 0 9 29.0

7

SBP NA NA 0 11 13.6

11


Pe

rce

nt

of

stu

den

ts in c

ate

go

ry

0

20

4

0

60

8

0

10

0


Proficient


Mathematics: Policy Results


Pe

rce

nt

of

stu

den

ts in c

ate

go

ry

0

20

4

0

60

8

0

10

0

Proficient with Distinction Proficient Partially Proficient Substantially Below Proficient

4 7 11

Writing: Policy Results


Documents

Standard Setting Report - MaineStandard Setting Report ... The computational programming used to carry out all analyses during the standard setting meeting was completed and thoroughly