Reproductions supplied by EDRS are the best that can be ...DOCUMENT RESUME. Zhang, Liru Delaware Student Testing Program: Report. Study. Delaware State Dept. of Education, Dover. Accountability

ED 455 271

AUTHORTITLE

INSTITUTION

PUB DATENOTE

DOCUMENT RESUME

Zhang, LiruDelaware Student Testing Program: ReportStudy.

Delaware State Dept. of Education, Dover.Accountability Branch.2000-10-00113p.; For a report on the comparison ofTesting Program results for 1998 through073

PUB TYPE Numerical/Quantitative Data (110) Reports Research(143)

MF01/PC05 Plus Postage.*Academic Achievement; Elementary Education; ElementarySchool Students; *Low Achievement; *Scores; Scoring; StatePrograms; Teachers; *Test Results; *Testing Programs;Writing Achievement; *Writing Tests*Delaware Student Testing Program

TM 033 074

on Special Writing

Assessment and

Delaware Student2000, see TM 033

EDRS PRICEDESCRIPTORS

IDENTIFIERS

ABSTRACTThis study invesitigated possible reasons for the low

performance on the text-based writing assessment of the Delaware StudentTesting Program (DSTP) in 2000, especially for grades 3 and 5, and consideredways to improve classroom instruction. In the first part of the study, apanel of teachers reviewed the anchor papers from the assessment and theprocess of testing. Panel members rescored the anchor papers of a given gradeindependently and then worked in a small group to discuss and finalize theirscores. In the second part of the study, a second panel of teachersparticipated in a rescoring session for a sample of 100 text-based writingsper grade, scoring students' writings holistically and analytically using thefive-point scoring rubric. Both panels also reviewed the directions foradministering the test and test booklets. Results of the study are reportedin the categories of: (1) the process of testing; (2) construct validityevidence; (3) text-based writing scoring; (4) text-based writing development;and (5) text-based writing instruction. In each of these categories, somepossible explanations for student performance were identified, including theintroduction of a new text-based writing task at each grade level in 2000, asexemplified by the change from persuasive writing to informative writing atgrade 3. The paper cautions that the study was done rapidly and that muchadditional research is needed, focusing on the stability of scores over timeand the processes of reader training and scoring. Attachments contain thescoring rubric, instructions for the anchor paper review, rescoringdirections, a list of panel members, correlation matrices, and records forthe rescoring. (Contains 14 tables, 4 figures, and 8 references.) (SLD)

Reproductions supplied by EDRS are the best that can be madefrom the original document.

Ntn

Delaware

Student

Testing

Program

Report on Special Writing Study

N-C)co) Prepared by the Assessment and Analysis Groupeir)

Assessment and Accountability BranchDelaware Department of Education

October 2000

EST COPY AVAIIABLE

0

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

This document has been reproduced asreceived from the person or organizationoriginating it.

Minor changes have been made toimprove reproduction quality.

Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position or policy.

1

PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL HAS

BEEN GRANTED BY

D _Blawman

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)

rM

Officers of the Delaware Department of Education

State Board of Education

James L. Spartz, Ed. D., PresidentJean W. Allen, Vice President

Mary B. Graham, EsquireJohn W. Jardine, Jr.

Joseph A. Pika, Ph. D.Dennis J. Savage

Claiboune D. Smith, Ph. D.

Valerie A. WoodruffSecretary of Education

Jennifer W. DavisDeputy Secretary of Education

David BlowmanExecutive Assistant

Robin R. Taylor, M. EdActing Associate Secretary, Assessment and Accountability Branch

Nancy J. Wilson, Ph. D.Associate Secretary, Curriculum and Instructional Improvement Branch

Wendy B. Roberts, Ph. D.Acting Director, Assessment and Analysis Group

Darlene J. Bolig, Ed. D.Jeffery Fleming, M. S.

James F. Hertzog, M. Ed.Nancy Maihoff, Ph. D.

Jon Marion. Ph. D., University ofDelawareJoann F. Prewitt, M. A.

Julie A. Schmidt, Ph. D., University of DelawareCarole D. White, M. B. A.

Liru Zhang, Ph. D.

Qi Tao, Ph. D.Technology Management and Design

Policy and Administrative Service Branch

Support Staff:Krista D. Holloway Erin L. PieshalaBarbara F. O'Neal Kimberly K. Rodriguez

2

The report was prepared by Liru Zhang

The panel comments on Test Development in the section ofResults of the Study was summarized by Darlene Bolig.

The panel comments on Classroom Instruction in the section ofResults of the Study was summarized by Mike Kelly.

This report was reviewed and edited by Robin Taylor, Wendy Roberts,Darlene Bolig, and Mike Kelly.

4

Report on a Special Writing Study

Introduction

3

The objective of the Delaware Student Testing Program (DSTP) is to measure studentprogress toward the Delaware Content Standards. Each spring, all public school studentsin grades 3, 5, 8, and 10 take the statewide assessment in reading, writing, andmathematics. The writing assessment consists of a text-based writing task and a stand-alone writing prompt. The text-based writing task links to a passage in the DSTP readingassessment and students' responses to this task are scored twice, once for a reading scoreand once for a writing score. Both stand-alone and text-based writings are untimed.Students usually take approximately 2 hours, including a 30 minute pre-writing session,to develop, organize, draft, and finalize their stand-alone writings. Only the final draft ofthis prompt is scored.

A 5-point scoring rubric (Please see Attachment A) is used to score both the text-basedand stand-alone responses. One reader scores the text-based writing; two readers scorethe stand-alone writing. The lowest score for the text-based writing is 1 and the highestpossible score is 5; the lowest score for the stand-alone writing is 2 and the highestpossible score is 10. The total writing raw score is the sum of the text-based writingscore and the stand-alone writing score with the lowest score of 3 and the highestpossible score of 15.

Over the past three years, the overall writing scores have declined in grades 3 and 5,remained steady in grade 8, and increased slightly in grade 10 (See Tables la -1d). Theaverage performance on the stand-alone writing shows a consistent pattern of increaseacross years for students in grades 8 and 10; minor fluctuations over time for students ingrades 3 and 5. Student performance on the text-based writing, however, dropped to thelowest level in 2000 for all grades except grade 10, where the average scores dropped by.66 from 1999 and .63 from 1998 to 2000 in grade 3; dropped by .59 from 1999 and .76from 1998 to 2000 in grade 5; and dropped by .28 from 1999 and .45 from 1998 to 2000in grade 8. Because of the drop in text-based writing scores, the Assessment . andAnalysis Group decided to conduct a special writing study to investigate the possiblereasons for the low performance in 2000, especially in grades 3 and 5.

Purpose of the Study

The primary purposes of this study were (1) to investigate the possible reasons for thelow performance on the text-based writing in 2000, especially in grades 3 and 5 and (2)to investigate ways to improve classroom instruction in writing.

Methods of the Study

General Design Due to the time constraints and the availability of information/data, thisstudy focused on the following five aspects:

4

Review the test process (i.e., review of test administration and testing materials);Review text-based writing scores (i.e., review anchor papers and re-score asample of students' responses to the text-based writing task, and compare thestatistics of text-based writing scores from the field test with and the 2000 DSTP);Examine construct validity evidence (i.e., review available data and conductadditional statistical analyses);Make recommendations for the development of text-based writing tasks; andMake recommendations on ways to improve classroom instruction in writing.

This study included two parts. In Part One, a panel of teachers reviewed the anchorpapers and the process of testing (See Attachment B). Anchor papers are a sample ofstudents' writings that are used as benchmarks in scoring. Each anchor paper represents ascore point and usually represents the upper and lower levels within each score point. Inthis study, the panel members re-scored the anchor papers of a given gradeindependently, and then worked in a small group to discuss and finalize their scores. InPart Two, a second panel of teachers participated in a re-scoring session for a sample of100 text-based writings per grade. They scored students' writings holistically andanalytically using the 5-point scoring rubric. Each writing sample was evaluated by up to5 teachers. Then, the panel members discussed related issues in test administration, testdevelopment, scoring, and classroom instruction (See Attachment C).

Sample of Student Writings A random sample of 100 student responses to the text-basedwriting task was selected from the population of each grade for re-scoring in this study.

Panels of Teachers Two panels of teachers were invited to participate in this study, onefor anchor paper review and one for the re-scoring session (See Attachment D). Theseteachers were selected based on their expertise in writing, teaching experience,experience in the development of writing assessment and scoring, familiarity with theDelaware Content Standards in English language arts and the writing scoring rubric,geographic location, and availability.

The Anchor Paper Review Panel consisted of 9 members, including 7 teachers (78%)from 7 school districts and 2 staff members from the Department of Education. Amongthem, 7 are females and 2 males. Seven of the panel members (78%) have served on thetest development committees and 2 (22%) were involved in the anchor paper pulling forthe 2000 DSTP writing 'assessment.

The Re-Scoring Panel included 22 members, 20 of them were teachers (91%) from 12school districts, 1 from the University of Delaware, and 2 from the Department ofEducation. Forty-one percent of the members (9) have served on the test developmentcommittees and nearly half (about a quarter) were involved in the anchor paper pullingfor the 2000 DSTP writing assessment.

Data Analysis and Summary of Comments To investigate the possible reasons for the lowperformance on the text-based writing in 2000, teachers reviewed, discussed, and maderecommendations for improving test administration, test development, scoring, and text-

6

5

based writing instruction. All comments and subjective evaluations are summarized inattachments. These comments are reported in the section of the "Results of the Study".The results of data analyses are presented in tables and charts. Data analyses for thisstudy include:

Three-year comparisons of students' writing scores by gradeThree-year comparisons of statistics of writing scores by gender and gradeCorrelation analysis of all types of writing scores and reading scores by gradefor 2000 DSTPComparisons of means and frequency distributions of text-based scoresbetween the 2000 DSTP and field test by gradeComparisons of means and frequency distributions of text-based scores fromthe field test, the 2000 DSTP, and re-scoringThe discrepancy of text-based scores from the 2000 DSTP and re-scoringThe reliability indicators of re-scoring the text-based writing

Results of the Study

The results of the study are reported in five categories, the process of testing, constructvalidity evidence, text-based writing scoring, text-based writing development, and text-based writing instruction.

Process of Testing Both panels reviewed the 2000 DSTP Directions for Administeringthe Test and test booklets and compared those directions with the previous years' testingmaterials and the process of testing. Comments on the process of testing focus on thefollowing issues:

1. Two text-based writing prompts, one for field test and one for operational test,should not be given on the same day, especially for younger students.

2. The text-based writing should be given in the beginning of the reading test ratherthan as the last item of the day.

3. The instructions for the text-based writing should be written to draw students'attention, such as bolded for emphasis, using separate pages to ensure thatstudents understand this item will be scored twice for both reading and writing.

4. The text-based writing task should be formatted similar to the stand-alonewriting prompt, such as using pre-writing.

Review of Text-based Writing Scores

Construct Validity Evidence The statistics of the three writing scores, text-based, stand-alone, and the writing. total raw scores are compared by gender and grade for 1998, 1999,and 2000 (Table 2). The data show that female students consistently outperformed malestudents on both text-based and stand-alone writings across all four grades and threeyears. A slight, but consistent decrease of gender differences in text-based scores forstudents in grade 3 is observed (Diff score=.30 in 1998; .22 in 1999; .13 in 2000) fromthe current analysis.

6

As indicated earlier in this report, the text-based writing tasks attach to a passage in theDSTP reading assessment. This passage includes several multiple-choice (MC) andconstructed-response (CR) items. The student's response to one of the CR items wasscored as part of the reading score using the reading scoring rubric and the text-basedwriting score using the writing rubric.

The multiple-choice item has a stern asking a question with four choices, from whichstudents select the best answer to the question. Multiple-choice items are scoreddichotomously. Constructed-response items include short answer and extendedconstructed-response items. Short answer items allow students to make decisions byconstructing brief responses that demonstrate the students' understanding of the text.Short answer items are scores on a 0-2 scale. Extended constructed-response items allowstudents to make decisions by constructing more lengthy responses that demonstrate thestudents' understanding of the text and require students to provide justification for theirresponses. The extended constructed-response items are scores on a 0-4 scale. The itemused as the text-based writing task is an extended constructed-response item.

Tables 3a and 3b present the correlation coefficients among five reading scores, threewriting scores, and the SAT9 reading comprehension test scores by grade from the 2000DSTP. The analyses are based on the following eight variables:

MCITEM: The multiple-choice item score is the sum of scores on all MC itemsattached to the reading passageCRITEM: The constructed-response item score is the sum of scores on allconstructed-response items attached to the reading passagePASSAGE: The passage score is the sum of scores on all MC and CR itemsattached to the reading passageIREADING: The reading item score is the score on the extended constructed-response item attached to the reading passage that was used as the text-basedwriting taskTEXT: The text-based writing score is the score on the extended constructed-response item attached to the reading passage that was used as the text-basedwriting taskPROMPT: The writing score on the stand-alone writing promptWRITING: The total writing raw score that is the sum of the text-based andstand-alone writing scoresREADING: The DSTP reading scoreSAT9: The reading score on the 30-item SAT9 reading comprehension test

The results show that the correlation coefficients between the text-based writing scores(TEXT) and the item reading scores (IREADING) from the same CR items are .22 forgrade 3, .45 for grade 5, .57 for grade 8, and .60 for grade 10. First, the statistics indicatea grade pattern, from the lowest value of the correlation coefficient in grade 3 to thehighest value in grade 10. Second, the low correlation in grade 3 suggests that only 5%of the variance from one score associates with the other score; in grade 5, about 20% of

7

the variance from one score associates with the other score. The correlations between thetext-based writing scores (TEXT) and the scores on the MC items (MCITEM) and the CRitems (CRITEM) from the reading passage, and the passage scores (PASSAGE) are .19,.31 and .31 in grade 3, which is the lowest among the four grades. Again, the lowcorrelation in grade 3 suggests that only 4% to 10% of the variance of text-based writingscores is associated with the MC item scores, CR item scores, and the passage scores,respectively. Similarly, the correlations between the text-based writing score and thescores on MC items (MCITEM) and CR items (CRITEM) from the reading passage, andthe score of the reading passage (PASSAGE) are .26 to .44 in grade 5, which indicate that7% to 19% of the variance from the text-based writing scores can be 'accounted for by thescores from reading. Statistics from the current analysis suggest that the text-basedwriting score is somewhat independent of the reading passage, especially in grades 3 and5.

The correlations between the text-based writing (TEXT) and stand-alone writing score(PROMPT) range from .36, .41, .41, and .48 for grades 3, 5, 8, and 10, respectively. Agrade pattern is observed, where the correlation coefficient for grade 3 is the lowestamong the four grades. The low and moderately low correlations across grades suggestthat about 13% to 23% of the variance from the text-based writing scores is associatedwith the stand-alone writing scores. Statistics appear to suggest that the text-basedwriting measures different types of writing skills or different constructs from the stand-alone writing.

The correlations between SAT9 reading scores, a standardized test, and the DSTP readingscores (READING) are stable across grades, ranging from .84 to .86, and no gradepattern is found. Moreover, the sizes of the correlation coefficients between the SAT9reading and the stand-alone writing scores (PROMPT) are very close, ranging from .41 to48 across grades without a grade pattern. The correlations between the SAT9 readingand the text-based writing scores (TEXT), however, show a grade pattern with the lowestcoefficient in grade 3 (r=.33) and the highest coefficient in grades 8 and 10 (r=.48). Thestatistics indicate that there may be more measurement errors involving in the text-basedwriting scores than that in the stand-alone writing scores, especially in the lower grades.

The correlation matrix among different types of writing scores and reading scores for the1998 and 1999 DSTP provides additional information for the construct validity (SeeAttachment D). The correlation coefficients between reading and writing scores areconsistent in 1998 and 1999. The correlations between text-based writing and readingscores are higher in 1998 and 1999 (r= .56 in 1998 and r=.60 in 1999 for grade 3; r=.60 in1998 and r=.56 in 1999 for grade 5) than that in 2000 (r=.33 in grade 3; r=.44 in grade 5).In grade 3, the correlation between text-based and stand-alone writings is lower in 2000(r=.36) than the previous years (r=.45 in 1998; r---.46 in 1999). The correlation betweentext-based writing and reading scores also shows the lowest value in 2000 for grade 3(r=.63 in 1998; r=.68 in 1999; r=.53 in 2000). Such variations of the statistics acrossyears of testing may be due to one or more of the following reasons:

8

Low generalization of writing scores across topics, the purposes of writing tasks,and occasions;More errors in text-based writing than standard-alone writing in 2000 than theprevious years;More errors in scoring text-based writing than in scoring stand-alone writingbecause of using one reader;Variations in the characteristics of the reading passages and attached items fromyear to year and from grade to grade; orVariations in writing skills among student populations from year to year.

Review of Text-Based Writing Scores

Re-scoring Anchor Papers To examine the accuracy of scoring, the anchor papers werereviewed and re-scored by the first panel. Anchor papers are typical writing samples thatrepresent each score point and used as benchmarks in reader training and scoring.Usually, writing samples are selected to represent the upper and lower levels of eachscore point to facilitate the process of scoring. The panel members reviewed, ranked, andre-scored each anchor paper independently. Then, they worked in small groups todiscuss their scores in order to achieve an agreement, if possible. The mean of the 4 - 5new scores from re-scoring process were calculated and compared with the originalscores. If the different 'score between the new score and the original score (Diff = re-score - original score) was equal or greater than a half point (Diff = >.5), this paper wasflagged as inconsistency in scoring. The results of re-scoring anchor papers show (SeeAttachment F) that the new scores and the original scores are highly consistent in grades3 and 5 (92%), and moderately high in grade 8 (86%) and grade 10 (73%). According tothe panel recommendation, three papers should not be used as anchor papers in the future.

Comparing Test Statistics The mean and the relative frequency distributions of text-basedwriting scores from the field test and the 2000 DSTP were compared. The discrepanciesof scores summarized in Table 4 show that the average scores from the field test areconsistently higher than the average scores from the 2000 DSTP across the four grades,where the different scores (2000 DSTP score - field test score) are -1.03, -.68, -.26, and.65 for grades 3, 5, 8, and 10, respectively. Chart 1 illustrates the discrepancy betweenthe relative frequency distributiohs of the text-based scores from the two occasions. Ingrade 8, the relative frequency distributions of the writing scores from the field test andthe DSTP 2000 are nearly identical with the same mode above the 2-point on the scorescale. The difference in mean scores is due to 15% more students receiving a 2-point and11% less students receiving a 3-point in the 2000 DSTP than that in the field test.Similarly in grade 10, because of the mode of 3-point from the field test and 2-point fromthe 2000 DSTP, the frequency distribution of the 2000 DSTP shift to left from thefrequency distribution of the -field test with a lower average score. In grade 3, thefrequency distribution of the field test scores approaches a normal distribution with themode of 3-point; while the frequency distribution of the 2000 DSTP scores is positivelyskewed with the mode of 1-point, which is 2 score points lower than the mode of thescore distribution from the field test. In grade 5, the mode of the writing scores is 3-pointof the field test scores with a nearly normal frequency distribution; while the mode is 2-

10

9

point of the 2000 DSTP scores with a positively skewed frequency distribution. Toexamine the differences between the two score distributions from the field test and the2000 DSTP, Chi-Square Goodness of Fit test, a non-parametric test, was conducted (SeeTable 4b). The results indicate that the relative frequency distributions of text-basedwriting scores from the two occasions are statistically significantly different at the p< .00level for all the four grades.

Re-scoring Text-Based Writing Samples To further investigate the possible reasons thatcaused the low performance in text-based writing, a re-scoring session was arranged. Asample of 100 students' responses to the text-based task was randomly selected from thepopulation of each grade. The panel members reviewed and re-scored the sample ofwritings independently. Each paper was evaluated holistically and analytically using thesame 5-point scoring rubric. The analytic scores were only used as evidence to supporttheir professional judgments on the quality of student writing. They worked in smallgroups to discuss their initial scores and assigned final scores (See Attachment F).Agreement among panel members was not required. Since the anchor papers were underreview, the process of re-scoring was conducted without anchor papers. The records ofre-scoring from the panel members can be found in Attachment G.

The different text-based writing scores (Diff = re-score - 2000 DSTP score) between the2000 DSTP and re-scoring by the panel were calculated based on the following criteria:

Majority rule was applied to decide the final re-score in the process of re-scoring.If the same number of readers agreed and the same number of readers did notagree with the 2000 DSTP score, the 2000 DSTP was used as the final re-score.If only one reader reviewed the paper, this reader's score was used as the finalre-score.

Tables 5a and 5b present the summary of the discrepancies in scoring by grade. Datashows that the average agreement between the two scoring process is 65% across grades,58% in grade 3, 66% in grade 5, 48% in grade 8, and 47% in grade 10 (Table 5b). Twopatterns are observed: First, the scores assigned in the 2000 DSTP are consistently lowerthan that in re-scoring across the four grades; second, the distributions of the differentscores are negatively skewed, which indicate more papers were scored lower than higherin the 2000 DSTP. On the average, 35% (113 papers) of the papers were scored 1-pointlower in the 2000 DSTP than that in re-scoring, 8% (27 papers) of the papers were scored2-points lower, and less than 1% (1 paper) of the papers were 3-points lower than theyshould be according to the panel's judgment. There are 9% (28 papers) of the papers thatwere scored 1-point higher and less than 1% (2 papers) of the papers scored 2-pointshigher than they should be across grades according to re-scoring. Grade by grade, it isfound that 37% of the papers were scored lower and 5% of the papers scored higher ingrade 3 from the 2000 DSTP; 29% of the papers were scored lower and 5% of the paperswere scored higher based on re-scoring in grade 5. The data show that 46% ofthe paperswere scored lower and 5% of the papers were scored higher in grade 8; 37% of the paperswere scored lower and 16% of the papers were scored higher in the 2000 DSTP.

1.1

10

Tables 6a and 6b list the mean and frequency distributions of text-based scores from threeoccasions, field test, 2000 DSTP, and re-scoring. The data indicate that the writingscores re-assigned in this study are, on the average, higher than the 2000 DSTP, but lowerthan the field test in grades 3, 5, and 10. In both grades 3 and 10, the average re-score iscloser to the 2000 DSTP in absolute value (.27 for grade 3; .16 for grade 10) than to thefield test results (.76 for grade 3; .49 for grade 10). In grade 8, however, the average re-score is .38-point higher than the field test and .64-point (about one standard deviation)higher than the 2000 DSTP.

To examine how reliable the re-scoring process was, the number of readers used to re-score each paper and the degree of agreement among readers were analyzed. Themajority of the papers were reviewed and re-scored by at least 3 readers, 96% in grade 3,71% in grade 5, 51% in grade 8, and 68% in grade 10. In grades 3, 5, and 8, less than 5%of the papers were re-scored by a single reader; in grade 10, 22% of the papers were re-scored by one reader, which may be due to the content and the length of student writingsin higher grade. In addition, the number of readers who assigned the same score thatdiffers from the 2000 DSTP scores is used as the indicator of reliability of scoring (Table7). The data indicate that the majority of the different scores were assigned by the panelin this study based on the agreement among 3 to 5 readers, 95% in grade 3, 70% in grade5, 61% in grade 8, and 69% in grade 10. Fifteen percent of the papers (26 papers)receiving a different score were based on one reader; but most of them are 1-point higherrather than lower than the 2000 DSTP scores.

The panel members were also asked to make professional judgments on the difficultylevel of the 1999 and 2000 text-based writing tasks by reviewing the reading passage andattached items. The results are summarized in Attachment F.

Text-Based Writing Development During group discussion, teachers provided commentsand suggestions related to the development of the text-based writing. Their commentsfocused on three major issues: passage selection, wording of the prompt, and use of thewriting rubric (See Attachment I).

Passage Selection: Passages should be engaging and the difficulty level should beconsistent from year to year. Third grade teachers preferred realistic stories as thebasis for the text-based writing. Fifth grade teachers thought passages should beinformative selections dealing with social studies or science.

Wording of the Prompt: The wording in the prompt should always direct thestudents back to the text so that information from the text is included in theresponse. "Use details from.the text to support your answer," should be in allprompts. Students should understand the concepts implied in the wording of theprompt. Developers should take care in using "user accessible" language inwriting the prompts.

12.

11

Use of the Writing Rubric: Teachers discussed the possibility of adapting thegeneral writing rubric so that each text-based writing item would have an itemspecific writing rubric. With specific guidelines provided in these item specificrubrics, it would be easier for scorers to determine when a student's writing is offtopic.

Text-based Writing Instruction Teachers' comments related to instructional issues of thetext-based writing focused on professional development in crafting text-based promptsand on identifying a variety of reading passages with which to write such prompts. Theyemphasized the need to have students write in response to a variety of text types (literary,informative, and technical) across content areas, and for teachers to model the process ofmaking connections to the text and pulling out relevant details (See Attachment I).

Tenth grade teachers pointed out that most of the writing done by high schoolstudents is text-based, and that text-based writing is not a separate type of writing.Written responses to texts are produced as forms of persuasive, expressive, orinformative writing. High school teachers also expressed a concern regardingblock scheduling, where students may have only five weeks of instruction prior tothe administration of the DSTP. Finally, tenth grade teachers suggested that highschool English teachers have gone away from literary analysis in lieu of anemphasis on stand-alone writing prompts, which may sacrifice students' writingin response to text.

Fifth grade teachers stressed the importance of students making connections withcharacters in a story. They suggested that grade-level -teams or districtcommittees (lead by reading cadre representatives) develop questioning activitiesfor teachers to use to improve students' performance on text-based writing. Theyalso pointed out the need to release sample student responses to text-based writingprompts.

Limitations of the Study

As indicated in the beginning of this report, the current study was designed andconducted based on the available data within a short period of time. Due to thelimitations of the study, the author suggests cautions in reviewing, interpreting, and usingthe results of this study.

Information, such as sampling procedures and students' scores on the field test, isnot available for review and additional analysis.Even though the sample of student text-based writings was randomly selected, thesmall sample size, only 1% of he grade population used in re-scoring, may notaccurately reflect the characteristics of the population because of sampling errors.In addition, since the anchor papers were under review, the re-scoring process wasconducted without using anchor papers.It is very important to note that previous studies have shown that thegeneralization of writing performance is low across the purpose (or discourse) of

13

12

writing tasks, writing topics, and occasions, especially when there are only acouple of items used in the writing assessment. In 2000, a new text-based writingtask was introduced at each grade level, which may be one of the reasons for thefluctuation of the test scores. For example, third graders responded to aninformative writing task instead of a persuasive writing task. Similarly, the fifthgraders responded to an informative writing task in 1998 and 1999, but anexpressive writing task in 2000. These changes could account for the lowperformance.An important issue of educational measurement is reliability. Reliability ofperformance-based assessment, such as writing, is often defined by agreement ofreaders in scoring a single task given on a single occasion, called inter-readerreliability or inter-reader consistency. However, another component of reliabilityinvolves the consistency of measurement over repeated occasions given fixedreaders is called score reliability. Findings from early studies suggest that readerconsistency differed considerably ranging from .33 to .91 and score reliabilityranged from .26 to .60, which was dependent upon the number of points on thescoring scale, rating conditions, and changes in assessment programs (Dunbar etal, 1991; Fitzpatrick et al, 1994). The results of an experimental study conductedin Virginia (Moon et al, 1996) indicate that methods used for training and scoring(i.e., training readers to score multiple writing prompts at a single session ortraining readers sequentially score a new writing prompt) impact both reliabilityand validity. They also found that readers scored differently using the samescoring method on the same set of students' papers across years. To betterunderstand the nature of direct writing assessment and provide valid and reliablemeasures of student achievement, more research questions, such as the stability ofscoring over time, the process of reader training and scoring, and score reliabilityacross topics, discourses, and occasions, need to be further explored.

14

13

References

Moon, T. R., Loyd, B. H., Highes, K. R. & Winter, P. (1996). Scoring and training issuesinvolved in large-scale performance assessment.

Cantor, N. K. & Hoover, H. D. (1986). The reliability and validity of writing assessment:An investigation of rater, prompt within mode, and prompt between mode sources oferror. Paper presented at the AERA Annual Conference, San Francisco, CA

Fitzpatrick, A. R., Ercikan, K., Yen, W. M. & Ferrara, S. (1994). The consistencybetween ratings collected in different test years. Paper presented at the NCME AnnualConference, New Orleans. LA

Dunbar, S. B., Kortez, D. M. & Hoover, H. D. (1991). Quality control in thedevelopment and use of performance assessment. Applied Measurement in Education.4, 289-303.

State Summary Report -- Reading, Mathematics, Writing -- Spring 2000 Administration

State Summary Report in Writing -- 1998 DSTP



Tab

le 1

aM

eans

, Sta

ndar

d D

evia

tions

, and

Frq

uenc

yD

istr

ibut

ions

of W

ritin

g S

core

s by

Gra

de a

nd Y

ear

GR

AD

E 3

Wri

ting

Scor

eT

ext-

Bas

ed

2000

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1999

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1998

Stan

d-A

lone

Wri

ting

Tot

alN

%N

%N

%N

%N

%N

%N

%N

%N

%

151

0.0

142

0.0

30.

013

30.

011

0.1

120.

212

120.

139

0.5

530.

711

310.

493

1.2

152

2.0

104

0.0

116

1.5

50.

126

93.

56

0.1

463

6.2

95

0.0

359

4.7

170.

271

99.

436

0.5

928

12.4

888

1.1

911

12.1

610.

811

3714

.813

81.

811

9015

.87

261

3.4

1337

17.7

262

3.4

1477

19.2

438

5.8

1373

18.3

614

4719

.217

3523

.011

4914

.914

9419

.417

4023

.213

9818

.65

20.

0217

2923

.019

3825

.749

0.6

1801

23.4

1094

14.2

70.

115

2320

.310

5314

.04

470.

627

3736

.464

38.

538

75.

021

9928

.677

910

.150

06.

723

5331

.352

06.

93

550

7.31

762

10.1

430

5.7

2147

27.9

1170

15.2

576

7.5

2132

28.4

739

9.8

367

4.9

226

4135

.14.

846.

433

9144

.110

2413

.328

4737

.954

17.

21

4277

56.9

1714

22.3

2028

27.0

Tot

al75

1710

0.0

7517

100.

075

1710

0.0

7688

100.

076

8810

0.0

7688

100.

075

1410

0.0

7514

100.

075

1410

0.0

Mea

n1.

52-4

.54

6.06

2.18

4.26

6.44

2.15

4.70

6.85

SD0.

661.

271.

630.

851.

401.

940.

901.

411.

98Pu

rpos

e of

Info

rmat

ive

Exp

ress

ive

Pers

uasi

veIn

form

ativ

ePe

rsua

sive

Exp

ress

ive

Wri

ting

Tas

k

1716

Tab

le lb

Mea

ns, S

tand

ard

Dev

iatio

ns, a

nd F

rque

ncy

Dis

trib

utio

ns o

f Writ

ing

Sco

res

by G

rade

and

Yea

r

GR

AD

ES

Wri

ting

Scor

eT

ext-

Bas

ed

2000

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1999

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1998

Stan

d-A

lone

Wri

ting

Tot

alN

%N

%N

%N

%N

%N

%N

%N

%N

%

151

0.0

20

143

0.0

40.

17

0.1

135

0.0

90.

125

0.3

1222

0.3

108

1.4

133

1.8

1188

1.2

281

3.7

282

3.8

1010

0.1

250

3.5

90.

162

28.

214

0.2

678

9.2

96

0.0

718

10.1

50.

114

0218

.432

0.4

1065

14.4

812

11.

714

0519

.834

94.

614

5319

.124

73.

312

0816

.37

398

5.6

1477

20.8

498

6.5

1294

17.0

518

7.0

1473

19.9

624

2434

.213

4218

.925

7333

.813

4417

.716

8222

.713

2517

.95

80.

015

2821

.511

8916

.74

0.1

1384

18.2

794

10.4

80.

114

6219

.876

910

.44

155

2.2

1898

26.7

374

5.2

624

8.2

2350

30.9

192

2.5

1205

16.3

2756

37.3

249

3.4

397

413

.744

66.

221

02.

927

6136

.327

83.

710

51.

425

9635

.139

65.

417

82.

42

2958

41.8

252

3.5

2850

37.5

162

2.1

2365

32.0

287

3.9

129

8842

.213

6918

1220

16.5

Tot

al70

8410

0.0

7084

100.

0.7

084

100.

076

0810

0.0

7608

100.

076

0810

0.0

7394

100.

073

9410

0.0

7394

100.

0M

ean

1.76

5.02

6.78

2.35

5.17

7.52

2.52

4.89

7.41

SD0.

781.

271.

740.

871.

291.

860.

961.

361.

98P

urpo

seof

Exp

ress

ive

Per

suas

ive

Info

rmat

ive

Info

rmat

ive

Info

rmat

ive

Per

suas

ive

Wri

ting

Tas

k

Tab

le lc

Mea

ns, S

tand

ard

Dev

iatio

ns, a

nd F

rque

ncy

Dis

trib

utio

ns o

f Writ

ing

Sco

res

by G

rade

and

Yea

r

GR

AD

E 8

Wri

ting

Tex

t-B

ased

2000

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1999

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1998

Stan

d-A

lone

Wri

ting

Tot

alSc

ore

N%

N%

N%

N%

N%

N%

N%

N%

N%

151

0.0

10.

014

10.

05

0.1

100.

113

60.

010

0.1

330.

412

220.

278

1.0

205

2.6

1110

21.

318

42.

449

76.

310

70.

035

34.

58

0.1

572

7.3

208

0.2

763

9.9

95

0.0

948

12.3

25.0

.312

3015

.835

30.

913

0916

.58

220

2.8

2483

32.2

183

2.3

1757

22.5

1710

7.6

1514

19.1

776

19.

817

9523

.376

49.

815

3119

.615

6612

.712

6616

.06

3691

47.9

1070

13.9

2755

35.3

1228

15.8

2393

30.2

1152

14.5

51

0.01

1633

21.2

770

10.0

50.

120

0625

.782

510

.62

0.0

1011

19.7

757

9.5

476

0.9

1204

15.6

108

1.4

360

4.6

1589

20.4

.24

33.

159

37.

560

621

.623

93.

03

902

11.7

135

1.7

260.

320

6926

.530

33.

913

01.

725

1931

.771

4.4

169

2.1

242

5455

.334

0.4

3297

42.3

.16

12.

132

5541

.016

2.6

124

5731

.920

6326

.515

6519

.7

Tot

al76

8510

0.0

7685

100.

076

8510

0.0

7794

100.

077

9410

0.0

7794

100.

079

3410

0.0

7934

100.

079

3410

0.0

Mea

n1.

825.

577.

392.

105.

297.

392.

275.

457.

72SD

0.67

1.05

1.46

0.84

1.22

1.79

0.86

1.46

2.06

Purp

ose

ofE

xpre

ssiv

eIn

form

ativ

ePe

rsua

sive

Info

rmat

ive

Pers

uasi

veIn

form

ativ

eW

ritin

g T

ask

2120

Tab

le ld

Mea

ns, S

tand

ard

Dev

iatio

ns, a

nd F

rque

ncy

Dis

trib

utio

nsof

Writ

ing

Sco

res

by G

rade

and

Yea

r

GR

AD

E 1

0W

ritin

gT

ext-

Bas

ed

2000

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1999

Stan

d-A

lone

Wri

ting

Tot

alT

ext-

Bas

ed

1998

Sta

nd-A

lone

Wri

ting

Tot

alSc

ore

%N

%N

%N

%N

151

0.0

142

0.0

139

0.1

110.

21

0.0

1276

1.1

290.

416

0.2

1118

92.

717

02.

511

31:

710

60.

042

76.

142

20.

136

25.

432

14.

89

100.

197

014

.056

80.

278

011

.61

0.0

848

12.6

826

73.

816

4423

.819

483.

110

9216

.368

1.0

1185

17.6

762

89.

113

1219

.013

148.

612

3918

.539

25.

812

4818

.66

2703

39.2

1126

16.3

1646

24.6

1265

18.9

1703

25.3

1508

22.4

510

0.1

1447

20.9

892

12.9

575

19.6

989

14.8

1426

21.2

1156

17.2

430

54.

415

4822

.415

62.

211

51.

720

729

.145

36.

829

54.

427

2440

.520

83.

13

1268

18.3

183

2.6

901.

315

3622

.916

8.5

312

4.7

1687

25.1

271

4.0

120

1.8

231

3245

.410

21.

429

0343

.36

6.3

2910

43.3

139

2.1

121

7931

.621

4832

.118

2327

.2

Tot

al68

9410

0.0

6894

100.

068

9410

0.0

6702

100.

067

0210

0.0

6702

100.

067

2410

0.0

6724

100.

067

2410

0.0

Mea

n1.

965.

387.

341.

944.

876.

822.

074.

856.

92S

D0.

831.

211.

770.

781.

451.

950.

831.

151.

71P

urpo

seof

Per

suas

ive

Info

rmat

ive

Per

suas

ive

Info

rmat

ive

Per

suas

ive

Info

rmat

ive

Wri

ting

Tas

k

Tab

le 2

Mea

ns a

nd S

tand

ard

Dev

iatio

ns o

f Writ

ing

Sco

res

by G

ende

r, G

rade

, and

Yea

r

Gra

de G

ende

rT

ype

ofW

ritin

g20

0019

9919

98N

Mea

nS

.D.

NM

ean

S.D

.N

Mea

nS

.D.

3fe

mal

est

and-

alon

e37

474.

731.

2637

564.

481.

3837

084.

981.

39te

xt-b

ased

3747

1.58

0.69

3756

2.29

0.86

3708

2.30

0.91

writ

ing

tota

l37

476.

321.

6337

566.

761.

9237

087.

291.

96

mal

est

and-

alon

e37

704.

351.

2539

274.

061.

3938

044.

431.

38te

xt-b

ased

3770

1.45

0.62

3927

2.07

0.83

3804

2.00

0.86

writ

ing

tota

l37

705.

811.

5839

276.

131.

9138

046.

431.

91

5fe

mal

est

and-

alon

e35

465.

251.

2337

175.

411.

2837

555.

131.

36te

xt-b

ased

3546

1.89

0.80

3717

2.49

0.87

3755

2.67

0.95

writ

ing

tota

l35

467.

141.

7237

177.

911.

8337

557.

791.

97m

ale

stan

d-al

one

3537

4.78

1.27

3886

4.94

1.26

3633

4.67

1.31

text

-bas

ed35

371.

630.

7338

862.

210.

8436

332.

360.

93w

ritin

g to

tal

3538

6.42

1.68

3886

7.15

1.81

3633

7.03

1.91

8fe

mal

est

and-

alon

e38

745.

711.

0239

405.

561.

1438

895.

781.

40te

xt-b

ased

3874

1.93

0.69

3940

2.25

0.85

3889

2.45

0.85

writ

ing

tota

l38

747.

651.

4439

407.

811.

6938

898.

231.

96

mal

est

and-

alon

e38

105.

421.

0737

935.

041.

2440

455.

13.

1.44

text

-bas

ed38

101.

700.

6237

931.

950.

8140

452.

100.

84w

ritin

g to

tal

3811

7.12

1.44

3793

6.99

.1.

7840

457.

232.

0210

fem

ale

stan

d-al

one

3547

5.56

1.17

3399

5.10

1.40

3455

5.10

1.14

text

-bas

ed35

472.

100.

8333

992.

060.

7834

552.

240.

83w

ritin

g to

tal

3547

7.66

1.72

3399

7.16

1.87

3455

7.34

1.68

mal

est

and-

alon

e33

475.

181.

2332

634.

661.

4632

694.

591.

09te

xt-b

ased

3347

1.82

0.81

3263

1.83

0.77

3269

1.89

0.80

writ

ing

tota

l33

477.

001.

7532

636.

491.

9632

696.

481.

62

2425

Tab

le 3

aC

orre

latio

n M

atrix

of D

ST

P W

ritin

g, D

ST

P R

eadi

ng, a

ndS

AT

9 R

eadi

ng S

core

s by

Gra

deG

RA

DE

3M

C It

ems

CR

Item

sP

assa

geIte

m 6

7T

ext-

base

d S

tand

-alo

neW

ritin

gR

eadi

ngS

AT

9

MC

Item

s1.

000.

460.

770.

240.

190.

270.

290.

630.

51C

R It

ems

1.00

0.92

0.66

0.31

0.36

.0.

400.

710.

51P

assa

ge1.

000.

580.

310.

380.

420.

780.

59Ite

m 6

7T

v.1,

41.

000.

220.

240.

280:

440.

29T

ext-

base

d1.

000.

360.

690.

370.

28S

tand

-alo

ne1.

000.

930.

50.0

.43

Writ

ing

1.00

0.53

0.45

Rea

ding

1.00

0.88

SA

T9

1.00

GR

AD

E 5

MC

Item

sC

R It

ems

Pas

sage

Item

56

Tex

t-ba

sed

Sta

nd-a

lone

Writ

ing

Rea

ding

SA

T9

MC

Item

sC

R It

ems

Pas

sage

Item

56

Tex

t-ba

sed

Sta

nd-a

lone

Writ

ing

Rea

ding

SA

T9

1.00

0.40

1.00

0.75

0.91

1.00

0.31

0.73

0.67

1.00

0.26

0.44

0.44

0.45

1.00

0.31

0.42

0.45

0.37

0.41

1.00

0.35

0.50

0.52

0.47

0.75

0.91

1.00

0.65

0.66

0.77

0.51

0.44

0.50

0.56

1.00

0.52

0.45

0.56

0.37

0.33

0.41

0.45

0.86

1.00 27

26

GR

AD

E 8

Tab

le 3

bC

orre

latio

n M

atrix

of D

ST

P W

ritin

g, D

ST

P R

eadi

ng, a

ndS

AT

9 R

eadi

ng S

core

s by

Gra

de

MC

Item

sC

R It

ems

Pas

sage

Item

47

Tex

t-ba

sed

Sta

nd-a

lone

Writ

ing

Rea

ding

SA

T9

MC

Item

s1.

000.

480.

790.

340.

340.

380.

430.

690.

58C

R It

ems

1.00

0.92

0.64

0.46

0.41

0.50

0.71

0.52

Pas

sage

1.00

0.61

0.47

0.46

0.54

0.81

0.63

Item

47

1.00

0.57

0.37

0.53

0.56

0.38

Tex

t-ba

sed

1.00

0.41

0.75

0.52

0.39

Sta

nd-a

lone

1.00

0.91

0.55

0.48

Writ

ing

1.00

0.63

0.52

Rea

ding

1.00

0.85

SA

T9

1.00

GR

AD

E 1

0M

C It

ems

CR

Item

sP

assa

geIte

m 4

6T

ext-

base

d S

tand

-alo

neW

ritin

gR

eadi

ngS

AT

9

MC

Item

s1.

000.

350.

700.

320.

300.

350.

380.

590.

50C

R It

ems

1.00

0.91

0.70

.0.

470.

440.

520.

640.

44P

assa

ge1.

000.

670.

490.

490.

560.

740.

55Ite

m 4

61.

000.

600.

460.

600.

590.

40T

ext-

base

d1.

000.

480.

800.

540.

39S

tand

-alo

ne1.

000.

910.

590.

48W

ritin

g1.

000.

660.

52R

eadi

ng1.

000.

84S

AT

91.

00

2829

Tab

le 4

Mea

ns a

nd F

requ

ency

Dis

trib

utio

ns o

f Writ

ing

Sco

res

in F

ield

Tes

t and

200

0 D

ST

P

scor

eG

rade

3fie

ld te

st20

00 D

ST

PG

rade

5fie

ld te

st20

00 D

ST

PG

rade

8fie

ld te

st20

00 D

ST

PG

rade

10

field

test

2000

DS

TP

N.

%N

.%

N.

%N

.%

N.

%N

.%

N.

%N

.%

56

0.7

20.

04

0.4

80.

18

0.9

10.

021

2.5

100.

14

132

14.5

470.

650

5.6

155

2.2

526.

176

0.9

139

16.5

304

4.4

336

440

.155

07.

438

142

.897

413

.719

022

.590

211

.829

535

.112

6818

.42

263

29.0

2641

35.1

356

40.1

2958

41.8

351

41.5

4254

55.4

263

31.2

3132

45.5

114

315

.742

7756

.999

11.1

2988

42.2

246

29.0

2457

31.9

124

14.7

2179

31.6

mea

n2.

5510

0.0

1.52

100.

02.

4410

0.0

1.76

100.

02.

0810

0.0

1.82

100.

02.

6110

.0.0

1.96

100.

0N

.90

875

1789

070

8484

776

8584

268

94

* T

he fi

eld

test

res

ults

are

base

d on

the

writ

ing

scor

es fo

r st

uden

ts in

gra

des

4, 6

,'9, a

nd 1

1.

3130

pr,

'ItA

IM'

4 ti

ii;I

If41

1]

&.,A

$1'I,

1t..

kakita

..

v-I

tT

a,

Vir

4injf

ed,

.,41,

, 001

Fiw

v.A

1r

ir,,

9.i

y,,,,

,, iii 1,.

'1,1

4j

1 1

i1nik

1tiit

-,

,11

..

1,'

d:Jo pr

Oif

;IL

,

4

II.

',511

liir

,A

a.1,I'(A

t

-'41'

Iet-

111

i. J0'

1

$ir

11 40

. $1.4I InA

'ilL1 i1;

Net

v: 1ga.

4

.I;

14. Ii 0

Ilii.Y

,,t1

0'11,.

r4iiq '

, 41.1

,-di

e

ea1P

Freq

uenc

y(%)O

6b

1 Try

i

Nilr

JrIIIi.

I'

fli,r'''P

P...1l

ell',1"

1qr

n

,ir.1

1,1 p

.4,1

,Jr

tOr

_fi

ri

1, IP`r

J1'

14. 1,1

a'1`

1

11'it

,N.

u icfi

rt

-I

I',C

,L,,.

'01'

J11

, -4,

IIII ;1.

4,,

41,

. ,

,..,4

6%

Aa

It. 04

Ar

ry4.

1.'''

I"'

i'V

IS

lbJ,

I14,

,,h

11

Poll'

11fr

, ,i,

1. I' 11

10

A,

Q,-

,1

Id,,

00 .,I

ii-' 11

4.

1 Itia

,

,' r11

.1,-

,i1(1

11I

ti4

r0:1k.

1,I

ls--

It'

Wei

t,g`

I11

.

.1V11

q

L.'I.1

IL

01k

41

,'11

g.0

1r

IDI.

I.1

4"7A

or

tql

r 10.

'0

III

, X-.k

ifi-

!AL

if,.'

' ''1

41.

!II,

pqj1,

111

T4

.

1hP"

I'1IP

.!,

g, op

0

aw

Tab

le 5

aS

umm

ary

of D

iscr

epan

cies

of T

ext-

Bas

edW

ritin

g S

core

sbe

twee

n 20

00 D

ST

P a

ndR

e-S

corin

gD

iff S

core

Gra

de 3

Gra

de 5

Gra

de 8

Gra

de 1

0S

ub-T

otal

N.

N.

N.

N.

N.

30

0.0

00.

01

1.0

00.

01

0.3

210

10.0

44.

07

7.0

66.

027

6.8

125

25.0

2424

.036

36.0

2828

.011

328

.30

5555

.065

65.0

4545

.044

44.0

.20

952

.3-1

33.

05

5.0

55.

015

15.0

287.

0-2

22.

00

0.0

00.

00

0.0

20.

5-3

00.

00

0.0

00.

00

0.0

00.

05

5.0

22.

06

6.0

77.

020

5.0

Sub

-Tot

al10

010

0.0

100

100.

010

010

0.0

100

100.

040

010

0.0

Tab

le 5

bS

umm

ary

of D

iscr

epan

cies

of T

ext-

Bas

edW

ritin

g S

core

sbe

twee

n 20

00 D

ST

P a

ndR

e-S

corin

g (A

djus

ted)

Diff

Sco

reG

rade

3G

rade

5G

rade

8G

rade

10

Sub

-Tot

alN

.N

.N

.N

.N

.

30

0.0

0.0.

01

1.1

00.

01

0.3

210

10.5

44.

17

7.4

66.

527

8.4

125

26.3

2424

.536

38.3

2830

.111

335

.00

5557

.965

66.3

4547

.944

47.3

209

64.7

-1.

33.

25

5.1

55.

315

16.1

288.

7-2

22.

10

0.0

00.

00

0.0

20.

6-3

00.

00

0.0

00.

00

0.0

00.

0

Sub

-T

otal

9510

0.0

9810

0.0

9410

0.0

9310

0.0

380

117.

6D

iff S

core

= R

e-S

core

- O

rigin

al S

core

*= N

umbe

r of

pape

rs w

ere

non-

scor

able

bec

ause

of m

issi

ngpa

ges.

3534

Tab

le 6

aS

umm

ary

of D

iscr

epan

cies

in S

corin

g T

ext-

Bas

edW

ritin

gfr

om fi

eld

test

, 200

0 D

ST

P, a

nd R

e-S

corin

g

Gra

de 3

Gra

de 5

scor

efie

ld te

st20

00 D

ST

Pre

-sco

ring

field

test

2000

DS

TP

re-s

corin

gN

.%

N.

%N

.N

.%

N.

N.

56

0.7

20.

00

0.0

40.

48

0.1

11.

04

132

14.5

470.

61

1.1

5.0

5.6

155

2.2

22.

13

364

40.1

550

7.4

1717

.938

142

.897

413

.719

19.6

226

329

.0.2

641

35.1

3941

.135

640

.129

5841

.842

43.3

114

315

.742

7756

.938

40.0

9911

.129

8842

.233

34.0

mea

n2.

5510

0.0

1.52

100.

01.

7910

0.0

2.44

100.

01.

7610

0.0

2.09

100.

0N

:90

875

1795

890

7084

97

Tab

le 6

bS

umm

ary

of D

iscr

epan

cies

of T

ext-

Bas

edW

ritin

g S

core

sin

Fie

ld T

est,

2000

DS

TP

, and

Re-

Sco

ring

Gra

de 8

Gra

de 1

0sc

ore

field

test

2000

DS

TP

re-s

corin

gfie

ld te

st20

00 D

ST

Pre

-sco

ring

N.

N.

N.

N.

%N

.N

.

58

0.9

10.

00

0.0

212.

510

0.1

00.

04

526.

176

0.9

88.

513

916

.530

44.

46

-6.5

319

022

.590

211

.828

29.8

295

35.1

1268

18.4

3234

.42

351

41.5

4254

55.4

4446

.826

331

.231

3245

.534

36.6

.124

629

.024

5731

.914

14.9

124

14.7

2179

31.6

1819

.4

mea

n2.

0810

0.0

1.82

100.

02.

4610

0.0

2.61

100.

01.

9610

0.0

2.12

96.8

N.

847

7685

9484

268

9490

Zer

o sc

ores

are

not

incl

uded

from

re-

scor

ing

for eq

uiva

lent

com

paris

on w

ith th

e fie

ld te

st a

nd th

e20

00 D

ST

P.

3 6

Tab

le 7

Num

ber

of R

eade

rs U

sed

in R

e-S

corin

g T

ext-

Bas

ed W

ritin

g

N. o

f Rea

ders

Gra

de 3

Gra

de 5

Gra

de 8

Gra

de 1

0N

.%

N.

%N

.N

.

12

2.1

11.

05

5.3

2021

.52

22.

127

27.6

4143

.610

10.8

310

10.5

3737

.838

40.4

3436

.64

2223

.226

26.5

66.

420

21_5

559

62.1

77.

14

4.3

99.

7

Sub

-Tot

al95

100.

098

100.

094

100.

093

100.

0

The

dat

a pr

esen

ted

in th

is ta

ble

show

the

num

ber

of r

eade

rs u

sed

inre

-sco

ring

the

text

-bas

ed w

riitn

g in

this

stu

dy.

38

Tab

le 8

Rel

iabi

lity

Indi

cato

r of

Tex

t-B

ased

Writ

ing

Sco

res

inR

e-S

corin

g

Gra

de 3

Gra

de 5

.G

rade

8G

rade

10

Diff

Num

ber

of R

eade

rs-

Num

ber

of R

eade

rsN

umbe

r of

Rea

ders

Num

ber

of R

eade

rsS

core

12

34

51

23

45

12

34

51

23

45

31

21

45

21

12

31

11

41

11

33

189

95

12

1417

21

91

710

1-1

12

12

11

31

12

27

31

-22

-3

Tot

al2

47

2710

137

32

1723

43

123

1813

35.

010

.0.

17.5

67.5

30.3

39.4

21.2

9.1

4.1

34.7

46.9

8.2

6.1

24.5

6.1

36.7

26.5

6.1

The

dat

a sh

own

in th

is ta

ble

only

incl

ude

the

diffe

rent

text

-bas

ed w

ritin

g sc

ores

bet

wee

n 20

00 D

ST

P a

ndre

-sco

ring.

41

40

Attachment AWriting Scoring Rubric

Del

awar

e St

uden

t Tes

ting

Prog

rn -

Gen

eral

Rub

ric

for

Wri

ting

The

fol

low

ing

char

acte

rist

ics

dete

rmin

e th

e su

cces

s of

the

resp

onse

in m

eetin

g th

e ne

eds

of th

e au

dien

ce a

nd f

ulfi

lling

the

wri

ting

purp

ose.

' '''.

::

',Sai

ii:e*

. Of

'...S

eafe

-Of

''co

res

oSc

ore

poin

t 5 m

eets

all

the

Uni

fied

with

sm

ooth

Gen

eral

ly u

nifi

ed w

ith s

ome

Min

imal

ly u

nifi

ed a

nd m

ayL

acks

uni

ty.

crite

ria

liste

d in

sco

re p

oint

tran

sitio

ns, a

cle

ar a

ndtr

ansi

tions

, a c

lear

lack

tran

sitio

ns o

r an

4. I

n ad

ditio

n, a

pap

erre

ceiv

ing

this

sco

re s

how

san

exc

eptio

nal a

war

enes

s of

logi

cal p

rogr

essi

on o

f id

eas,

and

an e

ffec

tive

intr

oduc

tion

and

clos

ing.

prog

ress

ion

of id

eas,

and

an

intr

oduc

tion

and

clos

ing,

intr

oduc

tion

or c

losi

ng.

Som

e sp

ecif

ic d

etai

ls b

ut

No

or f

ew s

peci

fic

deta

ilsth

at a

re m

inim

ally

elab

orat

ed.

read

ers'

con

cern

s an

d ne

eds.

Suff

icie

nt, s

peci

fic,

and

Spec

ific

det

ails

but

may

be

insu

ffic

ient

, irr

elev

ant,

orm

ay b

e in

suff

icie

nt,

irre

leva

nt, a

nd/o

r no

tFr

eque

nt a

nd s

ever

eT

he s

tude

nt m

ay h

ave

rele

vant

det

ails

that

are

ful

lyno

t ful

ly e

labo

rate

d.el

abor

ated

.se

nten

ce f

orm

atio

n er

rors

show

n an

exc

eptio

nal u

seel

abor

ated

.an

d/or

a la

ck o

f se

nten

ceof

:G

ener

ally

com

plet

eSo

me

sent

ence

for

mat

ion

vari

ety.

Con

sist

ently

com

plet

ese

nten

ces

with

suf

fici

ent

erro

rs a

nd a

lack

of

sent

ence

Dev

elop

men

t str

ateg

ies

spec

ific

to th

e pu

rpos

efo

r w

ritin

g

sent

ence

s w

ith a

ppro

pria

teva

riet

y in

leng

th a

ndst

ruct

ure.

vari

ety

in le

ngth

and

stru

ctur

e.,

vari

ety.

Som

etim

es g

ener

al a

nd

Oft

en g

ener

al, r

epet

itive

,an

d/or

con

fusi

ng w

ord

choi

ce.

Dis

tinct

ive

styl

e, v

oice

,to

neA

con

sist

ent s

tyle

with

Som

e st

yle

and

gene

rally

prec

ise

wor

d ch

oice

.re

petit

ive

wor

d ch

oice

.Fr

eque

nt a

nd s

ever

e er

rors

Lite

rary

dev

ices

prec

ise

and

vivi

d w

ord

Seve

ral k

inds

of

erro

rs in

in s

tand

ard

wri

tten

Eng

lish

Com

posi

tiona

l ris

ksch

oice

.So

me

erro

rs in

sta

ndar

dst

anda

rd w

ritte

n E

nglis

h th

atth

at in

terf

ere

with

wri

tten

Eng

lish

that

rar

ely

inte

rfer

e w

ithun

ders

tand

ing.

Few

, if

any,

err

ors

inst

anda

rd w

ritte

n E

nglis

h th

atdo

not

inte

rfer

e w

ithun

ders

tand

ing.

inte

rfer

e w

ithun

ders

tand

ing.

unde

rsta

ndin

g.

For

non-

scor

able

res

pons

es s

ee b

elow

:

aB

lank

Ref

usal

eO

ff to

pic

0Il

legi

ble

Wri

tten

in a

lang

uage

oth

er th

ana

Insu

ffic

ient

Eng

lish

/elp

5/10

/99

43B

EST

CO

PY A

VA

ILA

BL

E

44

Attachment BInstructions for Anchor Paper Review

INSTRUCTIONS FOR ANCHOR PAPER REVIEW

1. In the folder, you will find the following:

Two copies of the Form of Anchor Paper Review, one for each gradeNon-disclosure FormWriting Scoring RubricReading passage and the question for the text-based writingAnchor papers for selected grades

2. Procedures for Anchor Paper Review:

Step One: (On your own)Sign the Non-disclosure Form.Read the Instructions for Anchor Paper Review.Read the reading passage and the question for the text-based writing.Read the scoring rubric.

Step Two: (On your own)Read each anchor paper carefully.Rank all the anchor papers from the lowest (as #1) to the highest (whateverthe number applies). You may assign the same rank order to more than onepaper, such as_two papers are ranked as #3 and the following one should be #5instead of #4.Assign a score to each anchor paper using the 5-point rubric. You may usehigh or low to differentiate papers even though they receive the same scorepoint(s).Record your scores using the Form of Anchor Paper Review. Match the Codewith the 1-digit number sequential number on the right top of each paper andfill in the rank order and the score you have assigned to each paper.You also need to record the 11-digit paper number (located on the rightbottom of each paper) in under the column of Paper Number of the Form ofAnchor Paper Review.Provide your comments using the space below. the Form.Please return the entire folder to Kim by August 7, 2000.

Step Three: (Group meeting if necessary)Have a group meeting to share your scores with other reviewers.Group DiscussionMake your second judgmentCompare your judgments with the original scores.

Attachment CProcedures For Re-scoring

47

Agenda

Study for the DSTP Writing Assessment8:30 am-3:30 pm, Tatnall Building, Dover, August 17, 2000

Welcome and Study Brief (Zhang)

Scoring Rubrics Training

Holistic Scoring Rubrics for Writing (Bolig)Holistic Scoring Rubrics for Reading (Bolig)Analytic Scoring Rubrics for Writing (Kelley)

8:30-8:45

8:45-9:05

Evaluate Text-based Writing Tasks 9:05-9:30

First Round Evaluation

Review and Re-score Students' Writings 9:30-11:00

First Round Scoring

Small Group Discussion 11:00-12:00

Lunch 12:00-12:40

Continue Group Discussion and Second Round Scoring 12:40-2:00

Evaluate Text-based Writing Tasks 2:00-2:30

Second Round Scoring

Grade-Group Discussion 2:30-3:30

Recommendations. CommentsEvaluation

Appendix A

Delaware English Language Arts Content Standards

Students in Delaware public schools, using the process of effective readers, writers,listeners, viewers, and speakers, will be able to:

Standard #1Use written and oral English appropriate for various purposes and audiences.

Standard #2Construct, examine, and extend the meaning of literacy, informative, and technical textsthrough listening, reading, and writing.

Standard #3Access, organize, and evaluate information gained through listening, reading, and viewing.

Standard #4Use literary knowledge accessed through print and visual media to connect self to societyand culture.

49

17

Appendix B 1

Definitions of Cognitive Categories forDSTP Reading Comprehension Assessment

Three stances of are used to measure the depth of reading comprehension. These stancesare: Determining Meaning, Interpreting Meaning, and Extending Meaning.

Questions in the Determining Meaning stance require the reader to demonstrate anoverall understanding of the passage. The focus is on how the reader begins to makemeaning of the text.Questions in the Interpreting Meaning stance require the reader to go beyond the initialunderstanding to develop an interpretation of the text. The reader goes beyond firstimpression to construct a more complete understanding of what has been read.Questions in the Extending Meaning stance require the reader to stand apart from thetext and critically consider it. This stance involves critical examination, evaluation,and analysis.

520

Delaware Student Testing Program InStructional Guide for Writing(An Analytic Adaptation of the DSTP General Rubric for Writing)

The following characteristics determine the s.iicccss of.thP response n meeting the heeds of the audience andfulfilling . .

ptirP05.P.

_I

Generally unifiedwith sometransitions, a clearprogression ofideas, and anintroduction andclosing.

I

Minimally :unifiedand may lacktransitions, or anintroduction orClosing.

1 . I

Lacks unity.

.

Score point 5meets all the

. .criteria listed inscore point 4 . Inaddition, a paper

.receiving thisscore shows anexceptionalawareness ofreaders' concernsand needs.

The student mayhave shown anexceptional useof'

Developmentstrategiesspecific to thepurpose forwritingDistinctivesode, voice,toneLiterarydevices

Compositionalrisks

o9,

tig1.,,t40

0

ig.

CS

09.

'RI

iw,)0g

0cn

'-oc)..al0

3

v)

Iin0.)

0c--)

1

Unified Withsmooth transitions,a clear and logicalprogresSion ofideas; and an

. ,.effectiveintroduction andclosing,

Sufficient;specific, andrelevant detailsthat are fullyelaborated.

Specific detailsbut may beinsufficient,irrelevant, or notfully elaborated.

Some specificdetails but may beinsufficient,irrelevant, and/ornot elaborated.

No or few specificdetails that areminimallyelaborated.

i

Consistentlycompletesentences withappropriate varietyin length andstrucnire.

Generallycompletesentences withsufficient varietyin length andstrutture.

Some sentenceformation errorsand a lack Ofsentence variety.

Frequent and .

severe sentence 1

formation mars :

and/Or a lack of iv

sentence variety.

i

A consistent stylewith precise andvivid word choice

Some style andgenerally preciseword choice.

Sometimes generaland repetitiveword choice.

Often general,+,

repetitive, and/orconfusing word 1r:

choice. 1,

,i

Few, if any, errorsstandard written

English that do notinterfere withunderstanding.

Some errors instandard writtenEnglish that rarelyinterfere withunderstanding.

Several kinds oferrors in standardwritten Englishthat interfere withUnderStanding.

1!

Frequent and ;I

severe errors instandard written

.1.,

English that ...li

interfere with4,

Understanding)

51

pThe Joy of ScoringOne part holisticFive parts analyticApply to writing samplesPray for inter-rater reliability

Holistic vs. Analytic Scores

There is no concrete correlation between the holistic score for a. piece of writing and theanalytic scores for each of the textual features at work within that piece of writing. Aholistic 3 does not mean that each analytic score is also a 3. Nor does a holistic 3 meanthat the sum of the analytic Scores, is 15, which when divided by 5 (the number of textualfeatures) would "give us" that holistic 3. It's not that simple.

First, the textual features are not weighted equally. Generally, Organization andDevelopment carry far more clout than Conventions. Sentence Formation andStyle/Word Choice, While also considered more significant than Conventions, are notquite as critical to a piece as Organization and Development. Additionally, the nature ofa particular prompt, its audience, or its purpose may subtly emphasize a particular textualfeature and thereby influence the scoring.

Second, in addition to the distinct analytic scores, a holistic score also takes into accounthow those individual textual features work together to create the whole piece of writing.In essence, the whole is "greater" than the sum of its parts. But "greater" does not implythat the holistic score is higher; it means that there is more to consider in the holisticscore than just the analytic scores for each of the textual features.

Our TaSit

When. analytically scoring the samples for this study, consider only the fourlevels below.Score points 4 and 5 from the DSTP Rubric are lumped together for the purposes of thisStudy. We are less concerned with determining the differences between those scorepoints than we are with determining the differences, in terms of the textual features,between score points 3, 2, and 1.

'-'- Score Point from 4/5DSTP _Rubric 3 2 1

Descriptor . Very Good. -

Good Enough So-So Not So Hot

52 1/2

The Chicken and the Egg: Which comes first, holistic or analytic?

What holistic score should we assign to a paper with the following configuration ofanalytic scores? Why?

Example. A

Textual Feature AnalyticScore

Organization 1/2Development 1/2Sentence Forination 3/4Style/Word Choice 3/4Conventions 3/4

Example

Textual Feature AnalyticScore

Organization 1/2DeVelopinent 2/3Sentence Forniation 3/4Style/Word ChOice 3/4Conventibris 3/4

Example C

Textual Feature lAnalyticScore

Organilation 1/2Developrnent 1/2Sentene& Fon-nation 2/3Style/Word Choice 2/3ConVentions 3/4

Example D

Textual Feature Aria lytiCScore

Organization 3/4DevelOpMent 3/4Sentence Formation 3/4Style /Word Ch dice 3/4Conventions 3/4

Ejrample

Textual Feature AnalYticScore

Organization 2/3DeVelOPMent 3/4SehtenCe Formation 2/3Style/Word Choibe 2/3ConventioriS 3/4

E)rampip.F

Textual Feature Analytic.Score

Organization 3/4Development 3/4Sentence Formation 2/3Style/Word Choice 2/3Conventions 3/4

Begin developing a. sense of the holistic score by looking at Organization andDevelopment These two textual features carry the most weight.The combination of analytic scores for Sentence FormatiOn, Style /Word. Choice,and Conventions may shifteither up or downthe holistic score developed byconsidering the analytic scores for Organization and Developirient This shift willseldom be more than one score point.Conventions alone do not make a piece of writing better; they merely make itcorrect in terms of Standard Written English. Conventional correctness does notimprove the quality of a piece of writing that lacks unity, specific detail, sentencevariety; style, etc. Quality comes with improvements in the other textual features.Conventions do make the piece more readable, but readable writing that saysnothing and makes no sense still says nothing and makes no sense.

2/253

Attachment DList of Panel Members

54

Anchor Paper Review CommitteeFor FY 2000 DSTP Writing Study

Name Gender District School Test DevelopmentCommittee

Grade 3 & 5Cookie Bo lig F Department of Ed. Assessment & Analysis YJackie Shockley F Cape Henlopen R.A. Shields Elementary YMarty Hodgkins F Appoquinimink Redding Intermediate YLinda Mitchell F Indian River East. Millsboro Elementary N

Chris Evans F Brandywine Mt. Pleasant High Y

Grade 8 & 10Mike Boyd M Lake Forest Lake Forest High YKate Szegda F Red Clay District Office YDoug Grudzina M Capital Dover High NMike Kelly M Department of Ed. Curriculum Development Y

Anchor Paper Pulling Review CommitteeFor DSTP Writing Study

Name Gender District School Test DevelopmentCommittee

Grade 3Jackie Shockley F Cape Henlopen Shields Elementary YJanice Trainer F Christina Wilson Elementary NAnn Whitman F Milford Benjamin Banneker ElementaryNancy Carter F Milford Benjamin Banneker Elementary NCookie Bo lig F Department of Ed. Assessment & Analysis

Grade 5Linda Mitchell F Indian River East Millsboro NMarty Hodgkins F Appoquinimink Redding Intermediate YMary Currie F Milford Milford Middle YChristine Poehlmann F Appoquinimink Redding Intermediate NKaren Sheets F Woodbridge Woodbridge Elementary N

Grade 8Kate Szegda F Red Clay District OfficeGinger Angstadt. F Capital Central Middle School YLinda Poorman F Colonial Wallace Wallin NGwnne Ash F University of DEDenise Speicher F Indian River District Office Y

Grade 10Mike Boyd M Lake Forest Lake Forest High YCatherine Laverick F NCCVT Hodgson Vo-Tech NJohn Drumheller M Cape Henlopen Cape Henlopen High NJudy Smith F Polytech Polytech High NPam Wilson F Polytech Polytech High NDoug Grudzina M Capital Dover High NMike Kelly M Department of Ed. Curriculum Development Y

Attachment ECorrelation Matrix

Between Reading and Writing

Correlation Matrix between Reading and Writing

1999 READING TEXT-BASED STAND-ALONE WRITINGGrade 3READINGTEXT-BASEDSTAND-ALONEWRITING TOTAL

1.00

0.60

0.57

0.68

1.00

0.45

0.771.00

0.92 1.00Grade 5Reading 1.00Text-Based 0.56 1.00Prompt 0.55 0.46 1.00Writing 0.64 0.79 0.91 1.00Grade 8Reading 1.00Text-Based 0.65 1.00Prompt 0.55 0.49 1.00Writing 0.68 0.80 0.91 1.00Grade 10Reading 1.00Text-Based 0.55 1.00Prompt 0.59 0.47 1.00Writing 0.66 0.75 0.92 1.00

1998 Reading Text-B Prompt WritingGrade 3Reading.Text-BasedPromptWriting

1.00

0.56

0.52

0.63

1.00

0.44

0.771.00

0.91 1.00Grade 5Reading 1.00Text-Based 0.60 1.00Prompt 0.57 0.45 1.00Writing 0.68 0.79 0.90 1.00Grade 8Reading 1.00Text-Based 0.60 1.00Prompt 0.63 0.54 1.00Writing 0.70 0.80 0.94 1.00Grade 10Reading 1.00Text-Based 0.56 1.00Prompt 0.56 0.47 1.00Writing 0.65 0.81 0.90 1.00

All correlation coefficients are calculated based on aggregated data.

58

Attachment FSummary of Anchor Paper Review

59

Sum

mar

y of

An,

.,or

Pape

r R

evie

wG

rade

3

Sequ

entia

l1s

t Rou

nd2n

d R

ound

Ori

g.D

iff

Not

esN

umbe

rPa

per

N.

12

34

51

23

45

mea

nsc

ore

scor

e

157

6110

0443

01

2-1

11

11.

51

11

1.1

10.

1#1

& #

2 L

ong

sent

ence

! Pr

obab

ly s

core

d be

tter

for

read

ing.

257

8630

0947

01

11

11

11

11

11

10

357

8540

0021

01+

11+

11

1.5

11.

51

11.

21

0.2

#3 -

Poi

nt o

f vi

ew f

ew d

etai

ls.

457

8630

1070

02-

22-

12

1.5

21.

52

21.

82

-0.2

558

4680

0355

02

2+2

22

22.

52

22

2.1

20.

16

5785

4011

760

2-2

22

21.

52

22

1.9

2-0

.1#6

- S

houl

d sc

ore

low

for

rea

ding

.7

5785

4000

140

33

3-3

3-3

32.

53

2.5

2.8

20.

88

5810

9020

110

33

32

3-3

33

22.

52.

73

-0.3

958

2700

0925

03+

3+3+

33

3.5

3.5

3.5

33

3.3

30.

310

5887

4001

980

3+4+

43

33.

54.

54

33

3.6

3.5

0.1

#10

- O

rigi

nal s

core

sho

uld

be a

3+

acc

ordi

ngto

the

rubr

ic. I

t mee

ts th

e as

pect

s of

a 3.

It d

oes

not m

eet

all a

spec

ts c

onsi

sten

tly f

or a

4.

1158

1090

2828

04-

3+4+

33+

3.5

3.5

4.5

33.

53.

64

-0.4

1258

2800

1126

04+

45

44-

4.5

45

43.

54.

24

0.2

Dis

crep

ancy

%8%

Com

men

ts

Sum

mar

y of

An,

..or

Pape

r R

evie

wG

rade

5

Sequ

entia

lN

umbe

rPa

per

N.

1st R

ound

2nd

Rou

ndO

rig.

Dif

fsc

ore

Not

es1

23

45

12

34

5m

ean

scor

e

158

0830

1717

01

11

11

11

11

11

0.5

0.5

258

0830

1062

01

11

11

11

11

11

10

357

8540

0182

01+

2-1

11

1.5

1.5

11

1.2

1.5

-0.3

458

2700

1754

02-

2-2-

21+

1.5

1.5

1.5

21.

51.

61.

50.

15

5846

8001

120

22

22

2-2

22

21.

51.

92

-0.1

618

5400

1210

2+2+

22+

22.

52.

52

2.5

22.

32.

5-0

.27

5827

0013

740

33

3-3

2+3

32.

53

2.5

2.8

2.5

0.3

# 7-

Det

ails

not

ful

ly e

labo

rate

d. S

ome

styl

e.8

5784

5011

970

33+

33

3-3

3.5

33

2.5

33

09

5810

9021

090

34-

33+

33

3.5

33.

53

3.2

3.5

-0.3

1058

4590

0138

03+

4-4-

43

3.5

3.5

3.5

3.5

33.

43.

5-0

.111

5846

8800

930

44

43

3.5

44

43

3.7

4-0

.312

5845

9012

340

4+4+

44

3+4.

54.

54

43.

54.

14.

5-0

.413

5827

0017

270

55

4+4

4.5

55

4.5

44.

64.

50.

1#1

3 -

Voi

ce!

Ton

e!

Dis

crep

ancy

1

%8%

Com

men

tsT

hese

pap

ers

seem

ed to

be

in o

rder

alre

ady.

Sum

mar

y of

Art

-or

Pape

r R

evie

wG

rade

8

Sequ

entia

lN

umbe

rPa

per

N.

1st S

core

2nd

Scor

eO

rig.

scor

eD

iff

scor

eN

otes

12

34

51

23

45

mea

n

158

4590

0398

01

21-

11

10.

51

0.88

1-0

.13

258

0830

0248

01

2-1

11

11

1.00

10.

003

5845

9005

660

11

1-0

11

0.5

00.

631

-0.3

84

5808

3003

030

12-

11+

/2-

11

11.

51.

132

-0.8

85

5810

9012

020

23-

22

32.

252

0.25

657

8540

0433

03

33+

33

33.

53

3.13

21.

13D

id n

ot a

ncho

r7

582

32+

2+/3

-2

22.

52.

52.

252

0.25

858

2700

0269

03

33-

3+3

32.

53.

53.

003

0.00

958

4590

0463

03

44

3+4

44

3.5

3.88

4-0

.13

1058

4590

0306

04-

4-2+

3/4

33

2.5

3.5

2.83

3-0

.17

1158

0560

0831

0.4

44+

44

44.

54.

134

0.13

1258

0880

1288

03

33+

3+/4

-3

33.

53.

53.

253.

5-0

.25

1358

2700

0251

0°4

44

44

44

44.

004

0.00

1458

4590

0449

04+

3-3-

4+/5

?3

2.5

2.5

4.5

3.13

30:

13

Dis

crep

ancy

2%

14%

Com

men

ts

6465

Sum

mar

y of

Ari

-or

Pape

r Rev

iew

Gra

de 1

0Se

quen

tial

Num

ber

Pape

r N

.1s

t Rou

nd2n

d R

ound

Ori

g.sc

ore

Dif

fsc

ore

Not

es1

23

45

12

34

5m

ean

158

1090

1457

01

11

11

11

11

10

257

8450

0716

01

11

10.

51

0.87

51

-0.1

33

5810

9014

490

11+

12

11.

51

21.

375

10.

375

457

8540

0711

01

3-2/

2+2

12.

52

21.

875

2-0

.13

558

1090

1452

02

3+2/

2+3-

23.

52

2.5

2.5

20.

56

5786

3002

400

23-

32

2.5

2.5

32.

52

0.5

757

8540

0672

02+

3+4-

3-2.

53.

53.

52.

53

21

858

4590

1099

02

34-

3-2

33.

52.

52.

753

-0.2

59

5845

9010

530

34

3-3+

34

2.5

3.5

3.25

30.

2510

5785

4008

550

32

4-3

32

3.5

32.

875

3-0

.13

1157

8540

0703

03

3+3+

33

3.5

3.5

33.

253

0.25

1258

4590

0885

04

34+

/5-

4-4

34.

53.

53.

754

-0.2

513

5810

9015

450

53

4+/5

-4+

++

53

4.5

4.5

4.25

40.

2514

5785

4006

770

43

54

43

54

44

015

5785

4008

070

3+/4

-3

54-

3.5

35

3.5

3.75

5-1

.25

Not

an

anch

or

Dis

crep

ancy

4%

27%

Com

men

tsI

have

que

stio

ns a

bout

the

read

abili

ty /

legi

bilit

y of

the

scan

ned

copi

es, t

he s

core

s us

ed"o

n lin

e". I

s it

sim

ilar

to w

hat w

e se

e on

the

Scor

e-V

iew

CD

? H

owm

uch

of a

fac

tor

is th

e ab

ility

to r

ead

wha

t the

kid

wro

te in

sco

ring

?E

ligib

le te

xt s

low

s th

e re

ader

'sre

adin

g ra

te a

nd c

an a

ffec

t com

preh

ensi

on.

I fo

und

som

e of

the

copi

es in

my

pack

et h

ard

to r

ead

and

I ha

d to

wor

kex

tra

hard

to m

ake

sure

I u

nder

stoo

dw

hat t

he s

tude

nt w

as s

ayin

g.H

ow im

port

ant i

skn

owle

dge

of th

e re

adin

gpa

ssag

e to

sco

ring

this

for

wri

ting?

One

essa

y in

our

pac

ket d

emon

stra

ted

sim

ple

cont

rol o

f th

e fi

ve p

arag

raph

essa

ys. W

ritin

g,se

nten

ces,

con

vent

ions

and

so

fort

h. I

tw

asn'

t gre

at, b

ut it

was

ade

quat

e.H

owev

er, t

he e

ssay

cou

ld h

ave

been

wri

tten

with

out r

eadi

ng th

epa

ssag

e. T

here

are

no s

peci

fic

exam

ples

fro

m th

e te

xt. W

ait.

I'm n

ot s

ure

wha

t my

ques

tion

ishe

re. I

n th

e lo

wer

-end

resp

sons

es, k

ids

regu

rgita

ted

deta

ils a

nd m

ade

min

imal

or

sim

ple

obse

rvat

ions

. The

high

er e

nd r

espo

nses

incl

uded

refe

renc

es to

the

text

but

the

stre

ngth

of th

eir

wri

ting

was

in th

eir o

bser

vatio

ns,

conc

lusi

ons,

insi

ghts

, "pa

rtic

ular

take

on th

e su

bjec

t' ra

ther

than

in th

e sl

avis

hre

telli

ng o

f de

tails

fro

m th

epa

ssag

e. W

ell,

let's

see

wha

t oth

er p

eopl

tieto

66

Attachment GRecords For Re-scoring

GR

IDD

IST

SE

XR

AC

E R

SC

AL

WR

ITE

R W

PR

OM

P W

TE

XT

12

RE

SC

OR

E3

45

diff

new

relia

137

3 367901

33

M2

406

54

11

11

11

01

5173

3 282450

33 M

5374

54

11

11

11

01

5176

3 366623

33 F

5420

64

22

22

22

02

5206

3 169971

33 F

2482

76

13

33

33

+2

35

339

3 663722

33 F

5464

75

23

33

33

+1

35

340

3 408344

33 F

5499

65

12

32

22

+1

24

375

3 902913

33

M5

442

65

12

22

12

+1

24

399

3 985078

33 F

5359

43

11

11

11

01

5441

3 159275

33 M

4420

86

24

44

44

+2

45

511.

3 549456

33 M

5461

54

12

22

22

+1

25

619

3 200939

33 M

5428

54

11

22

12

01

3675

3 751398

33 F

2436

54!

12

22

22

+1

25

689

3 762567

33 F

5406

75

22

22

22

02

.5

910

3 779588

33

M2

420

76

12

33

33

+2.

34

927

3 106739

33M

5475

75

22

22

22

02

51493

3 439440

33

M5

504

43

1*

1501

3 168315

33 F

5532

96

3*

1509

3 506901

33 F

5464

54

13

23

33

+2

34

1676

3 195108

33

M2

431

75

22

22

32

02

41727

3 216213

33 F

2451

75

23

33

33

+1

35

2006

3 230378

24

M5

414

32

1*

2136

3 217638

23 F

2482

96

31

11

11

-2

15

2234

375317

23

M5

471

54

13

33

33

+2

35

2409

3 188615

32 F

4326

32

12

22

22

+1

25

2455

3 544747

32

M5

425

54

12

11

12

01

32477

3'78226

32

M4

379

54

11

11

10

1.4

2526

349005

32

M2

433

32

12

22

12

+1

23

2543

3 238821

32

M2

362

43

11

11

12

01

42880

3 737112

32 F

2401

54

11.

11

11

01

52947

3 924873

32 F

2379

32

11

11

11

01

52951

3 351243

32 F

5478

75

22

22

22

02

52955

333818

32 F

2448

86

22

22

22

02

53101

3 190639

32 F

5478

76

12

22

22

+1.

25

3190

3 150792

32 F

2451

76

12

22

22

+1

25

3255

3 151192

32M

5396

32

11

11

11

01

53284

3 174177

32 F

5425

32

11

11

11

01

53302

3 527531

32.M

5349

32

11

11

11

01

5

6970

3326

3 806611

32 F

2448

75

22

22

22

02

53463

3 866307

32 M

2401

54

11

11

11

01

53524

3 510073

32 F

4398

54

11

11

10

14

3603

3 724725

32 F

5388

54

11

11

11

01

53875

354689

36 F

2417

65

11

11

11

0_

15

3881

3 262872

36 F

2428

32

12

22

22

+1

25

3899

3 324793

36 F

5431

54

12

22

22

+1

25

4007

3 117443

36 M

2359

54

11

11

11

01

54033

3 321053.

36 F

5457

65

12

22

22

+1

25

4093

3 100674

36 M

5393

32

11

11

11

01

54166

3 377607

36 M

5431

64

22

22

22

02

54209

3 780056

36 M

2464

75

23+

33

33

+1

35

4274

3 445109

36 M

2439

43

12

22

22

+1

25

4555

3 994327

34 M

5433

43

11

11

11

01

54594

3 648103

34 F

5422

42

23

22

2+

30

23

4848

3 882657

34 F

5464

75

23

33

33

+1

35

4854

3 924279

34 F

3464

75

23

33

33

+1

35

4859

3 834471

34 M

2377

54

12

22

22

+1

25

4919

344322

34 M

5448

65

12

22

22

+1

25

4951

3 795610

34 M

5482

32

1.

11

11

20

14

5138

3 213309

34 F

2431

43

11

11

11

01

55468

3 104527

10 M

5404

43

12

11

11

01

55508

3 900592

10 M

5486

75.

23

33

33

+1

35

5628

3 208990

10 M

5486

76

13

33

.3

3+2

35

5634

3 344208

10 F

5439

53

22

22

22

02

55689

389409

10 M

5388

32

11

11

11

01

55763

3 494970

10 M

5393

54

-1

11

11

10

15

5852

3 875699

35 M

5409

76

12

12

12

+1

23

6148

3 385115

18 M

5390

43

12

12

12

+1

23

6162

3. 335241

18 F

2519

96

31

11

11

-2

15

6180

3 631982

18 M

2417

54

13

33

33

+2

3-

56461

316968

15 M

5436

53

22

22

20

24

6532

3 627630

15 F

5359

43'

11

11

10

14

6883

388027

13 F

5393

76

11

11

11

01

56895

3 662130

13 M

5390

54

11

11

11

01

56901

3 652308

13 F

2425

53

22

22

22

02

56984

3 675191

13 F

5471

85

34

33

33

03

47016

3 285088

13 F

2377

54

17022

3 286314

13 F

2420

53

22

22

23

02

4

72

7122

331

189

29 F

548

26

51

1.2

22

2+

12

471

243

1359

0229

M5

442

64

22

22

20

24

7146

3 33

0812

29 F

542

25

41

21

11

20

13

7224

3 38

2099

29 F

533

83

21

11

11

10

15

7295

3 96

8646

29 M

546

75

41

23

33

3+

23

474

373

1725

931

M5

442

75

2*

7481

3 50

6740

31 F

247

19

63

22

22

2-1

25

7553

3 18

5970

31 M

549

07

61

11

11

10

15

7672

3 88

5879

31 M

242

24

31

11

11

10

15

7673

314

769

31 F

238

83

2'1

23

32

3+

23

376

863

1078

8531

F5

454

75

21

12

12

02

377

223

1773

5931

M5

454

76

12

33

33

+2

34

7758

3 32

9557

31 F

346

47

52

11

11

1-1

15

7786

3 91

5274

31 F

544

27

61

21

11

20

13

7827

3 53

0891

31 M

243

69

63

23

2-1

22

7859

333

811

31 M

552

56

42

23

02

179

553

9166

3931

M5

454

43

12

22

2.2

+1

25

7961

3 64

8639

31 M

551

47

52

33

44

+1

32

7982

3 47

2929

31 M

543

36

51

11

11

01

479

903

6237

2331

M2

404

65

11

11

10

14

8062

3 35

7918

31 F

543

98

62

22

22

02

481

073

9573

631

M5

422

54

11

11

10

14

8115

3 74

4756

31 F

240

98

62

22

22

02

483

853

9738

2717

F5

464

75

22

02

1

Num

b of

ca

read

: 10

0N

mbe

r of

cas

es

list

ed: 1

00

74

'73

GR

IDD

IST

SE

X R

AC

E R

SC

AL

WR

ITE

RA

WP

RO

MP

WT

EX

T

12

RE

SC

OR

E3

45

diff

new

relia

8779

575

336

33 F

550

27

52

11

11

1-1

15

8792

582

7470

33 F

541

95

41

11

11

10

15

9156

526

7652

33 F

241

15

41

11

11

10

15

9179

575

485

33 F

246

78

62

22

23

20

24

9488

549

9368

33 F

249

88

62

22

11

2-0

23

9504

564

9423

33 M

551

37

43

33

34

30

34

9528

540

4329

33 M

547

06

51

22

22

21

25

9619

537

3309

33 M

547

36

51

22

21

21

24

9634

586

1363

33 M

243

67

61

2.5

32

33-

23

396

435

9422

7433

M5

434

54

11

12

10

13

9652

520

6655

33 M

546

57

52

22

23

20

24

9771

599

5356

33 M

247

310

73

00

00

03

498

225

2881

4133

F5

495

64

22

22

12

02

499

225

7417

3933

F4

444

86

21+

11

01

399

715

2902

5233

F5

495

54

'2

21

21

23

1004

75

7088

3233

M40

14

31

11

11

01

410

096

541

6506

33 F

548

85

41

10

0-1

02

1012

65

3405

4233

F2

444

86

22

23

20

23

1016

25

6849

2533

F5

556

117

43

33

3-1

34

1039

35

4912

8024

F5

485

76

12

11

01

210

421

554

2476

24 M

549

17

52

22

12+

02

310

422

586

6434

24 F

551

85

41

22

21

23

1046

75

9251

8824

F5

505

107

33

34

03

210

483

563

3385

24 M

344

16

51

2-1

10

12

1050

65

7871

7124

M5

479

86

22

22

02

310

671

519

3079

23 M

247

06

51

22

21

23

1075

85

1749

9723

M5

424

54

11

11

10

14

1081

45

1937

5823M.

550

58

62

.22

12

02

311

025

511

8792

32 M

554

97

52

33+

13

13

311

118

537

9913

32 F

242

94

31

2-1

31

01

211

239

563

303

32 M

547

65

41

2-1

11

01

311

314

520

4606

32 M

252

27

61

34

33

3-2

34

1150

95

4063

3032

M5

429

54

11

11

10

14

1156

25

2018

5032

M5

467

86

22

21

2+0

23

1158

35

2571

0132

F2

403

65

11

21

20

12

1159

75

7398

6732

F5

467

54

12

11

2-0

12

1171

65

2105

0132

M5

502

86

22

33

31

33

767.

5

11788

5177380

32 F

5527

85

33

4-

33

03

311876

555245

32 M

2406

54

11

11

'2

10

14

11923

5259036

32 F

4498

86

23

2+

30.

22

11999

5755348

32 M

5482

32

11+

11

1+

01

412119

5805167

36 F

5491

76

12

-

21

21

23

12226

5795689

36.M

5414

32

11

11

.1

01

412348

5862536

36 M

5462

32

1.

11

11

01

412417

5700730

36 F

5522

86

22+

32

'3

02

212594

5828385

36 F

5518

86

23+

43

4-

13

212726

543250

34 F

2495

75

23+

33

31

34

12824

5142480

34 F

2389

65

11

11

10

14

12845

5797280

34.M.

2446

54

12

21

2-

12

312952

5572284

34 F

2436

86

2,

22

12

02

312966

510637

34 M

2462

85

32

21

2-1

23

13027

5888974

34 M

5502

64

22

22

20

24

13113

5231993

34 F

4414

65

11

11

10

14

13320

5176890

34 F.

5470

86

22

22

02

13358

5384747

34 M

5462

43

12.

22

12

13444

5560812

34 F.

2485

53

22

21

02

213450

5196784

34 F

5479

75

22

32

02

213723

5360070

10 M

3465

54

11+

11

01

313825

5974179

10 M

2409

32

11

11

01

313837

5487971

10 F

5518

86

23

32

13

213881

5654538

10 M

5488

65

11

.1

10

1'

313950

5471218

10 M

2457

54

11

11

01

314154

5177031

35 M

2462

54

12

22

12

314201

5339384

35 F

4473

75

2*

14262

5206058

18 F

5502

9.6

33

3-

30

33

14338

5394600

18 M

5427

75

21

11

-1

13

14551

5317717

16 F

5451

43

12

1+

2-

12

214713

5825086

15 F

5459

64

22

23-

02

214732

5115215

15 M

5527

97

23

34

13

214737

5220091

1.5 M

5439

64

22

22

02

314843

595574

.15 M

5488

75

21

22-

12

214896

5230189

15 M

5476

54

12

12-

12

214904

5768636

15 M

2467

86.

22

32

02

214954

5120990

13 M

2377

32

11

01

114985

5501914

13 M

2446

65

1*

15119

5501759

13 M

.5

446

32

11

10

12

7778

1523

537

5826

13 F

549

87

52

22

1+ 2

-0

23

1524

15

8953

7613

F5

513

75

22

23-

02

215

331

530

1588

.13

M2

476

43

11

11

01

315

357

535

0104

13 M

552

710

73

33

32

30

34

1545

55

5748

829

M3

470

75

22

21

12

02

315

626

512

5143

29 M

546

55

41

11

11

10

15

1568

25

7830

029

F5

454

65

11

1+0

01

01

315

784

520

5098

29 F

546

75

41

22

11

10

13

1595

85

2419

2631

F5

454

64

22

22

12-

02

415

983

570

6099

31 M

244

16

51

11

11

10

15

1600

25

4973

1731

M5

457

54

12

22

12

12

416

004

596

4751

31 M

449

58

62

44

44

42

45

1608

55

4862

3331

F5

556

96

34

34

44

14

416

095

516

7048

31 F

554

310

73

55

44

52

53

1613

25

7355

4831

M5

451

75

22+

22

23-

02

416

155

594

8607

31 F

244

48

62

2+3

22

02

216

265

514

7686

31 F

543

96

42

33

33

13

416

339

534

3527

31 F

248

89

63

33

4-,

03

216

395

549

8619

31 F

554

310

73

33-

30

33

1650

85

5059

0531

F2

482

65

11

11+

01

316

523

522

2319

31 M

554

99

63

2+2

31

22

1663

05

5953

1531

F5

543

86

23

23-

13

216

940

577

2364

17 M

553

28

62

22

3-0

22

1701

75

6801

3017

M5

479

54

12

23+

12

2

Num

ber

of c

a re

ad: 1

0 0

Nu

mbe

r of

cas

es

list

ed: ,

100

80

79

GR

IDD

IST

SE

XR

AC

ER

SC

A W

RIT

ER

A W

PR

OM

P W

TE

XT

12

RESCORE

34

5dill

new

relia

17077

8937445

33

M5

532

87

12

12

21

23

17226

8214817

33 F

5532

75

22

12

30

22

17346

8884438

33 F

5556

85

33

43

03

217582

8924867

33

M5

513

86

2*

17844

8673683

33

M4

490

86

22

23

20

23

17996

8329707

33 F

4552

85

33

43

03

218009

8940992

33

M5

502

76

11

11

01

318056

8924818

33

M5

532

75

23

34

13

218060

8493567

33 F

5524

10

73

334

40

32

18067

8192943

33

M2

492

65

1*

18211

8916864

33 F

4521

65

12

12

12

218377

8307335

33 F

5535

86

22

12

20

23

18416

8752786

33 F

5513

76

12

12

21

23

18494

8690583

33 F

5487

86

22

11

1-1

13

18725

8593665

24

M2

510

76

13

22

21

23

18827

8306480

24 F

5515

86

22

33

20

22

19030

8860137

23

M5

548

75

22

33

13

219036

8231801

23 F

2480

86

22

32

02

219098

8649734

23

M2

532

97

22

32

02

219179

8193408

32 F

2468

86

22

2+3-

30

22

19182

8431133

32M

5444

76

1*

19239

8639143

32 F

2542

54

11

11

01

319288

8758424

32 M

5450

65

11

10

01

219863

8312403

32 F

5500

54

10

01

120147

8593838

32 F

5497

65

11

11

01

320148

8830675

32 M

5480

76

12

22

12

320153

8699218

32

M5

524

86

22

22

02

320162

8403732

32 F

5505

97

22

20

22

20380

8740581

32 M

5529

64

212

20

22

20528

8942121

36 M

5532

75

22

22

02

320553

8677223

36 M

4468

64

22

3-

31

32

20581

8672452

36 M

5552

86

24

44

24

320627

8456895

36 F

5545

10

73

44

41

43

20703

.8

44452

36 F

2577

9.6

33

30

32

20757

8721301

36

M5

458

64

22

00

21

20759

8771125

36

M5

545

86

23

31

32

20794

8686203

36 F

5510

86

22

20

22

8182

21010

8234133

36 F

5542

97

22

20

22

21066

8596917

97 M

5500

'7

61

1+2

10

12

21119

8269698

34 M

5515

87

112

21

22

21130

8951273

34 F

2483

54

123

33

33

23

521132

8104065

34 F

2441

43

12

22

32

12

421212

8108503

34 M

5470

54

11

11

11

01

521215

8261052

34 M

5502

86

21

11

11

-1

15

21285

8987757

34 M

5615

96

32

21.

22

-1

24

21352

850671

34 F

5526

85

34

4+

44

41

45

21426

8350461

34 M

5535

7I

52

22

02

02

02

421614

8110646

34 F

5542

86

22

22

30

23

21625

8676398

34 F

2538

10

73

33

43

03

321760

8473838

34 F

5521

76

13

33

32

34

21780

8784723

34 F

5529

76

12

22

12

321823

8416918

34 M

2548

8.

62

23

20

22

21846

8143951

34 F

5444

54

11

001

01

222130

8151451

10 F

5568

86

22

30

21

22461

8276423

10 F

2510

96

3*

22476

8279257

10 M

5473

75

2*

22488

8183440

10 F

5529

87

1*

22598

8855969

35 F

2485

76

11

21

01

222604

8922297

35 F

4535

64

22

33

13

222620

8815644

35 F

5502

65

12

2.

21

23

22694

8175531

71 M

5473

43

12

22

12

322808

8317809

18 F

5485'

64

21

11

-1

13

22855

863650

18 M

5518

54

13

33

23

322869

8226118

18 M

5.545

86

22

22

02

323054

869546

18 F

5572

10

73

44

41

43

23197

8773717

16 F

5526

75

22

33

13

223369

8480151

15 M

3548

97

22+3-

33

31

33

23442

8733068

15 F

5510

86

22+3-

23

2+

0'2

323481

8105320

15 F

2500

86

22

12

2+

02

323596

8536530

37 F

5500

86

22

12

20

23

23640

8512019

37 M

5538

76

120

12

21

23

23771

8393311

13 M

2524

75

22

12

20

323782

8336343

13 F

2526

86

23

23

31

33

23854

8450678

13 M

2532

54

11

22

21

23

23965

8790435

13 M

5450

43

12-

22

21

24

24173

.8

557401

29 F

5577

97

22+

33

31

33

83

2431

48

8874

1529

M5

532

86

22

22

20

24

2442

28

1170

5829

M5

505

65

12

12

21

2.

324

487

812

9208

29 M

550

26

51

212

33

23

224

557

836

489

31'F

244

47

52

0 3

22

02

224

578

888

3430

31 M

550

58

62

22

20

22

2461

18

1446

6531

M5

564

107

34

41

42

2462

48

2649

3631

M5

529

64

22

33

13

224

662

821

5581

31 M

557

79

63

33

30

32

2470

68

9522

3231

F5

524

96

32

4 5

14

124

723

885

2922

31 F

255

910

73

34

3 4

03

224

935

846

8679

31 F

'3

548

74

32

22

-12

325

076

.845

6464

31 F

548

05

41

24

3 4

34-

225

317

873

1942

31 M

559

37

52

23

41

31

2533

88

7481

6431

M5

577

86

22

33

1-

32

2546

38

2966

2431

F5

532

-97

22

33

13

225

573

857

7938

17 M

555

27

61

33

2 3

23

325

582

824

983

17 M

552

48

62

34

3+1

32

2558

98

4759

8917

M5

529

76

11

12

20

12

2564

08.

6113

517

M5

505

-76

11

22.

21

23

2565

58

7972

2317

F.5

485

54

11

12

20

12

2570

78

9312

2817

M5

545

86

22

45

4 5

24

225

781

894

2209

17 F

551

37

52

24

22

02

325

805

823

4851

17 F

555

29

72

23

33

.13

325

871

843

7136

17 F

252

48

62

44

33

13

2

Num

ber

of c

a re

ad: 1

0 0

N m

ber

of c

as e

s lis

t ed:

100

8685

GR

IDD

IST

SE

XR

AC

E R

SC

ALE

WR

ITE

RA

WP

RO

MP

WT

EX

T

12

RE

SC

OR

E3.

45

6D

IFF

new

sco

rre

lia

2601

510

419

991

38 F

252

78

62

22

20

23

2603

910

421

647

38 F

250

96

51

22

21

23

2610

510

652

857

38 F

551

26

42

22

20

23

2613

410

425

833

38 M

551

29

72

33

31

33

2621

010

796

376

38 F

553

78

62

33

31

33

2622

710

971

450

38 F

554

47

52

22

20

23

2634

410

964

948

38 F

547

97

52

22

20

23

2653

810

211

913

38 F

547

67

52

22

20

23

2656

710

500

437

38 F

552

48

62

33

31

33

2661

310

350

650

38 M

550

38

62

22

2.0

23

2665

210

908

659

38 F

250

91.

07

33

40

31

2680

710

425

261

33 F

556

39

63

34

03

126

831

10 8

4129

733

M5

540

107

33

30

32

2687

310

773

780

33 F

553

77

43

33

33

0.

35

2696

810

892

106

33 F

560

011

74

34

44

40

44

2698

910

779

708

33 M

553

78

62

22

22

20

25

2710

510

1894

833

F5

521

96

32

22

32

-12

4,2

7108

1019

499

33 F

555

99

63

33

33

30

35

2714

910

923

401

33 M

560

78

62

34

43

42

43

2721

210

877

800

33 M

248

75

41

11

12

10

14

2732

910

912

641

33 F

550

68

62

11

11

1-1

15

2734

810

629

522

33 F

548

15

41

11

11

10

15

2743

410

674

010

33 M

248

15

41

11

12

10

14

2743

510

892

33 F

553

75

41

22

22

21

25

2749

510

228

356

33 F

555

16

42

22

22

02

427

521

10 5

1986

033

M5

479.

.6

51

00

01

00

14

2760

910

232

470

33 M

551

89

7-2

33

32

31

34

2766

210

216

344

33 F

550

34

22

23

33

31'

34

2770

2.1

0 44

6515

33 M

542

85

41

00

00

00

15

2782

310

364

707

33 F

541

53

21

10

01

0-1

03

2784

010

7747

033

F3

521

75

22

33

33

13

427

914

10 6

6386

140

M5

540

86

23

33

23

1.

34

2801

710

263

219

40 M

549

57

61

22

22

12

428

159

10 4

2600

04.

0 M

244

97

61

11

11

10

15

2835

710

782

281

24 M

553

05

41

12

22

21

24

2859

910

491

714

23 M

555

18

62

2875

310

796

778

32 M

547

3.5

41

00

00

3

8788

GR

ID

DIST SEX

RACE RSCALE WRITERA WPROMP WTEXT

12

RESCORE

34

56

DIFF

new scor

relia

28782

10 559814

32 F

5537

65

12

22

21

24

28806

10 960590

32 M

4441

43

11

11

10

14

28819

10 881461

32 M

5600

86

23

44

42

43

28838

10 352215

32 M

5500

54

12

12

21

23

29096

10

34839

32 M

5455

54

10

00

0-1

04

29129

10 953257

32 F

5572

11

83

22

22

-1

24

29346

10

595

32 F

5518

96

32

21

2-1

23

29618

'10 661766

39 M

5495

65

11

11

20

13

29627

10 376334

39 M

4506

86

22

22

30

23

29653

10 982616

39M

5455

76

12

22

21

24

29800

10 389383

36 F

5582

10

64

43

44

04

329810

10 152017

36 M

5503

85

32

33

30

33

29854

10 763885

39 M

2498

54

1*

29965

10 760851

36 F

1582

10

73

44

44

14

429995

10 870781

36 F

5487

64

22

22

20

24

30091

10 913782

36 F

5572

98

13

32

32

33

30112

10 580310

36 F

5537

86

22

10

22

30186

10 898181

36 F

2498

86

22

10

22

30502

10 402309

34 F

5567

96

32

2-1

23

30582

10 878046

34 M

2492

86

21

11

-1

13

30834

10 925911

34 F

5521

86

22

20

22

30853

10

103251

34 F

2537

96

33

30

32

30904

10

187357

10 F

5489

76

1*

30929

10 990088

10 M

2500

86

23

31

32

30991

10 294931

10 F

4521

86

21

1-1

12

31134

10 615078

10 M

2463

32

11

10

12

31191

10 610598

10 M

5515

64

2*

31193

10 660459

10 M

5600

98

11

10

12

31326

10 206564

35 M

5563

96

32

2-1

22

31358

10 293609

35 F

5518

64

23

33

31

34

31617

10 376403

18 F

4481

54

13

33

33

23

531652

10

23777

16 M

5438

65

13

33

42

33

31721

10

92297

16 F

2457

54

12

12

20

13

31756

10 934124

16 F

2487

75

22

12

20

23

31869

10

107382

15 M

5527

54

11

12

10

13

31881

10

135887

15 F

5487

65

11

11

01

331882

10 761319

15 M

5479

75

22

12

2-1

13

9089

GR

IDD

IST

SE

XR

AC

E R

SC

ALE

WR

ITE

RA

WP

RO

MP

WT

EX

T1

2

RE

SC

OR

E3

45

6

DIF

Fnew scor

relia

31961

10

109773

37 M

5489

75

22

12

20

23

31993

10 673931

37

M5

509

65

12

12

32

12

332096

10 709947

70M

5555

86

23

33

41

33

32115

10 298406

70

M3

582

12

93

32

22

3-1

23

32117

10

4537

70 F

2521

96

3*

32133

10 329583

70 F

2527

96

33

23

30

33

32141

10 500167

70M

5509

97

2*

32238

10

106376

70

M5

537

86

23

13

132280

10 322119

13

M5

452

75

22

02

132295

10 295461

13 F

5527

64

22

02

132412

10

153750

13

M5

555

11

83

30

31

32464

10 840910

13

M5

492

64

232814

10 616254

29 F

5547

75

23

13

132825

10 639304

29 F

2463

43

11

01

132861

10 981248

29 F

5563

96

33

03

133062

10 600358

31

M5

524

54

12

12

133188

10 214833

31 F

5551

86

23

13

133334

10 409446

31

M5

484

75

22

02

133383

10 246632

31

M5

495

96

32

-12

133388

10

148398

31 F

5506

11

74

3-1

31

33446

10

110457

31 F

2498

75

23

1.3

133500

10 597104

31 F

2518

86

23

13

133552

10 386053

31 F

2572

10

73

41

41

33817

10 631370

17 F

5498

54

13

23

133863

10 120670

17 F

5527

75

23

13

133896

10 296373

17

M2

551

86

23

13

1

Num

ber

f cas

rea

d: 1

0 0

N m

ber

of c

as e

s lis

ted

: 100

92

91

Attachment HWriting Task Evaluation

94

Wri

ting

Tas

k E

vaiv

atio

n17

-fri\ug

-00

Gra

de 3

2000

Rev

iew

erte

Car

ter

Whi

tman

Tra

iner

Sho

ckle

y

4V

ery

Har

d

Har

d

Not

Har

dE

asy

Ver

y E

asy

Per

form

ance

50.

000.

020.

010.

004

0.15

0.13

0.05

0.10

30.

300.

250.

300.

232

0.25

0.40

0.44

0.42

1=

0.30

0.20

0.20

25.0

0

CO

PY A

VPL

ia'f:

-.

95

Writ

ing

Tas

k E

vaiv

atio

n17

- ug

-00

Gra

de 5

9920

00R

evie

wer

,-..S

tike:

tS-:

4P

oehl

man

nH

odgk

ins

u(ei

e::

She

ets

Poe

hlm

ann

Hod

gkin

sC

urrie

Lam

Ver

y H

ard

..,

Har

d4.

..:

-4

Not

Har

d

Eas

y

Ver

y E

asy

fl.--

--.,t

.

Per

form

ance

,.--

...-.

1.,

50

'5

,,_0.

000.

010.

020.

000.

024

.,-.

.-

.;=:..

0':2

0:`.

"0.

050.

050.

100.

050.

183

,.,

0;4.

00

.0.

200.

300.

300.

200.

202

,_0

,z

0.35

0.32

0.38

0.50

0.40

102

0 t r

,";:,

,,..

',..

0.40

0.32

0.20

0.25

0.20

9796

. ruin

g T

ask

Eva

lluat

ion

17-

ug-0

0G

rade

8

2000

Rev

iew

er,1

010-

ier-

)=

Sze

gda

g,..

=,

oort

ifaii

-An

gat;

Spe

iche

rS

zegd

aA

shP

oorm

anA

ngst

adt

.-;

Ver

y H

ard

;.'

and

..

44

44

Not

Har

d.t.

F'

Eas

y-

Ver

y E

asy

,:.:

i.,,

,1

Per

form

ance

''.

5.

0.01

0.04

0.01

0.03

0.02

0.10

0.16

0.12

0.07

0.07

3'

,,P.-

::!::1

:P''''

,.:"

..V::

0;'3

0 F

:-,

.;1'

0.60

0.50

0.40

0.40

0.20

,..,

.:-

:

0.19

0.20

0.20

0.40

0.21

"'

0.10

0.10

0.11

0.10

0.05

.06

off t

opic

9899

Wri

ting

Tas

k E

vaka

atio

n11

7-A

ug-0

0G

rade

10

999

...20

00R

evie

wer

pbsi

v_17

:iG

rudz

ina

,aN

iti4

sto.

KI,

,,4V

Yll§

p`ri-

-,B

oyd

Gru

dzin

aLa

veric

kS

mith

Wils

on, .

,:,.

...

Ver

y H

ard

,:

-\I

4H

ard

-.=

:,

,-,

4N

ot H

ard

,

'',

Eas

y:.

,.. '

"-..-

-:

4V

ery

Eas

y_

--'"

1-

,

Per

form

ance

5.

0.02

0.07

0.00

0.00

0.00

402

0:x'

':-Z

::,'.

..,

0.15

0.25

0.05

0.01

0.05

3I

O.

.'=

,,

0.33

0.50

0.40

0.34

0.25

2:

-,

4,'

0.30

0.10

0.30

0.30

0.40

10

21:'

-'.

0.20

0.08

0.25

0.35

0.30

'fc

,ti4,

6tit

fi

100

101

Writing Task EvaluationGrade 3

Comments:

Confusing for students. It arises that tigers don't live in forests with foxes. Thewording for the 1999 prompt explained it better than in the 2000 prompt did for 3rdgraders to understand. The 1999 story was more interesting. Children can relate tothose pets.

The question doesn't give enough stem to make students relate to the nature of thefolk tale.

I think it would be very hard for a child to achieve a 4 or 5 on this task because ifthey respond to the prompt and the reading, it's basically a retelling. There is littleopportunity to demonstrate any writing "flair" they may have which can push a paperto a 4 or 5.

102


Comments

Felt the students needed an organizer.

The 2000 test is harder.

103


Comments

Students have some difficulty with "heritage". The question implies that theauthor and narrator are the same person and they are not.

I'm not sure that this means anything to me, really. I felt like I was answeringwithout a lot to go on. My low estimate is based on confusion based on theprompt. It assumes/implies that the author and the narrator are the same when weteach them. That is not the case. It forces the reader/writer to shift betweenviewpoint and voice, and that kind of stuff that can confuse issues.

Comments


I think that the Cinderella question was not very engaging to the malepopulation.

This is most likely an overly biased or subjective evaluation. I'm not sure Ihave a good idea of an "average" student.

I feel that the 2000 prompt uses language ("broad appeal") that is confusingto the students. Many did not seem to understand that broad appeal meant"popular" which influenced their answer.

The word "choice" in the question (2000 essay) make it confusing for thegeneral student population. I would re-word "broad appeal". The 1999question had more familiar wording and topic choice. The selection was moreinteresting to students. The 2000 selection was more literary, and students ingrade 10 received less instruction in literature.

Perhaps the phrase "broad appeal" confuses some kids who cannotunderstand what it meant.

The format of the ? {Last on page, no bold in directions, no drafting) leadsitself to the difficulty level.

2000 I think that the Cinderella question was not very engaging to themale population.

Attachment IWriting Study Group Discussion

106

Writing Study Group DiscussionGrade 3

Text-Based Writing Development

Look at choice of text. There is a big difference between 1999 and 2000. It is much easierfor kids to connect to a person rather than a monkey. Prefer to see realistic stories usedwhen kids have to be a Text-based writing. Authentic audience.

The question in 2000 did not say use details and information for the text. The 1999 did notsay that verb tense (present) may have confused them. The question doesn't give enoughstem to make students relate to the nature of the folk tale.

Basically a retelling, so it is difficult for a child to achieve a 4 or 5. The question askedwhat do you see, what do you think and lots of kids responded, I see, I see, I see or I think.Confusion for students arises that tigers don't live in forests with foxes.

Text-Based Writing Score

We found a big discrepancy between Harcourt's and ours. They seem to have a biggerrange for a 1. Check the discrepancies between our "3's" and their same pieces whichwere scored "1".

We also had a problem with the invalids "the big rock" scored a 1, should it be a 0?

Teachers should be given a breakdown of "the score". This code can be done by codes.Not enough details for the story, etc...

Text-based scoring rubric for writing be developed. How much emphasis is incorporatedin the grading for using the text?

Test Administration

Certainly not 2 Text-based writing's on one day. Not authentic. Use a realistic story, not afantasy or folk tale. Word the prompt more carefully scaffold. Set up like the prompt isto emphasize the importance.

Test Development

107


DSTP Manual 2000. Page 17

The Directions include both questions 57/67 to take a 10 minute rest. Included "Dear Mrs.Cabot"



Text-based writing added to get a more valid score for writing. 1 page versus 2 pagesformatting. Change in short answer.

Navaho selection is not as engaging. Don't have background experience. Should be ableto get enough from text. (But familiarity helps connect to text.)

Do better with nature. Students connect. Both informative pieces (Science and SocialStudies).


Text-based writing used writing rubric. Need anchor papers. Is our scoring as goodwithout anchors as theirs is?

Test Administration

Instructions: Developmentally need to be aware of problems, of directions, understandgraded for reading and writing. Amount of instruction needed to listen and to understand.Not doing 2 text-based writing's on the same day. Consider doing text-based writingpassage first of a set. Prewriting area.

Classroom Instruction

Develop text-based responses to go home with class take home books. Content area doalso. Model text-base responses show, have to make connections and pull out details.

Make connections with characters in the story.

New reading people to look at district curriculum to find samples of things to work on.

Grade levels / districts developing questioning activities to improve text-based writing.Sample papers for text-based writing.


Test Development

DSTP Test 2000 Student Book 22 Response booklet 12-13 Use 2 full pages. 1999only 1 page.

Grade 5 directions read much better than grade 3. Especially need to look at breaks ingrade 3.

Writing Study Group Discussion.Grade 8


350461 Issues: Is it on topic? Should it be scored?143951 Topic issues. Is this scorable?11306480 Tuff to decide between 2+/3.193408 2/3 - --

758424 Off topic?721301 Is this off topic?

What is the purpose of text-based writing, is it to test for ability to write an audiencepurpose? "It didn't count".

1. When is a piece off topic? {and/or invalid). Is a summary that doesn't relate back tothe question scorable? When does 1999 tell us?2. How did media information one week prior to test impact scores?3. Stand-alone started in 90 text-based writing... 98 not as much experience.4. Author/narrator: Confusion or attention to author craft?5. Did kids understand the concept of heritage?


Does a narrative text lead itself to a persuasive prompt?



1999 Text-based writing prompt is more opinion based and more authentic.2000 prompt has less room for creativity. (Harcourt wrote it).

More like a SOAP prompt. (Subject 0 and purpose)

Use "user accessible" language

Doug says there should be more bone oriented Civil War passages.


Relation to off topic. Should the use of the word "Cinderella" earn credit? (Re-examinethe "invalids"). Insufficient information.

0-1 distinction (May be something that changes per prompt).

Test Administration

Prompt set up alone. Not the last to do in the day. Bold it . Emphasize it's importance.Separate page. Bolded for emphasize. Physically make it look like the stand aloneprompt.

For Special Ed. In unclear directions, can there be another version (allowablemodification) of the directions?

Classroom Instruction

Research and Instruction When appropriate to paraphrase/quote from a source.

T have gone away from literary analysis due to emphasis on stand-alone prompts.

Variety of selections in the classrooms.

Block schedule issues (5 weeks till DSTP)

1.1.2


Text-based writing is a form of persuasive, expressive, or informative writing not aseparate type.

T's need more practice in developing prompts, etc. for text-based writing.

11.3

U.S. Department of EducationOffice of Educational Research and Improvement (OERI)

National Library of Education (NLE)

Educational Resources Information Center (ERIC)

NOTICE

Reproduction Basis

TM033074

ERIC

This document is covered by a signed "Reproduction Release(Blanket)" form (on file within the ERIC system), encompassing allor classes of documents from its source organization and, therefore,does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission toreproduce, or is otherwise in the public domain and, therefore, maybe reproduced by ERIC without a signed Reproduction Release form(either "Specific Document" or "Blanket").

EFF-089 (3/2000)

Documents

Reproductions supplied by EDRS are the best that can be ...DOCUMENT RESUME. Zhang, Liru Delaware Student Testing Program: Report. Study. Delaware State Dept. of Education, Dover. Accountability