12/1/2014 Page 1

Constructed Response Writing (Type Two) as a Critical School Improvement Strategy

by John Collins, EdD

Introduction

For more than 30 years, Collins Education Associates has been providing professional development in writing across the curriculum. While we have refined our techniques and strategies over the years, the foundation of our approach, the Five Types of Writing, briefly described below, has not changed.

Type One is a structured process to capture ideas. Type Two is concerned with writing about the correct content. Type Three focuses on writing about content and attending to some aspects of writing craft. Type Four is peer editing Type Three assignments. Type Five is error-free writing.1

We refer to Type One and Type Two as writing to learn and Type Three, Type Four, and Type Five as learning to write. Until recently, when we worked in schools, we explained the power of the Five Types of Writing and encouraged teachers to use the Types at their discretion to suit the needs of their students. But, based on the examination of the Common Core State Standards (CCSS), we have refined our approach. In light of these standards and the tests that will be used to assess them, we are making three specific recommendations:

1. Give frequent Type Two constructed response quizzes 2. Assign Ten Percent Summaries of complex nonfiction text 3. Require students to write multi-paragraph argument essays based on source materials.

This document provides an in-depth explanation of our first recommendation, give frequent Type Two constructed response quizzes.

Emphasizing Type Two quizzes has many payoffs: It capitalizes on the research of the testing effect, provides reasonable and effective preparation for state and national tests, and can be used to formatively assess students’ progress.

The Testing Effect

As early as the 1900s, research revealed the power of writing to help students retain new knowledge. This phenomenon has become known as the testing effect which I believe is a misnomer. It should be

1 For a one-page description of the Five Types of Writing, see Appendix A, and for a complete description, see The

Collins Writing Program: Improving Student Performance Through Writing and Thinking Across the Curriculum by John Collins.

Why the Emphasis on Type Two Constructed Response Writing

12/1/2014 Page 2

called the quizzing effect because the research suggests that short, constructed response (not multiple-choice or true-and-false) questions produce long-term learning through the process of effortful retrieval. Searching our memory for the answer actually helps us remember the answer longer. Over the last 10 years, numerous studies have confirmed the original research: Open response or constructed response writing works a kind of magic. As David Gooblar (2014) states:

The mental processes involved in taking a test are powerful ones, able to help students retain memories of class material better even than studying that material. There is something in the act of taking a test, of being forced to try to recall what we know of a subject [effortful retrieval], that works very effectively on our brains. ... The testing effect is independent of other seemingly important factors: It persists even when students do poorly on the test, and works even if teachers don’t give feedback on the missed material.

In addition, quizzes take advantage of one of the oldest laws in cognitive psychology, the Yerkes- Dodson Law that states students perform better when there is some anxiety: not too much but not too little. Appendix B provides an annotated bibliography of research on the testing effect that I strongly urge you to read. The research is very persuasive.

Preparation for State and National Tests

In addition to the testing effect, frequent Type Two writing has a second benefit: It helps prepare students for the demands of the Common Core and other state standards because the prompts written as recommended below mimic the types of questions students will see on the tests. As you will see, there is a strong emphasis on the use of general academic vocabulary in the wording of the prompt so students become familiar with the academic language that is stressed in the CCSS.

Formative Assessment

In addition to using the power of the testing effect and helping to prepare students for high stakes tests, Type Two prompts take advantage of the research on the impact of formative assessment. Although there are a number of different definitions of formative assessment, one of my favorites comes from Laura Greenstein’s (2010) What Teachers Really Need to Know About Formative Assessment. For Greenstein, formative assessment:

Emphasizes learning outcomes Makes goals and standards transparent to students Provides clear assessment criteria Closes the gap between what students know and desired outcomes Provides feedback that is comprehensive, actionable, and relevant Provides valuable diagnostic information by generating informative data (p. 19)

Type Two prompts created using the six criteria listed below will provide the benefits Greenstein lists. In addition, the quizzes will boost student achievement. Dylan Wiliam (2011), one of the leading researchers on the effects of formative assessment, comments that:

after reviewing over 600 potentially relevant studies, of which 250 were directly relevant,

12/1/2014 Page 3

... we concluded that research suggested that attention to the use of assessment to inform instruction, particularly at the classroom level, in many cases effectively doubled the speed of student learning. (p. 36)

Some approaches to formative assessment downplay grading “Once the grade is assigned, the learning stops,” emphasizing that it is assessment for learning rather than assessment of learning. The approach presented here emphasizes both types of assessment. It has been my experience, mostly at the middle school, high school, and college teaching levels, that if students do not think the answers will be graded, they will not study as hard or try as hard (the effort in effortful retrieval). And, as you will see, part of the strategy is to recycle prompts to promote long term retention. The learning does not stop once the grade is received if students know they may see the question again as a repeated Type Two prompt or as a question on a summative test. This strategy recommends frequent low stakes quizzes. The art of this approach is finding the motivational sweet spot: How do we make the quiz count enough to change behavior (turn off video games, and review notes or read assignment) and not so much that the quiz produces high anxiety? My guess is the percentage of the total grade should be somewhere around 10% to 20%, but that could change with each class, grade level, and school’s culture. To refine the approach without adding too many complications, teachers may want to consider eliminating the lowest quiz grade. Below are six concrete criteria for prompts to help students gain the maximum benefit of the testing effect, new standards, and formative assessment. The rest of this document will explain these criteria.2 Appendix F provides ten examples of Type Two prompts that meet the six criteria.

1. Includes a number or range of numbers in the prompt to make the response quantifiable 2. Uses general academic vocabulary in the prompt 3. Requires the student to use higher-order thinking skills, whenever possible 4. Can be completed quickly: within 2–8 minutes 5. Can be graded quickly: within 10–20 seconds 6. Is graded for content only, not for conventions

1. Includes a number or range of numbers in the prompt to make the response quantifiable Vague questions encourage vague answers, and vague answers are difficult and time consuming to evaluate. Because Type Two assignments include a number in the prompt, a well-constructed Type Two prompt tells the student exactly what to do to be successful; the number becomes the rubric. For example, asking students to list in their own words three central ideas from an article is better than asking them to summarize the article because the task is clearer. Asking for three causes of World War II is better than asking for the causes. An even better approach is to determine a minimum number that you find acceptable and add a higher option. In the World War II example, the prompt might change from three causes to three to five causes. When answering this question, students have to effortfully retrieve the causes. If more than three

2 This document focuses on how to create Type Two prompts. My colleague, Joan Pokrant, has written an excellent

book, Open Response Writing: 18 Strategies to Improve Student Performance that focuses on strategies to help teachers and students create well-constructed answers leading to more content knowledge and deeper understanding.

Writing Type Two Prompts

12/1/2014 Page 4

causes come to mind, have students write them down. A note of caution: Tell students you will grade the responses in order so if one is incorrect, it will be marked down, not skipped. You want to discourage students from writing wild guesses that they hope are correct.

2. Uses general academic vocabulary in the prompt CCSS place a special emphasis on general academic vocabulary, that is, words that are “far more likely to appear in written texts than in speech” (CCSS, 2010, p. 33). General academic vocabulary words

are not unique to a particular discipline and as a result are not the clear responsibility of a particular content area teacher. What is more, many Tier Two words [general academic vocabulary] are far less well defined by contextual clues in the texts in which they appear. (CCSS, 2010, p. 33)

The Common Core argues that one of the main reasons students are not career or college ready is that they do not understand general academic vocabulary words like textual evidence, cite, analyze, and relevant. The new generation of tests designed by the national testing groups Partnership for Assessment of Readiness for College and Careers (PARCC) and Smarter Balanced Assessment Consortium (SBAC) feature questions that are loaded with general academic vocabulary, making the prompts more challenging to read and understand than the passages. For example, a model seventh-grade question from PARCC (2014) asks:

Based on the information in the text “Biography of Amelia Earhart,” write an essay that summarizes and explains the challenges Earhart faced throughout her life. Remember to use textual evidence to support your ideas (Sample Item, Student Directions).

SBAC lists general academic vocabulary (Tier Two words) likely to be in the test. Some grade 3–5 words include exactly, explain, declare, establish, construct, and interpret. Grade 6–8 words include assume, distinct, evident, complicate, and exaggerate. Grade 9–11 words include arbitrary, specify, formulate, benevolent, malign, empirical, and entity. How often do you include words at this level of difficulty in your prompts?

Giving students Type Two assignments that include general academic vocabulary will help them master scholarly diction and be better prepared to understand and respond to test prompts. For example, analyze is not a synonym for write about. Analyze means to break apart and examine each part closely. Applying academic vocabulary, the rather benign, unintimidating question “Yesterday in class we talked about the causes of World War II. Write three to four causes” becomes “List and succinctly describe three to four of the most significant causes of World War II.” This prompt gives students exposure to list, succinctly, describe, and significant—all words a middle or high school student should know but may not have the opportunity to learn unless they see these words in print, have them explained, and, finally, see an example of a list that succinctly describes the significant causes. Including general academic vocabulary in prompts is not easy to remember to do, and students may feel we are trying to fail them by using this level of language, but Type Two questions should include academic vocabulary. The first time

12/1/2014 Page 5

students see vocabulary this difficult should not be on a high-stakes test. 3(Appendices C, D, and E provide a starter list of general academic vocabulary by grade levels.)

3. Requires the student to use higher-order thinking skills, whenever possible When information about almost any subject is just a Google search away, remembering information is less important than being able to do something with that information. As Wiliam (2014) states, “If the questions are not causing the students to struggle and think, they are probably not worth asking” (p18). Questions that use Bloom’s Taxonomy (Bloom & Krathwohl, 2000) and Webb’s Depth of Knowledge (DOK) (Webb, 2005) to move students from remembering to understanding to analyzing to evaluating to creating not only require higher-order thinking skills but also motivate students. But students can’t analyze or evaluate unless they have content to analyze or evaluate, so remembering or knowing is always important. There is a place for Type Two prompts that ask students to remember important information, but if we can combine remembering and higher-order thinking into one prompt, we have the best of both worlds.

Sometimes using a two-part Type Two question is the easiest way to create this combination. For example, after reading an informational article, a teacher might ask students to list four precise facts from the article (remembering), reflect on which fact they think is the most important (analyzing) and explain why they feel this way (evaluating). An easy way to grade this response would be based on 100 points: 20 points for each of the four facts and 20 points for the quality of the evaluation. Or a teacher might ask students to list and succinctly describe three to four of the most significant causes of World War II and briefly reflect on which cause was the most significant. This prompt adds a level of analysis, giving students an opportunity to think more deeply about the information and teachers a way to assess students’ level of understanding and involvement with the topic. As Kuhrt and Farris (1990) state, “The upper reaches of Bloom’s Taxonomy could not be reached without the use of some form of writing” (p 437).

4. Can be completed quickly: within 2-8 minutes Type Two assignments must be done frequently to achieve maximum benefit; therefore, they must not take up too much class time. If the prompts only required remembering (Bloom’s lowest level) (Bloom & Krathwohl, 2000) or recall (Webb’s lowest level) (Webb, 2005), this requirement would be easy. In fact, one key to determine how well you know something is how fast you can recall it. But with the added dimension of higher-order thinking, crafting a short prompt is a challenge. Analyzing, evaluating, and creating all take time. If done too quickly, the effect is superficial. But if the prompt is specific, students do not have to spend extra time figuring out what to do or filling the page with extraneous information in the hope that they will eventually say what the teacher wants. Remember, the goal is to cause students to retrieve information and, if possible, do something with it. For example, if students read a chapter of a novel for homework, a question might be “Briefly describe three to four events in sequence that happened in the chapter and create a title for the chapter.” This prompt requires effortful retrieval (remembering the sequence of events) and has a dimension of high-order thinking (creating) in the creation of a title. Or in math, students might be asked to identify and define

3 In The Collins Writing Program: Improving Student Performance, there is an assignment instructing students how

to create vocabulary cards with illustrations and sentences that convey the meaning of each word. This is a terrific help for students who are trying to learn new vocabulary.

12/1/2014 Page 6

four math vocabulary terms in a word problem (remembering) and give the formula they would use to solve the problem (applying).

Beware of students complaining that they do not have enough time to complete the prompt. If they need lots of time to recall information, they do not know the content well enough. With the exception of students with significant learning disabilities, most students do not need more time, they need to study harder or read more carefully. In addition to the obvious benefit of not taking too much class time because the expectation is that they will be done quickly, students need to practice getting to the task, focusing on the questions, and not obsessing about perfection—all important test-taking skills.

5. Can be graded quickly: within 10–20 seconds In addition to being completed quickly, Type Two prompts need to be structured so that they can be graded quickly. There are a number of ways to do this. The most obvious is including a number in the prompt (the first criteria on the list of six). With the number as a focus, teachers can quickly scan and calculate. But be careful that the number you assign is reasonable for the time allowed; Type Two prompts are designed to require effortful not impossible retrieval. For example, “Reflect back on last night’s reading assignment. List and number three to four significant facts about the respiratory system, and write one question you still have about how the system functions. Underline the question.” Even if the reading has 10 or more significant facts, this prompt is demanding enough to determine if students thoughtfully read the assignment, particularly because the question a student writes should not have been answered in the reading.

A second strategy to speed up the evaluation process and aid grading is to add student annotations, such as list and number instead of write or circle five vocabulary words instead of use vocabulary. Circle, number, bullet, and underline are all helpful grading aids and have the extra benefit of requiring students to show what they know. In the example “Compose a paragraph about photosynthesis that correctly uses five of our eight science vocabulary words and circle the words,” the student has to consciously identify and circle five domain-specific vocabulary words. If you add student annotations, remember to ask students to quickly look to see if they completed the directions before they hand in their papers. Taking an extra 10 seconds to remind students to circle the vocabulary, number the facts, or underline the prediction will pay for itself in the amount of time saved grading and will give students a last chance to remember to follow the directions. A third time-saving technique is to grade random samples rather than the total class. If you give two Type Two quizzes per week and grade only a quarter of the students each time, you will have four quiz grades per eight-week quarter per student—a reasonable sample. But students will have had the benefit of sixteen quizzes (effortful retrieval) and frequent formative assessment to self-evaluate how they are doing. Warning: I find that some students and parents have difficulty with the random collection. For these classes, collect all quizzes until students get used to frequent quizzing, and then explain that you will be collecting samples for the rest of the year. Also, remind students that these quizzes account for a small part of the grade. You might also add that if you find a quiz question is too difficult, you will throw out the whole set. Stress that by quizzing frequently, you are not trying to punish students. You are instead encouraging regular, frequent study, not infrequent or inefficient, stressful cramming. I also recommend that you include some of the quiz prompts on future tests so that students will have a preview of test

12/1/2014 Page 7

questions and two opportunities to succeed: one on the quiz and one on the test. Occasionally, rather than a sample (say a row or group, by a roll of the dice, or some other random selection technique), I ask, “Everyone who wants to, pass in a quiz.” This tells students that I want them to be successful. Just by the number of papers you receive, you can tell how well you’ve taught. Finally, if you decide to collect random samples—it is important to read the best answers so that students who were not evaluated can hear the correct answer and learn what content is expected.

6. Is graded for content only, not for conventions If Type Two assignments are to be completed and graded quickly, something has to give and that something is conventions. To be clear, conventions are important, but to take advantage of the testing effect; to create a culture of frequent, low-stakes quizzes; to provide quick feedback; and to collect data about students’ learning, most teachers will not have the time to copyedit or proofread. The fact that you are not assessing for conventions should be crystal clear to students: “If you receive a high or perfect grade on a Type Two, do not assume it is a perfect paper.” In some cases, Type Two papers with a grade of 100 will be far from perfect, with spelling, punctuation, or usage errors, but to receive a 100, the content will be correct. For example, in the prompt “Briefly describe three to four events in sequence that happened in the chapter and create a title for the chapter,” students may misspell character names, write run-on sentences, or forget to use quotes around the chapter title. But, if they recall three events and come up with a reasonable title that reflects a sense of what actually happened in the chapter, then the paper gets full credit. Given limited class time to complete the task, these assignments are not edited or proofread and certainly not peer reviewed. They are rough, first drafts and can provide great formative data about student mastery of conventions. If many students write run-on sentences or forget to put the title in quotes, use this information to inform your teaching. Under the pressure of a short quiz, if students have not mastered a particular writing skill to the point where it is automatic, more practice is needed. A perfect next step would be to do a short mini-lesson on the skill in question and ask students to self- or peer-edit their quiz for that skill. You can say “We know to capitalize the words in a chapter title and put it in quotes like the example on the board. Please look at your quiz to see if you did so. If you did not, fix it and when you are done, show it to your partner to check it over.”

For readers who are concerned about the lack of feedback on conventions, one technique may include selecting some samples to be reviewed in a writing conference with the student author, reviewing the answer not only for content but also conventions and requiring that the answer be rewritten so that it is error free. This error-free answer is then shared so all students have a correct, well-written answer in their notebooks. Finally, realize that Type Two quizzes are only one of the Five Types of Writing in the Collins Writing Program. Type One and Type Two are the writing-to-learn components; Type Three, Type Four, and Type Five are the learning-to-write components. The ultimate goal is error-free, content-rich writing—Type Five. Because learning to write is a process, there are many steps toward mastery. Frequent Type Two quizzes will help students move toward mastery. The actual skills of conventions, sentence variety, strong introductions and conclusions, and so forth are best developed using focus correction areas (FCAs) in Type Three and Type Four assignments.

12/1/2014 Page 8

A key strategy in school improvement is to give frequent low-stakes quizzes using Type Two writing in all classes. How many is frequent? A simple answer is two per week: Type Two Tuesdays and Thursdays! In reality, school weeks are rarely the same, given special classes, holidays, and the like, so that a specific recommendation is almost impossible. Nevertheless, the goal is to make frequent quizzes so expected that students do not see them as “gotcha” experiences but as ongoing formative assessment. Type Two assignments help student get a realistic picture of how they are doing in class. In Switch: How to Change Things When Change is Hard, Chip and Dan Heath (2010) review research on positive illusion. “We’re all lousy self-evaluators. ... Only 2% of high school seniors believe their leadership skills are below average. A full 25% of people believe they’re in the top 1% in their ability to get along with others” (p. 114). Most students predict that they are better prepared for tests than they actually are. Frequent quizzes give teachers a way to see if students understand their lessons and give students a chance to test their belief that they know and understand the content. Frequent Type Two quizzes can destroy positive illusions. If a quick, five-minute activity can dispel these illusions, then it is worth the effort. Even better, it also helps to embed content, teach general academic vocabulary, encourage higher-order thinking skills, and is a reasonable test prep activity. How could we not be doing it? The answer is simple: It is not how most of us currently teach, it is not how we were taught, and it requires skill and practice to develop prompts that meet the six characteristics.

Practice Makes Perfect

To make the change, first, give yourself permission to learn. If you are like me, you will find that your third-period prompts are better than your first-period prompts. Prompts will become clearer because students will ask clarifying questions. You will remember to ask students to underline or circle elements of the answer because you’ll discover how much harder the prompts are to grade when students do not. You’ll teach general academic vocabulary when you discover that students think summarizing is to string quotes together and not to put central ideas into their own words. As you get better, the students will help you. If you forget to tell them, students will ask if they should circle the vocabulary. They will ask if they should define words in the prompt if they are not sure if you want them to do so. But they will only do it if they have had sufficient practice and know that their work may be graded.

Turn the Table

Here is the dream: Once or twice a week, class periods will end with this assignment: “Write a fair but challenging prompt that I could use at the beginning of the next class.” In Make Just One Change, Rothstein and Santana (2011) make a strong case for teaching students to ask questions. They feel it is the single most important skill to teach! In Embedded Formative Assessment, Wiliam (2011) states:

For older students (say, fourth grade and up), one technique that is particularly useful both for getting students to clarify, share, and understand learning intentions and for informing the teacher about the students’ level of understanding is to have the students design test items, with correct answers, about what they have been learning. In a study involving several

Implementing This Strategy

12/1/2014 Page 9

experiments with 260 college students, this was shown to be more effective than administering practice tests, giving students study guides, or leaving them to prepare for tests in their own way. ... This can be a particularly effective strategy with disaffected older students, who often feel threatened by tests. Writing a test for the topic they have completed, and knowing that the teacher is going to grade the questions rather than the answers, can be a hugely liberating experience for many students. (p. 68)

By taking a few minutes at the end of class to have students create a prompt, you are giving them a chance to reflect on what went on during class. By reviewing the prompts, you gain insight into what they understand and what they think is important. Prompt generation also actively engages students so they are not spending the last five minutes packing up or thinking about their next class. After students have had experience answering two teacher generated questions each week for the first couple months of the school year, they will have had approximately 16 quizzes. That is a lot of examples of precise and clear prompts that include general academic vocabulary and encourage higher-order thinking that they can use as models for their own questions. Additionally, they will have developed a sense of how long a prompt takes to answer, so they can develop prompts that are not too time consuming. Imagine a mid-November scenario where students have lists of general academic vocabulary (see Appendices C, D and E) in their notebooks and have been seeing these words in prompts for eight weeks. A teacher asks students to write a prompt and then share it with a partner. The pair then determines the better of the two or combines prompts to create an even better one. In the last minute or two of class, the teacher calls on random pairs to share their best prompt. Listening to a few, the teacher announces one or two prompts that are candidates for next class’s quiz—no surprises. The next class begins with the prompt generated by the students. Maybe the teacher gives the team who generated the question an automatic 100 on the quiz for creating such a good prompt, encouraging students to generate demanding and challenging prompts, not softball questions. Students keep the prompts and answers as a study guide. Occasionally, these prompts are recycled as quizzes, especially prompts that demand students recall critically important information. Significant parts of major tests are made up of these past prompts, giving students incentive to keep and review past prompts. Tests become more predictable, students feel they have had an influence in the design of summative assessments, and teachers get a steady stream of information about how they are teaching and what students are learning—a winning situation for both teachers and students.

12/1/2014 Page 10

Works Cited

Anderson, L., & Krathwohl, D., et al. (2000). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Longman.

Collins, J. (2007). The Collins writing program: Improving student performance through writing and thinking across the curriculum. West Newbury, MA: Collins Education Associates.

Gooblar, D. (2014, February 5). We should give students more tests. Seriously [blog post]. Vitae. Retrieved from https://chroniclevitae.com/news/315-we-should-give-students-more-tests-seriously

Greenstein, L. (2010). What teachers really need to know about formative assessment. Alexandria, VA: ASCD.

Heath, C., & Heath, D. (2010). Switch: How to change things when change is hard. New York: Broadway Books.

Kuhrt, B., & Farris, P. (1990, March). “Empowering students through reading, writing, and reasoning.” Journal of Reading, 33(6), 431–441. http://www.jstor.org/stable/40031979

National Governors Associates Center for Best Practices, Council of Chief State School Officers. (2010). Common core state standards for English language arts & literacy in history/social studies, science, and technical subjects (Appendix A: Research supporting key elements of the standards). Retrieved from http://www.corestandards.org/ELA-Literacy/

Partnership for Assessment of Readiness for College and Careers. [2014] Grade 7 prose constructed response from research simulation task (summary) [website page]. (2014). Retrieved from http://www.parcconline.org/samples/english-language-artsliteracy/grade-7-prose-constructed-response-research-simulation-task

Rothstein, D., & Santana, L. (2011). Make just one change: Teach students to ask their own questions. Cambridge, MA: Harvard Education Press.

Webb, N. and others. (2005). Web alignment tool [PDF document]. Retrieved from http://static.pdesas.org/content/documents/M4-Slide_14_JD_Depth_of_Knowledge.pdf

Wiliam, D. (2011). Embedded formative assessment. Bloomington, IN: Solution Tree Press. Wiliam, D. (2014). “The right questions, the right way.” Educational Leadership, 71(6), 16-19.

12/1/2014 Page 11

Appendices Appendix A – One Page Description of the Collins Writing Program’s Five Types of Writing

Appendix B – Annotated Bibliography of Research on the Power of Frequent Testing and Quizzing

Appendix C – General Academic Vocabulary, Grades 3-5

Appendix D – General Academic Vocabulary, Grades 6-8

Appendix E – General Academic Vocabulary, Grades 9-12

Appendix F – Ten Great Type Twos

Appendix G – One-page Version of Six Characteristics: Frequent Low-stakes Quizzes Using Type Two

Writing

Appendix H – Evaluate Type Two Prompts

Appendix I – Editing Type Two Prompts (High School Science)

12/1/2014 Page 12

Appendix A

12/1/2014 Page 13

Appendix B Annotated Bibliography of Research on the Power of Frequent Testing and Quizzing

Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C. C. (1991). Effects of frequent classroom testing. Journal of Educational Research 85(2), 89–99. Retrieved from http://www.jstor.org/stable/27540459

A meta-analysis on 40 studies on testing revealed the effects of frequent classroom testing. Researchers found that increased testing frequency can improve post-instruction achievements but there is a point of diminishing returns as well. Other findings include change in student attitude toward increased testing frequency, student performance on shorter tests versus longer tests, and the change in performance based on the distribution frequency of tests.

Carey, B. (2014, September 4). Why flunking exams is actually a good thing. The New York Times Magazine, http://www.nytimes.com/2014/09/07/magazine/why-flunking-exams-is-actually-a-good-thing.html?_r=0.

Carey’s article looks at pretests and frequent tests, such as pop quizzes, as learning devices rather than measures of achievement. Pretests prime the brain for learning. By guessing or failing to answer correctly, students are alerted to what to watch for during the instruction and remember it better as a result. Studies show that frequent tests are an effective study aid. The best results come from students spending the first third of study time memorizing and the remaining two thirds self-testing or taking practice tests.

Dempster, F. N. (1991, April). Synthesis of research on reviews and tests. Educational Leadership, 71–76. Retrieved from http://www.ascd.org/ASCD/pdf/journals/ed_lead/el_199104_dempster.pdf

The author reviewed research and theories on the efficacy of several methods of reviewing information in preparation for classroom tests and on the effectiveness and use of testing. His review revealed effective methods of studying, including spaced repetition and in-class reviews. The author also determined that testing is a teaching tool as well as a measurement tool and that frequent tests are beneficial to students. He concludes by offering educators ideas for incorporating review and more testing into the classroom environment.

Dunlosky, J. (2013, Fall). Strengthening the student toolbox: Study strategies to boost learning. American Educator, 12–21. Retrieved from http://www.aft.org/pdfs/americaneducator/fall2013/Dunlosky.pdf

This article states that teaching study skills is as important as teaching the content itself, because good study skills produce better long-term results. The article reviews 10 learning strategies that the author and his colleagues examined in a literature review, rating their effectiveness and outlining best approaches to the most successful learning strategies. Practice testing and distributed practice were rated as the most effective learning strategies. The author concludes with seven tips for teaching students to use these two techniques.

Glenn, D. (2007). You will be tested on this. The Chronical of Higher Education, 53(40), A14. Retrieved from http://chronicle.com/article/You-Will-be-Tested-on-This/14732

This article examines the opinions of several education professionals on the effect of quizzing more often in the classroom. Professionals interviewed included a psychology professor studying the testing effect in an Illinois middle school, a graduate running an experiment on different testing methods, and an associate psychology professor experimenting with testing in

12/1/2014 Page 14

large, introductory college courses. All show evidence that frequent short quizzes help students learn and retain information better than less-frequent longer tests do.

Rohrer, D., & Pashler, H. (2010). Recent research on human learning challenges: Conventional instructional strategies. Educational Researcher 39(5), 406–412. doi: 10.3102/001389X10374770

In a research review, the authors sought to determine how instruction methods and their timing affect students’ ability to learn and retain information, studying the effects of several methods of testing and study on long-term learning results. Additionally, the authors examined why testing and study strategies that produce poorer results remain more popular with teachers and students. The authors conclude that if education is to become more evidenced based, more successful methods of teaching and learning, such as frequent testing, spaced practice, and interleaving, should be applied.

Rosenshine, B. (2012, Spring). Principles of instruction: Research-based strategies that all teachers should know. American Educator, p. 12–19, 39. Retrieved from https://www.aft.org/pdfs/americaneducator/spring2012/Rosenshine.pdf

The author presents 10 effective principles of instruction, based on research in cognitive science, classroom practices, and cognitive supports. Each principle is outlined and then followed with research findings and classroom applications by master teachers. Effective principles include a daily review of the previous lesson, breaking up new material into small chunks, and asking many questions and checking all student responses. The article offers many ideas teachers can easily apply in their own classrooms.

Wiliam, D. (2014, March). The right questions, the right way. Educational Leadership, 16–19. Retrieved from http://www.ascd.org/publications/educational-leadership/mar14/vol71/num06/The-Right-Questions,-The-Right-Way.aspx

In this article, the author outlines the problem with initiation-response-evaluation (IRE) model of evaluation: the teacher asks the class a question, several students raise their hands to respond, one student is chosen to respond, and the teacher evaluates the response for the entire class. The author then discusses two systems to use instead—randomly selecting a student to answer and asking all students to respond—outlining procedures, benefits, and dangers. The author concludes with methods for incorporating these systems into the classroom.

12/1/2014 Page 15

Appendix C General Academic Vocabulary, Grades 3–5

caption the words printed under a picture or cartoon that explain what it is about category the name for a group of things organized according to shared qualities

character a person in a story, novel, play, or film clause one part of a complex or compound sentence with its own subject and

predicate clear/clearly easy to understand; or, to say something in a way that is obvious and not

easily mistaken closure a finish of something; an ending

comprehension the ability to understand something concluding statement a way to wrap up a piece of writing; it comes at the end of a composition

and reminds the reader what the central ideas of the writing are concrete something that is definite and specific; not general connect provide a link or relationship between information or ideas convey to tell or explain clearly so that the reader can understand without confusion

demonstrate to make clear by explaining or showing with words descriptive giving an explanation of what something is like; how things can be seen,

heard, or felt in some way detail facts or information about something

develop gradually becoming more detailed and advanced dialogue conversation between people in a book, play, or film

domain-specific words words and phrases that are specific to a particular subject or field of study e.g. a Latin abbreviation that means “for example”

establish to set something up, start; to build in a secure, solid way event a happening, especially an important one

event sequence an order in which events happen or are arranged; continuing and connected series of events

evidence information that you give to prove a point examine to look at carefully and closely; to question

facts information that is true and correct formatting the way you arrange your writing

general observation carefully looking at or watching something in order to learn something illustration labeled drawings or pictures that help explain ideas

information facts about something or someone introduce to tell about a topic at the beginning of a piece of writing introduce to tell about an event or experience at the beginning of a story

key the most important thing in a group linking words words that connect ideas, sentences, and paragraphs together

logical each idea that follows is connected to the one before so that it makes sense logically showing good sense or reasoning

multimedia text, pictures, and video that you use to explain something narrator the person who is telling the story, usually from their own point of view opinion what you think or believe about something

organizational structure the order of ideas in your writing that makes it easy for readers to follow and understand

organize to order the ideas in your writing so it is easy for readers to follow and

12/1/2014 Page 16

understand orient to give knowledge of a new situation; to become familiar with pacing the speed at which the story moves along and the way in which the story is

revealed phrase a small group of words that form a part of a sentence. A phrase can be on its

own or within a sentence. point of view in opinion writing, a position or attitude a writer takes on a subject

precise exact; expressed clearly and not in general terms quotation

a sentence or phrase taken from a book, poem, or play, which is repeated by someone else

reasons facts or ideas that support a belief, position, or attitude sensory relating to the senses of sight, touch, sound, smell, and taste

signal an word or action that sends a message situation what is happening in a particular place at a particular time

specific exact and detailed support to back up an opinion with facts, details, and examples

temporal relating to time text any written material

topic the subject being written about transitional words words or phrases that show a change from one situation, condition, or event

to another unfold to happen, become clearly understood

12/1/2014 Page 17

Appendix D General Academic Vocabulary, Grades 6–8

analyze to consider more than one opinion on a topic or question before drawing your own conclusions or to consider a topic or question carefully by breaking your response to it into parts

argument a way to convince another person of your opinion by using evidence from a text or another relevant source

audience the person or group of people, in addition to your classmates and teacher, who may read your writing

capture to hold the attention of the reader

category the name for a group of things organized according to shared qualities

cause/effect a way to organize writing that explains what made something happen (its cause) and what the results were (its effect)

citation a passage or phrase from a book or other piece of writing

cite if you cite something, you quote it or mention it, especially as an example or proof of what you are saying

claim an idea you believe to be true that you can support with evidence

classification to arrange by similar qualities

clause one part of a complex or compound sentence with its own subject and predicate

cohesion the order of your ideas and how you link them together to form a unified whole

comparison/contrast a way to organize writing that shows how things are alike (comparison) and how they are different (contrast)

concept an idea, usually something you cannot touch or see

concrete dealing with facts and certainties; not general ideas

concrete details specific, particular points, features, or parts

connection linking events together, showing the reader how they are related

content the ideas or subject that your writing expresses

context the information surrounding a word or phrase that determines exactly how it was meant

convey to tell or explain clearly so that the reader can understand without confusion

counterclaim the response given by someone who doesn’t agree with your claim

credible believable, trustworthy; based on data or experience

descriptive giving an explanation of what something is like; how things can be seen, heard, or felt in some way

domain-specific vocabulary words that are used in a subject that are usually not used in other subjects

e.g. a Latin abbreviation meaning “for example”

engage to get and hold the attention of

evidence the relevant and concrete information and examples you give to prove your point and support your reasons

formal style written in a serious voice that avoids casual phrasing, slang, and contractions but can include I

12/1/2014 Page 18

graphics a drawing, usually made up of lines

heading/subheading a short statement, usually in a different typeface, that describes what a section of text is about

knowledgeable a clear understanding of a particular topic; having had extensive experience with the topic

logical each idea that follows is connected to the one before so that it makes sense

multimedia use of several types of media such as text, pictures, video, and sound

objective tone a reasonable voice that relies on evidence, not emotions, to develop an argument

organization in writing, the order of ideas and information

orient to give knowledge of a new situation; to become familiar with

pacing the speed at which the story moves along and the way in which the story is revealed

paraphrase to express what is said or written using your own words and in about the same number of words as the original speech or piece of writing

plagiarize to use or copy another person’s idea or work and present it as your own

point of view in narrative writing, the position of the narrator in a story: first-person, second-person, or third-person

precise exact; expressed clearly and not in general terms

preview to show what is coming, to tell a reader how a piece of writing will be organized

quotation a phrase, line, or paragraph put in quotation marks that you include, word-for-word, in your own writing while attributing the author

quote to include the exact words of another writer or speaker in your own writing to support a particular point or claim while attributing the author

reason a statement that explains why the reader should believe your claim

reflection in narrative writing, when a character describes learning resulting from the events or actions

relevant closely relates to and supports the point you are making

sensory language words relating to the senses of sight, touch, sound, smell, and taste

setting the particular place and time where an event occurs

shift a change in time frame or setting

slang words or expressions that are not formal, “street language” not “school language”

source the book, magazine, or internet reference in which the story, argument, or piece of research you are summarizing, paraphrasing, or quoting first appeared

substantive important, serious; not silly or extreme

subtitle the second title, usually under the main title, that is often longer and explain the title

time frame the length of time during which an event or experience develops

transition a word or phrase that helps the writer change from one idea to another

transitional words a word or phrase that helps the story change from one setting or time frame to another

well-structured organized; logical

12/1/2014 Page 19

Appendix E General Academic Vocabulary, Grades 9–12

accurate correct to a very detailed level

analogy a story or situation similar to another that helps the reader understand

analyze to consider more than one opinion on a topic or question before drawing your own conclusions or to consider a topic or question carefully by breaking your response to it into parts

argument a way to convince another person of your opinion by using evidence from a text or another relevant source

audience the person or group of people, in addition to your classmates and teacher, who may read your writing

caption the words printed under a picture or cartoon that explain what it is about

capture to hold the attention of the reader

category the name for a group of things organized according to shared qualities

central idea the chief point or main idea (expressed in a sentence or two, not just a few words) that an author is making about a topic

citation a passage or phrase from a book or other piece of writing

cite if you cite something, you quote it or mention it, especially as an example or proof of what you are saying

claim an idea you believe to be true that you can support with evidence

close reading the careful reading of a text to understand the central ideas and to analyze how the author uses words, details, and organization to make an impact

coherent clear, logical; well planned so that all parts go together well

cohesion the order of your ideas and how you link them together

comparison/contrast a way to organize writing that shows how things are alike (comparison) and how they are different (contrast)

complex ideas ideas with many different or related parts and, therefore, often difficult to understand

connection linking ideas or events together and showing the reader how they are related

convention a traditional method or style; a standard custom

convey to tell or explain clearly so that the reader can understand without confusion

counterclaim the response given by someone who doesn’t agree with your claim

credible believable, trustworthy; based on data or experience

discipline a field of study

distinctions a clear difference between similar things

e.g. a Latin abbreviation meaning “for example”

evidence the relevant and concrete information and examples you give to prove your point and support your reasons

formal style written in a serious voice that avoids casual phrasing, slang, and contractions but can include I

gist the general meaning of a piece of writing, usually expressed in a sentence or two (similar to central idea)

implications likely to happen as a result of; consequences

12/1/2014 Page 20

knowledgeable a clear understanding of a particular topic; having had extensive experience with the topic

metaphor a figure of speech suggesting a similarity between two unrelated things; an imaginative comparison without using like or as

mood how the reader feels as a result of reading the narrative

mystery a puzzling problem with no obvious solution; a story in which strange things happen that are not explained until the end

norm official level of achievement or standard of behavior expected

objective tone a reasonable voice that relies on evidence, not emotions, to develop an argument

organizational structure how a text is organized; using design features to clearly present information

outcome the effect, result, or consequence

paraphrase to express what is said or written using your own words and in about the same number of words as the original speech or piece of writing

plagiarize to use or copy another person’s idea or work and present it as your own

plot the causes and logical structure which connect the events of a story

plot line an event that centers around one or more groups of characters

point of view in narrative writing, the position of the narrator in a story: first-person, second-person, or third-person

precise expressed exactly and in a way that distinguishes your idea from others

purpose the reason an author decides to write about a specific topic. Once a topic is selected, the author must decide whether his purpose for writing is to inform, argue, tell a story, or some combination of these.

quotation a phrase, line, or paragraph put in quotation marks that you include, word-for-word, in your own writing while attributing the author

reason a statement that explains why the reader should believe your claim

reflection in narrative writing, when a character describes learning resulting from the events or actions

relevant closely relates to and supports the point you are making

resolution a solution to a problem, or result of an action

sensory language words relating to the senses of sight, touch, sound, smell, and taste

setting the particular place and time where an event occurs

significance the importance or meaning of something

significant not trivial; important or crucial

simile a comparison using like or as

smooth progression gradual development of a series of related events

source the book, magazine, or internet reference in which the story, argument, or piece of research you are summarizing, paraphrasing, or quoting first appeared

subplot plot which is less important than and separate from the main plot but usually linked to the main plot

substantive important, serious; not silly or extreme

subtitle a second title, often longer and more detailed than the first title

summary a brief statement of the central ideas of a speech, event, process, or piece of writing

12/1/2014 Page 21

supporting details the facts, statements, and examples that give us a full understanding of the central idea. Supporting details answer the six basic research questions of who, what, where, when, why, and how.

suspense tension experienced by the reader due to an uncertain outcome of events in the narrative

syntax the way that words are put together in order to make a sentence; the order and relationship of phrases in a sentence

technique a particular method of doing an activity; usually involves skills that are developed through training and practice

telling details details that reveal the true nature of a person or situation; details that have a strong impact

text structure how a text is organized and if it has any graphics, headings, or illustrations that make it easier to understand

theme the central idea, subject, or meaning of a literary work

tone the attitude an author has toward the subject. The tone of a narrative could be formal or informal, playful, ironic, optimistic or pessimistic, etc.

topic what the text is about, usually expressed in a word or phrase

topic sentence the sentence in a paragraph that contains the central idea of that paragraph

transition a word or phrase that helps the writer change from one idea to another

vivid easy to see or imagine; intense, clear, or detailed

well-chosen carefully selected

12/1/2014 Page 22

Appendix F Ten Great Type Twos

1. Create two or three*quiz questions about today’s content that I can ask at the beginning of

tomorrow’s class. Ensure that the questions will require higher order thinking skills.

2. Compose a short informative paragraph about________________for a student who missed today’s class. Correctly use, explain, and circle five to seven* key terms.

3. Describe two* ways________________and_________________are similar and two* ways they are different. Label and number the similarities and differences.

4. Closely examine the item (picture, illustration, graph, etc.) on the screen. Create and label a descriptive title. Write and label a short explanatory caption.

5. Here is an article about the subject we have been studying. All the text features (title, subtitle, subheadings, captions, etc.) have been removed. Imagine you are the editor. Create three to five* text features on the article.

6. Explain two or three* reasons this answer would not receive full credit on a test. Number your reasons.

7. Write a claim statement on your position on________________________. Make sure your claim statement includes the topic, your position on the topic, briefly stated reasons without details, and does not include “I”.

8. (for reading fiction) List four to five* events that happened in the reading assignment and predict one event that will happen in the next (chapter, section, etc.). Underline your prediction.

9. List and briefly describe at least four* discrete steps you would do to solve the following problem:_________________________________________. Number the steps in order.

10. We have completed reading the text_________________________________________. List and number two to four* central ideas in your own words in the order they appeared in the text. Do not give your opinions. (*Or some other appropriate number.)

12/1/2014 Page 23

Appendix G Frequent Low-Stakes Quizzes Using Type Two Writing

For more than three decades, the Collins Writing Program has advocated frequent use of Type Two

assignments, especially assignments that quiz students on content. With the advent of rigorous state

standards, we have enhanced our Type Two prompts to include six characteristics:

Include a number in the prompt to make it quantifiable and easy to score

Use general academic vocabulary in the prompt

Require the student to use higher-order thinking skills, if possible

Complete quickly, 3–8 minutes

Grade quickly, 10–20 seconds

Grade for content only, not for conventions

Performed repeatedly, Type Two writing assignments harness the power of frequent quizzes to promote

the retention of content. Research overwhelmingly finds that the more we quiz students using open

response and low-stakes assessments, the more students learn.

In addition, for students to succeed based on the new standards, they must:

Understand general academic vocabulary, the language of college and the workplace. Type Two

prompts must include academic vocabulary, such as identify, explain, support, and prioritize.

Be able to analyze, evaluate, infer, and synthesize. Type Two prompts used for quizzes must

require higher-order thinking and utilize the principles of formative assessment to provide

teachers with ongoing feedback on what is being learned.

12/1/2014 Page 24

Appendix H Evaluate Type Two Prompts

Listed below are six Type Two prompts a social studies teacher might ask a middle school or high school

class. Rank them in order of quality by using #1 to indicate the best example and using #6 to indicate the

worst example.

Your Rank

1. Using your best writing skills (don’t forget capitalization), write a paragraph about

the causes of World War II. ___________

2. Over the last few days we have been discussing the causes of World War II. Tell me

some of the causes. ___________

3. Compare and the contrast the causes of World War I and World War II. ___________

4. We are concluding our section on the causes of World War II. Create one Type Two

prompt that I can use at the beginning of our next class about the causes. Make sure

your prompt is specific, fair, clear, and does not take more than four to six minutes

to answer. Also be sure it requires thinking skills (e.g., analysis, explanation,

evaluation), not just memorization.

___________

5. Analyze the causes of World War II. Include in the analysis details about the causes

and an explanation of how these causes led to this horrific war. Write in a formal

style with an objective tone.

___________

6. List and number three to four of the most significant causes of World War II. After

the list, write a brief four to six line reflection on what cause you feel was the

primary cause. Bracket the reflection.

___________

12/1/2014 Page 25

Appendix I Editing Type Two Prompts (High School Science)

Listed below are ten prompts on cell growth and division taken from a popular high school biology text

book. Review each prompt and determine if it meets the six criteria for a Type Two. If not, edit the

prompt so it aligns with our criteria.

1. Summarize what happens during the process of cell division.

2. Explain how a cell’s DNA can limit the cell’s size.

3. Describe what is meant by each of the following terms; cell volume, cell surface area, ratio of

surface area to volume.

4. How is a cell’s potential growth affected by its ratio of surface area to volume?

5. Describe how a cell’s chromosomes change as a cell prepares to divide.

6. What is the relationship between interphase and cell division?

7. Summarize what happens during interphase.

8. Explain how the following terms are related to one another: DNA, centromere, chromosome,

chromatid.

9. List the following events in the correct sequence, and describe what happens during each event:

anaphase, metaphase, prophase, and telophase.

10. Summarize what happens during the cell cycle.