Assessing Student Knowledge: Going to the Next Level

Assessing Student Knowledge:Assessing Student Knowledge:Going to the Next LevelGoing to the Next Level

Knowledge Network of Experiences Knowledge Network of Experiences Social Studies Idea Social Studies Idea

Sunal & Hass. A Constructivist Approach to Social Studies, 2005

Meaningful Social Studies instruction,“enables individuals to personally explain

relationships and decide how to encounter socialevents in ways that are consistent with their values and those of their culture…memorizing facts doesnot encourage students to find meaning in what

they are learning, nor does it help them make andtest decisions.”

How to Evaluate and Create Quality Assessment Items

This training will introduce the basics of aligning and evaluating assessment items. Using Blooms II and the research of James Popham, Hilda Taba and others, participants will learn how to align knowledge and cognitive levels and use them to identify and develop quality assessment items.

A Note about Test BanksA Note about Test Banks Not all test items are created equallyNot all test items are created equally Socials Studies textbook test banks Socials Studies textbook test banks

are predominantly (90%+)made up are predominantly (90%+)made up of lower level questions.of lower level questions.– Why?Why?

Most nationally produced banks are Most nationally produced banks are include a majority of lower level include a majority of lower level questionsquestions

Which of the following cognitive levels is Which of the following cognitive levels is best aligned to the learning objective best aligned to the learning objective

below.below.

6.9B Identify problems and issues that may arise when one or more of the assessments that students receive is not aligned to the learning objective and explain how this might effect a student’s academic success.

A. Remember

B. Understand

C. Apply

D. Analyze

Identify

explain

The Knowledge The Knowledge DimensionDimension

The Cognitive Process DimensionThe Cognitive Process Dimension11

RememberRemember22

UnderstandUnderstand33

ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate

A.A.Factual KnowledgeFactual Knowledge

B.B.Conceptual Conceptual

KnowledgeKnowledge

C.C.Procedural Procedural

KnowledgeKnowledge

D. D. Metacognitive Metacognitive

KnowledgeKnowledge

A Taxonomy for Learning, Teaching and AssessingA Taxonomy for Learning, Teaching and AssessingLorin Anderson and David KrathwohlLorin Anderson and David Krathwohl

Assessment AlignmentAssessment Alignment

Read the TEKS SEs and Plot it on Read the TEKS SEs and Plot it on the taxonomy chartthe taxonomy chart

Read the corresponding Read the corresponding question, plot it, and then decide question, plot it, and then decide whether or not the assessment whether or not the assessment item is aligned to the TEKS SE.item is aligned to the TEKS SE.



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

1.6A Identify and describe the 1.6A Identify and describe the physical characteristics of places such as landforms, bodies of water, natural resources, and weather

PhysicalPhysicalCharac.Charac.

Identify Identify and and describedescribe


http://www.tea.state.tx.us/ssc/teks_and_taas/teks/gloss1.htm#physchar

11stst Grade Assessment Item Grade Assessment Item

The picture shows a –The picture shows a –1.1. PlainPlain2.2. ValleyValley3.3. MountainMountain4.4. ForestForest



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

1.6A Identify and describe the physical characteristics of 1.6A Identify and describe the physical characteristics of places such as landforms, bodies of water, natural places such as landforms, bodies of water, natural resources, and weatherresources, and weather

PhysicalPhysicalCharac.Charac.



ITEMITEM



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

3.2B Compare ways in which people in the local community 3.2B Compare ways in which people in the local community and communities around the world meet their needs for and communities around the world meet their needs for government, education, communication, transportation, and government, education, communication, transportation, and recreation, over time and in the presentrecreation, over time and in the present

Gov, Ed.Gov, Ed.Com., etc.Com., etc.

CompareCompare CompareCompare

Need

3rd Grade Assessment Item3rd Grade Assessment Item In 1900, people traveling far from home could stay in touch with friends using a __________, but after 1980, they could use a __________.A. computer; pay phoneB. newspaper; telegramC. radio; walkie-talkieD. telegram; cell phone



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

3.2B Compare ways in which people in the local community 3.2B Compare ways in which people in the local community and communities around the world meet their needs for and communities around the world meet their needs for government, education, communication, transportation, and government, education, communication, transportation, and recreation, over time and in the presentrecreation, over time and in the present

Gov, Ed.Gov, Ed.Com., etc.Com., etc.

CompareCompare CompareCompare

Need

ITEMITEM



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

5.6B Translate geographic data into a variety 5.6B Translate geographic data into a variety of formats such as raw data to graphs and of formats such as raw data to graphs and maps. maps.

TranslateTranslateTranslateTranslate

Geographic Geographic DataData

TranslateTranslate


5th Grade Assessment Item5th Grade Assessment Item

The elevation at the western edge of the Caspian Sea is below sea level. The elevation of the Himalayas is 4000 meters and above. Which map indicates this information?A. IB. IIC. IIID. IV



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

5.6B Translate geographic data into a variety 5.6B Translate geographic data into a variety of formats such as raw data to graphs and of formats such as raw data to graphs and maps. maps.

TranslateTranslateTranslateTranslate


TranslateTranslate


ITEMITEM



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

Understand data

6.3(B. pose and answer questions about geographic 6.3(B. pose and answer questions about geographic distributions and patterns for selected world regions distributions and patterns for selected world regions

and countries shown on maps, graphs, charts, models, and countries shown on maps, graphs, charts, models, and databasesand databases

Map, chart skills

Geographic distributions- regions


Pose questions

Pose questions

Pose questions

6th Grade Assessment Item6th Grade Assessment ItemWhich of the following is a question a geographer might ask about tea production in Sri Lanka to find information that is not on the map?

A. How many oceans surround Sri Lanka?B. What other crops are grown in Sri Lanka?C. How far is Sri Lanka from theequator?D. What are the natural features of central Sri Lanka?



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

Understand data

6.3(B. pose and answer questions about geographic 6.3(B. pose and answer questions about geographic distributions and patterns for selected world regions distributions and patterns for selected world regions

and countries shown on maps, graphs, charts, models, and countries shown on maps, graphs, charts, models, and databasesand databases

Map, chart skills



Pose questions

Pose questions

Pose questions

ITEMITEM



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

8.4 A analyze causes of the American Revolution, including 8.4 A analyze causes of the American Revolution, including mercantilism and British economic policies following the mercantilism and British economic policies following the French and Indian WarFrench and Indian War

Analyze Analyze causes: causes: mercantilismmercantilism

8th Grade Assessment Item8th Grade Assessment ItemWhich of the following was NOT part of themercantilist policy of England?A. Regulating colonial trade by taxationB. Requiring colonial businesses to tradewith EnglandC. Increasing exports of manufacturedgoods to the coloniesD. Decreasing use of natural resources from

the colonies



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

8.4 A analyze causes of the American Revolution, including 8.4 A analyze causes of the American Revolution, including mercantilism and British economic policies following the mercantilism and British economic policies following the French and Indian WarFrench and Indian War

Analyze Analyze causes: causes: mercantilismmercantilism

ITEMITEM



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

WH1(D. explain the significance of the following dates: 1066, 1215, 1492, 1789, 1914-1918, and 1939-1945.

Explain significance of date

Sig. of date-

assess

World History Assessment ItemWorld History Assessment ItemThe events of 1914 to 1918 marked the firsttime in history that —A. the Western European nations foughtas alliesB. nations from five continents fought ina single conflictC. the United States took part in a waroutside its territoryD. nations forced men to fight by settingup military drafts



RememberRemember22


ApplyApply44

AnalyzeAnalyze55

EvaluateEvaluate66

CreateCreate



KnowledgeKnowledge


KnowledgeKnowledge


KnowledgeKnowledge

WH1(D. explain the significance of the following dates: 1066, 1215, 1492, 1789, 1914-1918, and 1939-1945.

Explain significance of date

Sig. of date-

assess

ITEMITEM

It is not important which taxonomy It is not important which taxonomy that you use but that you use onethat you use but that you use one

If you do not know the knowledge If you do not know the knowledge and cognitive levels of your learning and cognitive levels of your learning objective then you cannot effectively objective then you cannot effectively align your instruction, assessment align your instruction, assessment and questioning to itand questioning to it

Learning Objective: Blooms IILearning Objective: Blooms II

L. Anderson, D. Krathwohl, Taxonomy for Learning, Teaching, and Assessing, A: A Revision of Bloom's Taxonomy of Educational Objectives, (2001..

BreakBreak

Determining Difficulty Determining Difficulty LevelLevel

Determining Difficulty LevelDetermining Difficulty Level There are two different difficulty There are two different difficulty

levels that we are looking at with any levels that we are looking at with any assessment item:assessment item:

Cognitive DifficultyCognitive Difficulty Procedural DifficultyProcedural Difficulty

Determining Difficulty LevelDetermining Difficulty Level

Cognitive DifficultyCognitive Difficulty This is basically: How hard is This is basically: How hard is

question based on Blooms question based on Blooms Taxonomy?Taxonomy?

No question on a test should be No question on a test should be above the level of the learning above the level of the learning objective (TEKS SE) **objective (TEKS SE) **

It is OK to have questions that are It is OK to have questions that are lowerlower **Unless you have a stated learning objective

that has been clearly expressed to the students and taught.


Cognitive DifficultyCognitive Difficulty EASYEASY

– RememberRemember– UnderstandUnderstand

MEDIUMMEDIUM– ApplyApply– AnalyzeAnalyze

HARDHARD– EvaluateEvaluate– CreateCreate


Procedural DifficultyProcedural Difficulty This is basically: How many mental This is basically: How many mental

processing steps does the student processing steps does the student have to go through to answer the have to go through to answer the question?question?

The greater the number of The greater the number of processing steps the higher the processing steps the higher the difficulty leveldifficulty level

Determining Difficulty LevelDetermining Difficulty LevelProcedural DifficultyProcedural Difficulty EASYEASY

– The item includes only the stem and the answer choicesThe item includes only the stem and the answer choices MEDIUMMEDIUM

– The item includes a graphic, short reading selection, The item includes a graphic, short reading selection, map etc. (Stimulus piece) The student only has to map etc. (Stimulus piece) The student only has to interpret the stimulus or pull information from it to interpret the stimulus or pull information from it to select the correct answer.select the correct answer.

HARDHARD– The item includes a graphic, short reading selection, The item includes a graphic, short reading selection,

map etc. (Stimulus piece) The student has to infer, map etc. (Stimulus piece) The student has to infer, analyze, summarize, etc. and apply that to the stem or analyze, summarize, etc. and apply that to the stem or answer choices to select the correct answer.answer choices to select the correct answer.

What does the student have to “do” What does the student have to “do” to answer the question? to answer the question?

What does the student have to “do” What does the student have to “do” to answer the question? to answer the question?

What What does the does the student student have to have to “do” to “do” to answer answer

the the question? question?

Specificity DifficultySpecificity Difficulty We DO NOT want to use specificity to increase the

difficulty level of the question!5.2(A) identify the contributions of significant individuals during the revolutionary

period, including Thomas Jefferson and George Washington;

George Washington was in charge of which army during the American Revolution?

American, German, Chinese, Russian Revolutionary, Continental, Virginia Regulars, American

George Washington defeated the British in which battle that ended the Revolutionary War?

Yorktown, Saratoga, Trenton, Cowpens

George Washington credited some of his ability to win the battle of Yorktown to his horse -

Faithful, Liberty, Constitution, Magnolia

Specificity DifficultySpecificity Difficulty Choices and Distracters

5.2(A) identify the contributions of significant individuals during the revolutionary period, including Thomas Jefferson and George Washington;

Thomas Jefferson wrote which important historical document -

Declaration of Independence, Civil Rights Act of 1964, Bible, Quran

Declaration of Independence, Constitution, Mayflower Compact, Gettysburg Address

Wall of separation letter, Manual of Military Practice, Constitution for Virginia, Virginia Resolutions of 1798

QuestionsQuestions

How to Evaluate and Create Quality Assessment Items

Question Alignment ApplicationQuestion Alignment Application Move to your grade level groupMove to your grade level group Read the selected student Read the selected student

expectationexpectation Create at least two appropriate Create at least two appropriate

questions that target the level of questions that target the level of thinking required by the TEKSthinking required by the TEKS

Create one lower level questionCreate one lower level question Create one question at a higher level Create one question at a higher level

than the TEKS SEthan the TEKS SE

Processing InfoProcessing InfoEach grade level will share Each grade level will share

their worktheir work

Social Studies Social Studies Item Review ProcessItem Review Process

Paper work:Paper work: Read and fill out Read and fill out Oath of Test Oath of Test

Security Form Security Form and return it to the and return it to the coordinator.coordinator.

Item Review:Item Review: Make any corrections or Make any corrections or

comments directly on the test.comments directly on the test.1.1. Check to see if the item is properly Check to see if the item is properly

aligned to COL for the units covered aligned to COL for the units covered during this benchmark period. during this benchmark period. (Pacing Guide)(Pacing Guide)

2.2. Check for content alignmentCheck for content alignment

Item Review:Item Review:3.3. Check to make sure that the item Check to make sure that the item

TEKS SE is on the BlueprintTEKS SE is on the Blueprint4.4. Check to see if the item is aligned Check to see if the item is aligned

to the TEKS SE.to the TEKS SE.a.a. Check for content alignmentCheck for content alignment

i.i. If the TEKS SE is focused on content then If the TEKS SE is focused on content then be sure that the item tests that particular be sure that the item tests that particular content.content.

Item Review:Item Review:

4.4. Check to see if the item is aligned to the Check to see if the item is aligned to the TEKS SE.TEKS SE.

b)b) Check for alignment to the “verb” in the SE.Check for alignment to the “verb” in the SE.i.i. The item can test at a higher level than the verb, The item can test at a higher level than the verb,

but not a lower level.but not a lower level.c)c) Check for skill alignmentCheck for skill alignment

i.i. If the TEKS SE is asking for a specific skill to be used If the TEKS SE is asking for a specific skill to be used make sure that that skill is what is being tested.make sure that that skill is what is being tested.

ii.ii. Some items will test both skills and content as is the Some items will test both skills and content as is the case on TAKS. Again, check for alignment to the SE.case on TAKS. Again, check for alignment to the SE.


5.5. Check for correct answer on key. Check for correct answer on key. (Last Page)(Last Page)

6.6. Check for “Level” on the answer Check for “Level” on the answer key. (If you have question please key. (If you have question please let me know.let me know.

7.7. Check for typos and grammar Check for typos and grammar problems.problems.


8.8. Check for “unreadable” graphics.Check for “unreadable” graphics.9.9. Check to make sure that graphics Check to make sure that graphics

and reading selections are at the and reading selections are at the appropriate level.appropriate level.

10.10. Note: Formatting for all tests will Note: Formatting for all tests will follow TAKS social studies based on follow TAKS social studies based on the 8th grade test.the 8th grade test.

Four Keys to QualityFour Keys to Quality Student Assessment Student Assessment

Key 1Arise from clear and appropriate learning

targets

Key 2Serve focused and

appropriate purposes

Key 3Based on clearly

defined performance objective/descriptor

Key 4Are clear, concise and eliminate sources of bias and distortion


Key 1Arise from clear and appropriate learning

targets

The TEKS: Clarification needed where it is not present.

6.4D identify and explain the geographic factors responsible for the location of economic activities in places and regions.

Include location of natural resources, trade routes/transportation, physical features, etc.

Student Expectation Clarification


Key 2Serve focused and

appropriate purposes

The test item assesses information that is important for the student to understand: TAKS test, local benchmark, teacher benchmark, etc.




What does the TEK actually ask the student to do?




History. The student understands traditional historical points of reference in US history through 1877. The student is expected to

A. Identify major eras in US history through 1877 and describe their defining characteristics:

Given a list of events, people, or period by their distinguishing characteristics the student will sequence or choose the correct sequence


Key 4Eliminate

sources of bias and distortion

Evaluating Tests-OverviewEvaluating Tests-Overview1.1. Do the items cover a range of Do the items cover a range of

understanding for each TEKS SE?understanding for each TEKS SE?

2. Do the items reflect the weight given to 2. Do the items reflect the weight given to various TEKS? (TAKS/Benchmark various TEKS? (TAKS/Benchmark Blueprint)Blueprint)

Evaluating Tests-OverviewEvaluating Tests-OverviewTAKS BLUEPRINT FOR GRADE 8 SOCIAL STUDIES

TAKS

Objectives Number of Items

Objective 1—History ……………………………13

Objective 2—Geography ………………………...6

Objective 3—Economic and Social Influences .9

Objective 4—Political Influences ……………….12

Objective 5—Social Studies Skills ………………8

Total number of items …………………………48

TAKS Objectives April 21State TAKS Test

State TAKSBlueprint (Grade 8.

November 9-15 BENCHMARK

March 2-8 BENCHMARK

Objective 1 – History

13 questions *All T3

7 questions 7.1C, 7.2B, 7.2C, 7.2D, 7.2F

5 questions 7.1C, 7.4A, 7.4B, 7.5B, 7.13B, 7.19B

Objective 2 – Geography

6 questions*All T3

10 questions 7.8B 7.9A, 7.9B, 7.9C, 7.10A, 7.10B,

5 questions 7.8A, 7.9A, 7.9B, 7.10A, 7.10B,

Objective 3 – Economic and Social Influences

9 questions*All T3

0 questions

1 question 7.5B

Objective 4 – Political Influences

12 questions*All T3

0 questions

6 questions 7.5A, 7.14A, 7.14B, 7.16A

Objective 5 – Social Studies Skills

8 questions*All T3

8 questions 7.21A, 7.21B, 7.21C, 7.21H

8 questions 7.17A, 7.21A, 7.21B, 7.21C, 7.21E, 7.21H

Total number of items

48 questions 25 25

Evaluating Tests-OverviewEvaluating Tests-Overview3.3. Are the items easy to understand?Are the items easy to understand?

4.4. Do test items ask for more than recall?Do test items ask for more than recall?

5. Do the items contain superfluous 5. Do the items contain superfluous information or art?information or art?

6. Do items provide test-wise students 6. Do items provide test-wise students with an advantage?with an advantage?

Opposites as answer choicesOpposites as answer choices One option much longer, shorter, or One option much longer, shorter, or

more complex than othersmore complex than others Clues to answer in the item stemClues to answer in the item stem

Evaluating Tests-OverviewEvaluating Tests-Overview

7. Do items favor students who are 7. Do items favor students who are good readers?good readers?

Negatives in stems or optionsNegatives in stems or options Answers including “all of the above” Answers including “all of the above”

or “none of the above”or “none of the above” Specific determiners such as “all”Specific determiners such as “all”

Evaluating Tests-OverviewEvaluating Tests-Overview

The fatal pedagogical error…The fatal pedagogical error… to throw answers like stones to throw answers like stones

at the heads of those who at the heads of those who have not yet asked the have not yet asked the

questions.questions.

Paul TillichPaul Tillich

GraphicsGraphics

TAKS requires the use of more TAKS requires the use of more graphics!graphics!Is the graphic that is used aligned to the Is the graphic that is used aligned to the

performance descriptor?performance descriptor?Is the graphic clear (readable.?Is the graphic clear (readable.?Is the graphic at the appropriate level?Is the graphic at the appropriate level?Do you use a variety of graphics?Do you use a variety of graphics?

Maps--http://www.nationalgeographic.com/Maps--http://www.nationalgeographic.com/siteindex/index.html#mapssiteindex/index.html#maps

Multiple Choice QuestionsMultiple Choice QuestionsMultiple choice items consist of a direct question Multiple choice items consist of a direct question

or incomplete statement called the stem. or incomplete statement called the stem. This is followed by a list of choices or the This is followed by a list of choices or the

options. options. The options consist of the correct option (the The options consist of the correct option (the

answer. and the distracters. answer. and the distracters. The distracters are The distracters are plausible competitorsplausible competitors of the of the

answer. answer. The student must select the appropriate The student must select the appropriate

response to the stem using the options given.response to the stem using the options given.

Multiple Choice QuestionsMultiple Choice QuestionsI. Be Clear and Concise.

You can't overstate the importance of clarity. Think of this as a rule, not a guideline.

BE CLEAR.

Multiple Choice QuestionsMultiple Choice QuestionsBe Clear and Concise. (Cont.)

Without sacrificing clarity, be as concise as possible. Keep a sense of focus.

Remember that your purpose is to measure your students' knowledge, reasoning, and ability.

The idea is to discriminate levels of understanding, not to trap the unwary.

Multiple Choice QuestionsMultiple Choice QuestionsI.I. Be Clear and Concise Be Clear and Concise (Continued)(Continued)

Simply stated: It is not sufficient to write a question that can be understood.

To paraphrase the advice of Gen. Douglas MacArthur to the Corps of Cadets at West Point:

Write questions that cannot be misunderstood, not merely questions that can be understood.

Multiple Choice QuestionsMultiple Choice QuestionsII. Writing Multiple Choice Stems II. Writing Multiple Choice Stems

A. The multiple choice stem needs to A. The multiple choice stem needs to present one problem to the student present one problem to the student before the options are considered. before the options are considered.

Multiple Choice QuestionsMultiple Choice Questions

Example of a faulty item: Example of a faulty item: The Spanish Civil WarThe Spanish Civil WarA. broke out in June 1931 and lasted until April A. broke out in June 1931 and lasted until April

1936.1936.B. was fought between the upper class and the B. was fought between the upper class and the

working class.working class.C. was caused by the collapse of Rivera's military C. was caused by the collapse of Rivera's military

dictatorship.dictatorship.D. was a religious war to convert the country to D. was a religious war to convert the country to

Catholicism.Catholicism.


Example of a sound item:Example of a sound item:Which statement describes an immediate effect of Which statement describes an immediate effect of

the Spanish Civil War?the Spanish Civil War?A. Spain was divided into two ideological groups.A. Spain was divided into two ideological groups.B. The government banned the Catholic church.B. The government banned the Catholic church.C. Workers gained political power in the C. Workers gained political power in the

government.government.D. Spain adopted a democratic form of D. Spain adopted a democratic form of

government. government.

Multiple Choice QuestionsMultiple Choice QuestionsRevise the following faulty item based upon the Revise the following faulty item based upon the

above guidelines.above guidelines.

The Supreme CourtThe Supreme CourtA. was established by the Judiciary Act of 1789.A. was established by the Judiciary Act of 1789.B. first contained a Chief Justice and eight Associate B. first contained a Chief Justice and eight Associate

Justices.Justices.C. was first called to assemble on Feb. 1, 1790.C. was first called to assemble on Feb. 1, 1790. D. as an organization is detailed in Article III of the D. as an organization is detailed in Article III of the

US Constitution US Constitution


B. Avoid writing stems that contain extra B. Avoid writing stems that contain extra information such as an introduction to the information such as an introduction to the question. The stem should only include the question. The stem should only include the question or the incomplete statement. question or the incomplete statement.


Example of a faulty item: Example of a faulty item: The "new kid on the retirement block" is called the The "new kid on the retirement block" is called the

Roth IRA. It was introduced as a vehicle for Roth IRA. It was introduced as a vehicle for retirement investments in the 1997 Taxpayer retirement investments in the 1997 Taxpayer Relief Act. What is the special feature of a Roth Relief Act. What is the special feature of a Roth IRA?IRA?

A. There is no dollar limit on contributions.A. There is no dollar limit on contributions.B. The withdrawals are tax-free.B. The withdrawals are tax-free.C. There is no penalty for early withdrawal.C. There is no penalty for early withdrawal.D. The contributions are tax-deductible.D. The contributions are tax-deductible.


Example of a sound item: Example of a sound item: What is the special feature of a Roth IRA?What is the special feature of a Roth IRA?A. There is no dollar limit on contributions.A. There is no dollar limit on contributions.B. The withdrawals are tax-free.B. The withdrawals are tax-free.C. There is no penalty for early withdrawal.C. There is no penalty for early withdrawal.D. The contributions are tax-deductible.D. The contributions are tax-deductible.

Multiple Choice QuestionsMultiple Choice QuestionsRevise the following faulty item based upon the above Revise the following faulty item based upon the above

guidelines.guidelines. The National Park System now manages 379 parks throughout the The National Park System now manages 379 parks throughout the

United States. Which of the following is a correct fact about the United States. Which of the following is a correct fact about the history of the National Park Service?history of the National Park Service?

A. The United States Department of the Interior was responsible for A. The United States Department of the Interior was responsible for parks and monuments before the National Park Service.parks and monuments before the National Park Service.

B. Mesa Verde was the world's first area to be designated as a national B. Mesa Verde was the world's first area to be designated as a national park.park.

C. President Franklin Roosevelt approved legislation to create the C. President Franklin Roosevelt approved legislation to create the National Park Service.National Park Service.

D. George Marks is credited with the idea of establishing a national park D. George Marks is credited with the idea of establishing a national park serviceservice


C. Include as much information in the stem C. Include as much information in the stem and as little information in the options as and as little information in the options as possible.possible.

Multiple Choice QuestionsMultiple Choice QuestionsExample of a faulty item: Example of a faulty item: Hawaii:Hawaii:A. is a group of 30 volcanic islands in the Pacific Ocean A. is a group of 30 volcanic islands in the Pacific Ocean

that contains one of the largest active volcanoes in the that contains one of the largest active volcanoes in the world.world.

B. contains the volcano called Mauna Loa that is located B. contains the volcano called Mauna Loa that is located on the Hawaiian island called Kahoolawe.on the Hawaiian island called Kahoolawe.

C. entered the Union in the year of 1959 along with Alaska C. entered the Union in the year of 1959 along with Alaska and these were the last two states to enter the Union.and these were the last two states to enter the Union.

D. has a state flower that is the gardenia and its state flag D. has a state flower that is the gardenia and its state flag has an orange blossom drawn on it.has an orange blossom drawn on it.


Example of a sound item: Example of a sound item: Which state was the last one to be admitted Which state was the last one to be admitted

to the United States?to the United States?A. AlaskaA. AlaskaB. New MexicoB. New MexicoC. HawaiiC. HawaiiD. ArizonaD. Arizona


Revise the following faulty item based upon the above Revise the following faulty item based upon the above guidelines.guidelines.

Utah:Utah:A. has largely been shaped by the ethnic and cultural A. has largely been shaped by the ethnic and cultural

influence of Mormons.influence of Mormons.B. has largely been shaped by the ethnic and cultural B. has largely been shaped by the ethnic and cultural

influence of Hispanics.influence of Hispanics.C. has largely been shaped by the ethnic and cultural C. has largely been shaped by the ethnic and cultural

influence of Native Americans.influence of Native Americans.D. has largely been shaped by the ethnic and cultural D. has largely been shaped by the ethnic and cultural

influence of Nordic peoples.influence of Nordic peoples.


D. Restrict the use of negative terms in the D. Restrict the use of negative terms in the stem. stem.


Example of a faulty item: Example of a faulty item: Which of the following reasons is not a main Which of the following reasons is not a main

cause of the Civil War?cause of the Civil War?A. Political differencesA. Political differencesB. Economic differencesB. Economic differencesC. Cultural differencesC. Cultural differencesD. Landscape differencesD. Landscape differences


Example of a sound item: Example of a sound item: Which of the following reasons was a main Which of the following reasons was a main

cause of the Civil War?cause of the Civil War?A. Political differencesA. Political differencesB. Ethnic differencesB. Ethnic differencesC. Language differencesC. Language differencesD. Landscape differencesD. Landscape differences


Revise the following faulty item based upon the Revise the following faulty item based upon the above guidelines.above guidelines.

What is a health and welfare program that is not What is a health and welfare program that is not administered by state governments?administered by state governments?

A. MedicaidA. MedicaidB. Temporary Assistance to Needy FamiliesB. Temporary Assistance to Needy FamiliesC. Social SecurityC. Social SecurityD. Unemployment CompensationD. Unemployment Compensation


E. Construct stems that require the E. Construct stems that require the selection of the best answer when each of selection of the best answer when each of the options contain elements of the options contain elements of correctness.correctness.


Example of a faulty item: Example of a faulty item: Which of the themes of geography listed Which of the themes of geography listed

below relates to cultural diffusion?below relates to cultural diffusion?A. RegionA. RegionB. MovementB. MovementC. Human/Environment InteractionC. Human/Environment InteractionD. LocationD. Location

Multiple Choice QuestionsMultiple Choice QuestionsExample of a sound item: Example of a sound item: Which of the themes of geography listed Which of the themes of geography listed

below relates most closely to the below relates most closely to the concept of cultural diffusion?concept of cultural diffusion?

A. RegionA. RegionB. MovementB. MovementC. Human/Environment InteractionC. Human/Environment InteractionD. LocationD. Location



What is one of the roles delegated specifically to What is one of the roles delegated specifically to the office of the President of the United States?the office of the President of the United States?

A. answering 911 emergency callsA. answering 911 emergency callsB. being the pilot of Air Force OneB. being the pilot of Air Force OneC. inviting celebrities to the White House C. inviting celebrities to the White House D. nominate cabinet membersD. nominate cabinet members


III. Use the Active Voice.

Everybody processes information much more easily in the active voice. Various research studies have confirmed this. (Olsen and Tichy.


III. Use the Active Voice.

In sentences written in active voice, the subject performs the action expressed in the verb; the subject acts.

The dog bit the boy. The boy was bitten by the dog.

Historians researched the project before presenting it. The project was researched by Historians before presenting it.

Teachers will write good items because they care. Good items will be written by teachers, because they care.


IV. Focus each test question on a specific IV. Focus each test question on a specific objective of the instructional unit.objective of the instructional unit.


7.5B The student will analyze the political, 7.5B The student will analyze the political, economic, and social effects of the Civil War and economic, and social effects of the Civil War and Reconstruction in Texas.Reconstruction in Texas.

Which of the following was a governor of Texas Which of the following was a governor of Texas during the Civil War?during the Civil War?

A. Sam HoustonA. Sam HoustonB. Francis LubbockB. Francis LubbockC. John A. WhartonC. John A. WhartonD. Jefferson DavisD. Jefferson Davis


Example of a sound item: Example of a sound item: Which of the following was a political effect of the Which of the following was a political effect of the

Civil War on Texas?Civil War on Texas?A. Lack of a civil or military force to keep orderA. Lack of a civil or military force to keep orderB. Increased amount of money flowing to the stateB. Increased amount of money flowing to the stateC. Lack of leadership by the governorC. Lack of leadership by the governorD. Lack of leadership by the congressD. Lack of leadership by the congress


V. Answer ChoicesV. Answer ChoicesA. Each distracter should have about the A. Each distracter should have about the same number of words as the correct same number of words as the correct option.option.


Example of a faulty item: Example of a faulty item: Abraham Lincoln is famous for:Abraham Lincoln is famous for:A. the years he served in the Illinois legislature.A. the years he served in the Illinois legislature.B. issuing the Emancipation Proclamation on B. issuing the Emancipation Proclamation on

January 1, 1863 stating that all persons held as January 1, 1863 stating that all persons held as slaves in the United States shall be free and the slaves in the United States shall be free and the executive government will recognize all such executive government will recognize all such persons as free.persons as free.

C. his law practice in Springfield, Illinois.C. his law practice in Springfield, Illinois.D. serving as captain in the Black Hawk War.D. serving as captain in the Black Hawk War.


Example of a sound item: Example of a sound item: Abraham Lincoln is most famous for:Abraham Lincoln is most famous for:A. the years he served in the Illinois legislature.A. the years he served in the Illinois legislature.B. issuing the Emancipation Proclamation.B. issuing the Emancipation Proclamation.C. his law practice in Springfield, Illinois.C. his law practice in Springfield, Illinois.D. serving as captain in the Black Hawk War. D. serving as captain in the Black Hawk War.



Example of a faulty item: Example of a faulty item: William Barret Travis is known William Barret Travis is known A. for abandoning his wifeA. for abandoning his wifeB. for buying land from Stephen F. AustinB. for buying land from Stephen F. Austin C. as the colonel who was born in Saluda Co., SC, C. as the colonel who was born in Saluda Co., SC,

and became the Texas commander at the Battle and became the Texas commander at the Battle of the Alamo of the Alamo

D. for being a newspaper publisher.D. for being a newspaper publisher.


B. Does the item have one clearly correct B. Does the item have one clearly correct answer?answer?

Abraham Lincoln was president of the United Abraham Lincoln was president of the United States during:States during:

A.A. A time of strifeA time of strifeB.B. The Revolutionary WarThe Revolutionary WarC.C. A time of expansionA time of expansionD.D. The Civil WarThe Civil War


C. Make all distracters plausible.C. Make all distracters plausible.


Example of a faulty item: Example of a faulty item: Abraham Lincoln is famous for:Abraham Lincoln is famous for:A. being the first man to walk on the moon.A. being the first man to walk on the moon.B. issuing the Emancipation Proclamation.B. issuing the Emancipation Proclamation.C. his role in the musical Les Miserables.C. his role in the musical Les Miserables.D. inventing the electric light bulb.D. inventing the electric light bulb.


Abraham Lincoln is most famous for:Abraham Lincoln is most famous for:A. the years he served in the Illinois A. the years he served in the Illinois

legislature.legislature.B. issuing the Emancipation Proclamation.B. issuing the Emancipation Proclamation.C. his law practice in Springfield, Illinois.C. his law practice in Springfield, Illinois.D. serving as captain in the Black Hawk D. serving as captain in the Black Hawk

War.War.


D. Do not repeat wording or common D. Do not repeat wording or common elements from the stem in the correct elements from the stem in the correct option.option.


Example of a faulty item: Example of a faulty item: Which of the following occurred during the Which of the following occurred during the

Children's Crusade?Children's Crusade?A. Crusaders loot Constantinople.A. Crusaders loot Constantinople.B. Crusaders take Antioch, then Jerusalem.B. Crusaders take Antioch, then Jerusalem.C. Crusaders win coast from Tyre to Jaffa.C. Crusaders win coast from Tyre to Jaffa.D. Thousands of innocent children march to death D. Thousands of innocent children march to death

and slavery in North Africaand slavery in North Africa


Example of a sound item: Example of a sound item: Which of the following events occurred Which of the following events occurred

during the Fourth Crusade?during the Fourth Crusade?A. Constantinople was looted.A. Constantinople was looted.B. Antioch was defeated, then Jerusalem.B. Antioch was defeated, then Jerusalem.C. The entire coast from Tyre to Jaffa was C. The entire coast from Tyre to Jaffa was

defeated.defeated.D. Anatolia defeated the invaders.D. Anatolia defeated the invaders.



Example of a faulty item: Example of a faulty item: Who designed the Eiffel Tower?Who designed the Eiffel Tower?A. Gustave EiffelA. Gustave EiffelB. Adolf LoosB. Adolf LoosC. Frei OttoC. Frei OttoD. Eero SaarinenD. Eero Saarinen


E. To increase item difficulty include "none E. To increase item difficulty include "none of the above" as a final option.of the above" as a final option.


Example of a sound item: Example of a sound item: Which of the following relationships describes the Which of the following relationships describes the

relationship between Spanish and Anglo relationship between Spanish and Anglo settlements of Texas?settlements of Texas?

A. friendshipA. friendshipB. cooperationB. cooperationC. hatredC. hatredD. None of the aboveD. None of the above


F. When a list of choices is included in each F. When a list of choices is included in each option increase the similarity of all the option increase the similarity of all the answer choices to increase the difficulty of answer choices to increase the difficulty of the test. the test.


Example of a faulty item: Example of a faulty item: Who were the leaders of the Axis Powers Who were the leaders of the Axis Powers

during World War II?during World War II?A. Grant, Adams, CarterA. Grant, Adams, CarterB. Michelangelo, Cortes, EinsteinB. Michelangelo, Cortes, EinsteinC. Curie, Columbus, MacArthurC. Curie, Columbus, MacArthurD. Hirohito, Hitler, MussoliniD. Hirohito, Hitler, Mussolini


Example of a sound item: Example of a sound item: Who were the leaders of the Axis Powers Who were the leaders of the Axis Powers

during World War II?during World War II?A. Hirohito, Hitler, StalinA. Hirohito, Hitler, StalinB. Hitler, Stalin, MussoliniB. Hitler, Stalin, MussoliniC. Hirohito, Mussolini, StalinC. Hirohito, Mussolini, StalinD. Hirohito, Hitler, MussoliniD. Hirohito, Hitler, Mussolini



Example of a faulty item: Example of a faulty item: The atomic bombs in World War II were dropped The atomic bombs in World War II were dropped

on these two cities:on these two cities:A. Washington and FukuokaA. Washington and FukuokaB. Nagoya and CairoB. Nagoya and CairoC. Hiroshima and NagasakiC. Hiroshima and NagasakiD. Kyoto and LondonD. Kyoto and London


VI. Tap Higher Level Cognitive Domains

Rote memorization of facts, laws, and definitions has its place in the overall scheme. However, at least 80-90% of the test should be devoted to higher levels of cognition.


A. Watch Difficulty Levels

The ideal question will be answered correctly by 60-65% of the tested population. This level of difficulty maximizes discrimination on exams.


B. Avoid word choices that are above the B. Avoid word choices that are above the

student's vocabulary level.student's vocabulary level.


Example of a faulty item: Example of a faulty item: If a numismatist was alive 4000 years ago, If a numismatist was alive 4000 years ago,

what would he be looking for?what would he be looking for?A. WampumA. WampumB. Tally sticksB. Tally sticksC. Cowrie shellsC. Cowrie shellsD. ManillaD. Manilla


Example of a sound item: Example of a sound item: What is the form of money that was used in What is the form of money that was used in

ancient Egypt?ancient Egypt?A. WampumA. WampumB. Tally sticksB. Tally sticksC. Cowrie shellsC. Cowrie shellsD. ManillaD. Manilla



Phrenologist suggested which of the following?Phrenologist suggested which of the following?A. horses were faster than trainsA. horses were faster than trainsB. whites were more intelligent than blacksB. whites were more intelligent than blacksC. space travel could not happenC. space travel could not happenD. the theory of evolutionD. the theory of evolution


VII. Avoid constructing negative options that VII. Avoid constructing negative options that follow a negative stem. follow a negative stem.


Example of a faulty item: Example of a faulty item: Which of the following characteristics is not Which of the following characteristics is not

among the attributes of an among the attributes of an Human/Environment Interaction?Human/Environment Interaction?

A. Lack of buildings.A. Lack of buildings.B. Lack of pollution.B. Lack of pollution.C. Lack of rivers.C. Lack of rivers.D. Lack of recycling.D. Lack of recycling.


Example of a sound item: Example of a sound item: Which of the following characteristics is not Which of the following characteristics is not

among the attributes of an among the attributes of an Human/Environment Interaction?Human/Environment Interaction?

A. Buildings.A. Buildings.B. Pollution.B. Pollution.C. Rivers.C. Rivers.D. Recycling. D. Recycling.



Example of a faulty item: Example of a faulty item: What is not a general characteristic of wars?What is not a general characteristic of wars?A. lack of shortages A. lack of shortages B. lack of diseaseB. lack of diseaseC. not poor economic times for familiesC. not poor economic times for familiesD. not loss of rightsD. not loss of rights


VIII.VIII. Clear and ConciseClear and ConciseA. Make sure the stem is grammatically A. Make sure the stem is grammatically consistent with all of the options. consistent with all of the options.


Example of a faulty item: Example of a faulty item: Movement is an example of an:Movement is an example of an:A. application.A. application.B. theme.B. theme.C. diffusion.C. diffusion.D. location.D. location.


Example of a sound item: Example of a sound item: Movement is an example of-Movement is an example of-A. applicationA. applicationB. themeB. themeC. diffusionC. diffusionD. locationD. location



Example of a faulty item: Example of a faulty item: An example of diffusion is an-An example of diffusion is an-A. Jesuit church opening in SpainA. Jesuit church opening in SpainB. McDonalds opening in DetroitB. McDonalds opening in DetroitC. Hispanic song being played in TexasC. Hispanic song being played in TexasD. Tex-Mex meal being eaten in the CongoD. Tex-Mex meal being eaten in the Congo


B. Carefully proofread items.B. Carefully proofread items.


Example of a faulty item: Example of a faulty item: What is Hernan Cortes famous for?What is Hernan Cortes famous for?A. He captured Mexico for Spain.A. He captured Mexico for Spain.B. He explored Centrol America.B. He explored Centrol America.C. He explores the Amazon River.C. He explores the Amazon River.D. He conqered the Incas for Spain.D. He conqered the Incas for Spain.


Example of a sound item: Example of a sound item: What is Hernan Cortes famous for?What is Hernan Cortes famous for?A. He captured Mexico for Spain.A. He captured Mexico for Spain.B. He explored Central America.B. He explored Central America.C. He explored the Amazon River.C. He explored the Amazon River.D. He conquered the Incas for Spain.D. He conquered the Incas for Spain.



Example of a faulty item: Example of a faulty item: A word that modifies a noun is called: A word that modifies a noun is called: A. an adverdA. an adverdB. a conjunctionB. a conjunctionC. a verrbC. a verrbD. an adjectiveD. an adjective


C. Avoid using overlapping distracters. C. Avoid using overlapping distracters.


Example of a faulty item: Example of a faulty item: A person in the first tax bracket makes less A person in the first tax bracket makes less

than:than:A. $125,000A. $125,000B. $100,000B. $100,000C. $75,000 C. $75,000 D. $50,000D. $50,000


Example of a sound item: Example of a sound item: A person in the third tax bracket makes:A person in the third tax bracket makes:A. $14,651-$24,650A. $14,651-$24,650B. $24,651-$59,750B. $24,651-$59,750C. $59,751-$124,650 C. $59,751-$124,650 D. $124,651-$271,050D. $124,651-$271,050



Example of a faulty item: Example of a faulty item: Midnight occurs at what time of day?Midnight occurs at what time of day?A. eveningA. eveningB. nightB. nightC. 12 PMC. 12 PMD. 12 AMD. 12 AM


D. Avoid the use of indefinite terms such as, D. Avoid the use of indefinite terms such as, “would”, “could”, “usually” and “generally” “would”, “could”, “usually” and “generally” in the options and distracters. in the options and distracters.



Example of a faulty item: Example of a faulty item: The length of a senator's single term is:The length of a senator's single term is:A. 2 yearsA. 2 yearsB. 3 yearsB. 3 yearsC. Usually 4 yearsC. Usually 4 yearsD. Generally 6 yearsD. Generally 6 years


E. Avoid the use of absolute terms such as E. Avoid the use of absolute terms such as “never” and “always” in the options and “never” and “always” in the options and distracters. distracters.


Example of a faulty item: Example of a faulty item: Which of the following is a sound investment principle?Which of the following is a sound investment principle?A. Keep emergency money funds in highly liquid A. Keep emergency money funds in highly liquid

assets.assets.B. Mutual funds are always the safest investment B. Mutual funds are always the safest investment

vehicle.vehicle.C. Social security is a risk free income source for C. Social security is a risk free income source for

retirement.retirement.D. Pay only the minimum balance on the credit card D. Pay only the minimum balance on the credit card

each month.each month.

Multiple Choice QuestionsMultiple Choice QuestionsExample of a good item: Example of a good item: Which of the following is a sound investment Which of the following is a sound investment

principle?principle?A. Social security is a risk free income source for A. Social security is a risk free income source for

retirement.retirement.B. Mutual funds are the most growth oriented B. Mutual funds are the most growth oriented

investment vehicle.investment vehicle.C. Keep emergency money funds in highly liquid C. Keep emergency money funds in highly liquid

assets.assets.D. Pay only the minimum balance on the credit card D. Pay only the minimum balance on the credit card

each month.each month.



Example of a faulty item: Example of a faulty item: An earthquake has the following property:An earthquake has the following property:A. it always causes tidal waves.A. it always causes tidal waves.B. it always cracks the earth.B. it always cracks the earth.C. is caused by plate tectonics.C. is caused by plate tectonics.D. is never explosive.D. is never explosive.


F. Avoid using "all of the above." F. Avoid using "all of the above."


Example of a faulty item: Example of a faulty item: Which of the following are examples of Which of the following are examples of

physical features?physical features?A. MountainsA. MountainsB. Bays.B. Bays.C. GlaciersC. GlaciersD. All of the aboveD. All of the above


Example of a sound item: Example of a sound item: Which of the following is an example of a Which of the following is an example of a

physical feature?physical feature?A. Pollution.A. Pollution.B. Bay.B. Bay.C. Culture.C. Culture.D. Location.D. Location.



Example of a faulty item: Example of a faulty item: An event leading to the American Revolution wasAn event leading to the American Revolution wasA. the Stamp ActA. the Stamp ActB. the Sugar ActB. the Sugar ActC. the Tea Act C. the Tea Act D. all of the aboveD. all of the above

Contact InformationContact Information John L. DavidJohn L. David 972-466-6180972-466-6180

[email protected]@cfbisd.edu

Jennifer CarlisleJennifer Carlisle 972-466-6138972-466-6138

[email protected]@cfbisd.edu

mailto:[email protected]

mailto:[email protected]

Documents

Assessing Student Knowledge: Going to the Next Level