WELCOME TO EXAM REVIEW - ASQ Tucson Old Pueblo … › download › presentations › May...Committee needs to ensure that each item: • tests candidates’ knowledge of the subject

WELCOME TO EXAM REVIEW

Thank you for your cooperation.

No cell phones, laptops or palmtop computers are to be

used during the meeting.

Meeting Protocols

Workshop Recruiting The Recruiter for the Program…

• Receives lists of certified ASQ individuals from HQ twice a year

• Chooses a stratified sample for recruiting new volunteers to ensure diversity

• Reviews past volunteer lists to select veteran participants who can help new volunteers

• Supports diverse participation for each workshop with respect to geography, expertise & experience, industry sector, and company size

Workshop RecruitingHow was I selected

as “one of the chosen”?• #1: Workshop diversity• Luck / Chance• Availability

How do I remain a “chosen one”?• Be a team player• Complete the back of the Feedback form and

tell us you would like to participate again

OBJECTIVE

To produce a final exam that meets test specifications

and accepted assessment standards

Presenter

Presentation Notes

1- Our objective today is to…. 2- ….to do this, each item….

TASKS TO GET THERE

Committee needs to ensure that each item:• tests candidates’ knowledge of the

subject matter• has one and only one correct answer

(Key)• does not cue or overlap other items• is worded clearly and unambiguously

Presenter

Presentation Notes

Changed 03/02/2011 Changed. Marilyn Schmid Sheila Connolly5/11/2009 Not sure "Division Liaison" is the right title here. It sounds like somebody who is pretty far removed from the Cert process. Suggest: The Recruiter for the Program It's less a title than a position description, but for many of our programs the exam chair wears multiple hats.

STEPS IN TEST DEVELOPMENTNew Cert.approved by ASQ BOD

5-year Reviewof Existing Program

Job Analysis Workshop

Job Analysis Survey

Test SpecificationsWorkshop

EnoughItems in

Pool?

Item-Writing Workshop

No

Item-Review Workshop

Exam-Review Workshop

ReviewCamera-Ready

Version

Administer Exam Item Analysis & Scoring Changes

Score Exams andNotify Candidates

Cut-Score Study

New BOK?

Yes

No

You AreHere!

Presenter

Presentation Notes

This process is what gives validity to our exams, so that we are able to show that the process is very thorough and that many people are involved to do our best to create items go into an exam that tests knowledge and will offer the lowest chance for candidates’ misunderstanding of an item. Item writing results in RAW items that are then reviewed at Item Review

ASSESSMENT CONCERNS

Job Analysis Test Specifications

Body of Knowledge (BOK) Test Question Classification

Reference Text & Rationale

Evidence that the test is measuring what it says it will measure. Processes that provide test validity evidence include:

VALIDITY

Presenter

Presentation Notes

1-This evidence is provided by 2- Items must have a ref. and rationale 3- items must be classified according to what they are really testing 4- which links to the BOK 5- The BOK with sub text is created from the 6- test specifications (bare bones) 7- which were created from the Job Analysis which was done every 5 years. Basically these are the points we covered on the process flow chart and all these steps give the exam validity Because all of these parts of flow are linked to one another, the test will be valid * Easiest to look at it in reverse order, Reference text linked to Classification, Item/Question stems from BOK, BOK stems from Test Specs, and Test Specs stem from Job Analysis Person cannot argue that an item does not apply to his/her industry because we can trace it all the way back to the Job Analysis, which increases the exams validity.

ASSESSMENT CONCERNS

Is the behavior being measured consistently?

Consistency is based on• Test Length• Consistency in Test Administration

• Clarity of Questions• Appropriateness of Questions

RELIABILITY

Presenter

Presentation Notes

Reliability also comes from consistency. 1- number of questions 2- over time; at test sites The above components were established with Test Spec meeting, cannot be changed at this meeting! _________________________________________________ Why we are here today. You took the test and we are discussing problem items. Committee member responsibilities: 3-unambiguous wording or presentation 4- if it doesn’t fit into the BOK its deleted

EQUATERS

Equating = Fairness to Candidates

Equaters defined: A set of 25-30 items selected from a previous administration of the exam and placed in the newly created exam in same position as in the previous exam. This set of items represents a “Mini-test” in terms of BOK coverage and difficulty.

Presenter

Presentation Notes

In the world of assessment there has to be proven reliability of the scoring process and this is achieved through the use of equaters. It is very important for us to be able to have some measure of how the two different groups performed and whether or not one exam was more difficult than another.

9. Which of the following items is NOT part of inspection planning?

(A) Characteristics to inspect(B) Sampling plans(C) Cost of inspection(D) Calibration status

10. The extent to which an instrument replicates its result when measurements are taken repeatedly on the same unit is called

(A) real bias(B) precision(C) accuracy(D) true value

11. Inspection accuracy increases with

(A) reductions in defect rates(B) repeated inspections(C) additional product complexity(D) order of sample selection

12. Which of the following tools would be best for measuring projected surface intersections?

(A) Optical comparator(B) Thread micrometer(C) Profilometer(D) Protractor

13. The accuracy of a profilometer is verified with

(A) fingernail comparators(B) surface plates(C) calibration patches(D) optical flats

14. What is the recommended minimum number of subgroups necessary to calculate the limits for a control chart?

(A) 10(B) 15(C) 25(D) 35

15. Which of the following types of micrometers is best for detecting external out-of-roundness on a cylindrical part?

(A) Blade(B) Three-point(C) V-anvil(D) Depth

16. Which of the following tools can only be used to collect attributes data?

(A) Dial indicator(B) Ring gage(C) Micrometer(D) Steel scale

10. In preparation for construction of a cause and effect diagram, it is important to

(A) plot separate charts for each source(B) focus only on what makes things go

wrong(C) record everything people suggest(D) validate possible root causes

9. What is the percent yield for a normally distributed process in which the item length specification is 5.750 ± 0.005, is 5.754, and the standard deviation is 0.002 ?

(A) 19.15%(B) 30.85%(C) 50.00%(D) 69.15%

14. The ability of a material to resist mechanical penetration is a test of

(A) surface finish(B) hardness(C) tensile strength(D) porosity

15. Which of the following measures is a sufficient statistic for the parameter µ ?

(A) Median(B) Mid-range(C) Mean(D) Mode

12. Which of the following describes the Deming method for continuous improvement?

(A) Cost of quality analysis(B) Process map(C) Tree diagram(D) Plan-do-check-act cycle

Equated to JUNE 2014

DECEMBER 2015 EXAM

EQUATERS continued…

100-ITEM TEST WITH 25 EQUATERS

DEC. 2015 ADMIN JUNE 2014 ADMINTotal Exam Total ExamMean score = 80 Mean score = 70

Mini Exam (equaters) Mini Exam (equaters)

Mean score = 20 Mean score = 20

Is there a difference in the tests or a difference in the groups?

Presenter

Presentation Notes

An example in how we use the equaters to compare the 2 groups… This is purely hypothetical! What this tells us………… the mean score of the Mini exam is the same, therefore the two groups of candidates should have similar overall scores. However, the total exam score was 10 points lower in the October admin, which leads us to the conclusion that the EXAM administered in October was more difficult than the exam administered in March. So here’s what you need to know about items that have “EQUATER 2” status: Unless some EXTERNAL EVENT has occurred that impacts the key [and this typically happens ONLY with changes in the regulatory environment], the item must stay AS IS and IN THE SAME PLACE.Think of Equaters as the one variable in an experiment that we can truly control.

BREAKING DOWN THE ITEM

THE STEM:

THE OPTIONS:

Which of the following must be specified by the client in an external systems audit?

(A) An engineering specification

(B) A reference standard

(C) A policy manual

(D) A quality manual

Distractor

Key

Distractor

Distractor

Presenter

Presentation Notes

This is to help you understand the language we use in item development. We refer to the “question” as an item because may have two or three sentences and not be a question at all It also refers to the WHOLE item rather than just the first portion of it. The STEM should be clear, complete and provide adequate direction or information. The candidate should not have to go to the options to figure out what is being asked. Stem should not contain unnecessary words. The OPTIONS should be parallel in structure and should fit logically and grammatically with the stem. The DISTRACTORS should be plausible, have a rationale and should not draw attention to the key. The KEY should not be made obvious in inappropriate ways, must be supported by a reference and have team agreement or affirmation. We refer to the correct answer as the Key because of Negative Stem Qs, the Key is actually the “wrong” answer

COGNITIVE LEVELSIV. AUDIT REPORTING (10 questions)

A. Review and Finalize Audit ResultsIdentify and organize information including evidence compiled during the audit.

B. Written Report Format and Content

1. Audit details

Identify, organize, and summarize audit details to be covered in the

report such as purpose, scope, team members, etc.

2. ComplianceOrganize and summarize evidence to support conclusions about compliance.

(Apply)

(Create)

Expanded BOK

(Apply)

Presenter

Presentation Notes

This is the version that the candidate sees. The cognitive levels tell the candidate how far they will be asked to use their knowledge. Qs can be written up to that level but not past it! For 1A1 (previous slide), you can ask what quality tool shows history or how to use it but you couldn’t ask them to look at how someone used that tool and whether or not it was a good decision to use it. These are steps of mental process that people will go through, not difficulty levels. I.e. Knowledge doesn’t mean easy and analysis doesn’t necessarily mean difficult.

LARGE GROUP REVIEW

FLAGGED ITEMS TO BE REVIEWED ARE THOSE WITH

• 3 or more KEY disagreements

• 3 or more committee comments

Presenter

Presentation Notes

The 3 comments/3 key disagreement rule is based on our experience. Fewer than three is considered common cause variation, it is within the norm, based on people mis keying their answers, seeing issues that might or might not be problems. Having a quarter of the committee comment, or mis key the item indicated special cause variation. “Too many to live with” more than simple error or perception error. Needs to be resolved. 3 key disagreement could indicate good distractors—and therefore the key is fine, or distractors that are TOO good, which means getting it wrong for the wrong reasons and therefore needs to be fixed.

Turn to the first checked item Read item, read comments Discuss Disposition item

(Reset timer)

LARGE GROUP PROCESS

3-minute time limit

• Move on to the next flagged item and repeat the process

Presenter

Presentation Notes

Disposition Ship as is Rework and approve Scrap and replace, automatic parking lot

• Documents we will be working from Expanded version Collated comments

Documents that are nice to have BOK Personal notes or

“At-home” copies

LARGE GROUP PROCESS cont’d.

DISCUSSION POINTERS• Authors of comments should start the

discussion • Remaining committee members can Agree Disagree Ask for further explanation

Add relevant comments Propose immediate revisions (quick-fix) Park It immediately – “We know it’s broke:

Put it in the parking lot & move on!”

Presenter

Presentation Notes

Irrelevant comments/tangents lead to unnecessary changes, e.g., changed the graphic on an item, but didn’t change the content of the item resulting in no key. Assessment reminders may be interjected look at statistics look at classification look at type All cmte members should feel free to suggest revisions to an item It may be obvious that an item needs to go to the Parking lot measures of this are: half of the cmte made comments multiple things wrong with the item severe key disagreement Mostly we dissuade you from spending three minutes violently agreeing that it is a bad item, instead spend three minutes focusing on what is wrong with the item if you are going to fix it. (Same rules apply for Approved as is)

ITEM DISPOSITIONPARKING LOT:Revise: needs further work in small group

Replace - Only for this Exam: cueing or overlap with another item requires a replacement from “9’ers”

Replace - Put on Hold: item worth saving, but needs further work at a future item review meeting

Replace - Delete and Kill Forever: reserved for items the committee never wants to see again

Presenter

Presentation Notes

Show where 9’ers are in the binder.

VOLUNTEERS NEEDED . . .

PARKING LOT ATTENDANT TIME KEEPER

QUESTIONS ? ? ?