Item Writing for CMAS Item Bank
April 23, 2004American Association of Medical Audit Specialists
10th National ConferenceNashville, TN
Marilyn Balcita RN CMASCMAS Council
2Marilyn Balcita CMAS Council
Do you remember your CMAS Exams?
• Four Parts– Professional Standards– Audit Process and Methodology– Audit Skill– Medical Audit Environment
• 200 mostly multiple choice questions• Few matching type questions• Collection of questions from members
?
3Marilyn Balcita CMAS Council
Before and After• Not “officially” linked to Job Practice
Analysis Studies (Role delineation study)
• Item Writing principles not formally followed
• Need more contributions from Subject Matter Experts (SME) representing the broad spectrum of medical audit
• For future CMAS Item Banking • To fulfill NOCA requirements
4Marilyn Balcita CMAS Council
How are questions created ????
Benjamin Bloom's Taxonomy of the Cognitive Domain (1956)
Classification of Educational Goals and Objectives
SynthesisEvaluation
AnalysisApplication
ComprehensionKnowledge
*Adapted from: Bloom, B.S. (Ed.) (1956) Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green.
5Marilyn Balcita CMAS Council
Taxonomy for Learning, Teaching, and Assessing*
A Revision of Bloom's Taxonomy of the Cognitive Domain
RememberUnderstand
ApplyAnalyzeEvaluateCreate
*Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Addison Wesley Longman.
6Marilyn Balcita CMAS Council
Bloom's Taxonomy of the Cognitive Domain
RememberKeywords: list,
define, tell, describe,
identify, show, label, collect, examine, tabulate, quote, name, who, when, where
UnderstandKeywords:
summarize, describe, interpret, contrast, predict, associate, distinguish, estimate, differentiate, discuss, extend
7Marilyn Balcita CMAS Council
Bloom's Taxonomy of the Cognitive Domain
ApplyKeywords: apply,
demonstrate, calculate,
complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment,
discover
AnalyzeKeywords:
analyze, separate, order, explain, connect, classify, arrange, divide, compare, select, explain, infer
8Marilyn Balcita CMAS Council
Bloom's Taxonomy of the Cognitive Domain
EvaluateKeywords: categorize, combine, compile, compose, create, devise, design, explain, generate, modify, organize, plan, rearrange, reconstruct, relate, reorganize, revise, rewrite, summarize, tell, write
CreateKeywords: appraise, compare, conclude, contrast, criticize, critique, defend, describe, discriminate, evaluate, explain, interpret, justify, relate, summarize, support
9Marilyn Balcita CMAS Council
What do we need for the CMAS Exams???
RememberUnderstand
ApplyAnalyzeEvaluateCreate
RecallHave
Enough..
MoreMore
More …
10Marilyn Balcita CMAS Council
CMAS Item Development Process
Define Medical Audit Functions
(JTA)
Develop Med. Audit Test Blueprint
Develop, collect and
validateitems
Item Bank storing
Randomly select items for testing
Refine content
ItemReady
for Testing
11Marilyn Balcita CMAS Council
The Basis of the CMAS Exams
• Job Practice Analysis (Role Delineation Study) Constructed from ….. Job descriptions AAMAS membership information – bulletins, informal
communication, networking Onsite and telephonic interviews SMEs – Medical Auditors, Employers, Consultants,
Academicians Healthcare delivery reports Official books and publications, journals, and other
references
12Marilyn Balcita CMAS Council
Job Task Analysis Survey
• Initial JTA instrument design developed and tested
• Beta testing by AAMAS and non AAMAS volunteers
• Focused on CMAS respondents• One state chapter volunteered• Tabulated and analyzed• Analysis used for the first time in the 2004
CMAS Examination• Demographic information included to gauge
AAMAS general membership make-up
13Marilyn Balcita CMAS Council
• Measures“Criticality: how important or critical
it is" to the overall performance of your job
Frequency: "its frequency" to the overall performance of your job
• Ideally, should be measured every five years
Job Task Analysis Survey(Role Delineation Study)
14Marilyn Balcita CMAS Council
Job Testing Analysis SurveyInitial Result
10%
40%
30%
20%
Professional Standards And Audit Behavior
Medical Audit Process and Methodology
Audit Skill
Medical Audit Environment
15Marilyn Balcita CMAS Council
Medical Audit Process and
Methodology (40%)• Investigate and verify
charges against medical record documentation
• Audit Process, Work Flow, and Audit Findings
• Other Relevant Medical Audit Responsibilities
• Quality Improvement Activities, Education and Training
• Compliance and Special Investigations
• Contracts and Negotiations
• Denial and Appeals Management
• Health Information Management (Medical Records)
• Health Informatics and TechnologySee
Detailed List
16Marilyn Balcita CMAS Council
Medical Audit Environment (30%)
• HIPAA• Medicare/Medicaid
Policies• National Audit
Guidelines• OIG Compliance
Guidance/Work Plan • JCAHO
• Medicare Integrity Program
• GAAP/GAAS Principles • CIA-AICPA Audit Rules• NCQA/Hedis• DHHS Audit Process• US Sentencing Rules• Health Insurance
reimbursement methodology
Application of laws, guidance, standards, guidelines and other accrediting bodies
17Marilyn Balcita CMAS Council
Audit Skill (20%)• Interaction and Communication• Leadership and Managerial • Medical Audit Specific Knowledge and Skill
Set RequiredClinical Judgment and Nursing
ProcessAccounting/FinanceProblem Solving StatisticsNegotiationQuantitative AnalysisResearchProject Management
18Marilyn Balcita CMAS Council
Professional Standards And Audit Behavior
(10%)• Principles of objectivity • Principles of independence • Internal control • Goal setting, strategic planning, and
mission/vision development • Code or standards of conduct • Conflict of Interest• Patient access and confidentiality • Enforcement, expectations and systems of
accountability
19Marilyn Balcita CMAS Council
Why Multiple Choice??PROS
• Quick and easy to score
• Measures all levels of candidates’ ability
• Allows objective scoring
• Can be analyzed for effectiveness
• Covers a broad range of knowledge in limited time
CONS• Challenge in measuring
individual differences in skill
• Hard to develop good items
• Tendency to measure simple recall
20Marilyn Balcita CMAS Council
National Hospital Billing Audit Guidelines require that:
A. Auditors must be certifiedB. Hospitals must charge audit fees in
excess of $275C. Auditors must conduct themselves in a
professional manner, adhere to ethical standards, confidentiality requirements, and objectivity
D. All of the above
STEM
OPTIONS
ITEM
What is the correct answer?What are the distractors? Is this a good item ?
What’s wrong with it?
Reference: National Health Care Billing Audit Guidelines. AAMAS 2003 Directory
21Marilyn Balcita CMAS Council
CMAS Item Writing Guiding Principles
Ask yourself: Can a medical auditor with at least 2 years of basic medical audit experience answer this question correctly?
Measure knowledge, skill, abilities (KSA) that is generally known in the medical audit field = JTA
Present only ONE central idea, and ONE correct answer
Provide 4 options to support the stemAvoid constructing items for the “street-smart” test
takersCheck for grammars all the timeNo questions with connotations of race, creed,
religion, or ethnic/national background
22Marilyn Balcita CMAS Council
• Present stem in a complete sentence, even if it is in a question. Bad Example: A medical audit specialist is …
• Ideally – no more than 50 words• Try to present stems positively. Avoid using: All (or which)
of the following except, which of the following is NOT or Never
• Avoid trick questions, or measure trivial things• Good item writers begin with: Who, What, Where, How, and
When “Which” should be followed by a noun “What” should be followed by a verb
• Use “should” instead of “would/could”• Avoid using always, all, and never• No double negatives – negative stem, and negative options
CMAS Item Writing Guiding PrinciplesThe Stem
23Marilyn Balcita CMAS Council
• No! No! --- All of the above, None of the above, No slang, No silly options
• Correct answer must be 100% correct• Distracters MUST be the wrong answer• If numeric, arrange in logical sequence• Equally balance the options – long/short, or
logical choice of options. Ex: 3 ICD codes, one CPT ---- don’t make it obvious …
• Do not overuse Partial K type items – answers with (A) 1 and 2; (B) 3 and 4 etc.
CMAS Item Writing Guiding Principles
The Options
24Marilyn Balcita CMAS Council
• Include author, title/edition, publisher, publication, date, (month for journals), and page numbers
• Official, widely known and readily available references only• References must clearly support the correct answer• For practical type questions i.e., mathematical calculation -
include the steps in calculation• DO NOT copy the exact wording from the reference source.
Exception: Legal quotes, federal regs, contracts• Use official ONLINE reference sources ONLY – CMS, DHHS,
OIG, other government sites. Cite the website in html if possible
Examples: http://www.gao.govaud/d02870g.pdf/ http://www.cms.hhs.gov/manuals/10_hospital/ho200.asp#_1_16
Baker, Judith, (2000) Health Care Finance: Basic Tools for Nonfinancial Manager. Aspen Publishers Inc. pg 165-168
Abbey, Duane. “Designing and Maintaining an Effective Chargemaster." Healthcare Financial Management, (2001 March): 50-55.
CMAS Item Writing Guiding PrinciplesReferences
25Marilyn Balcita CMAS Council
CMAS Subject Matter Expert Technical Advisory Group Team
SME-TAG• Like that name????• Who are we …• What is it for me ??
– Ability to share your knowledge– Prestige as an item developer– Earn points applicable for
recertification and membership fees, discounts to AAMAS marketplace, etc.
– Great learning experience• What do I need to do??• ……………………. Care to
join??????The TAG Team
26Marilyn Balcita CMAS Council
CMAS SME TAG Team• CMAS certified fellows only• Well rounded experience in medical audit• Experience in test item development
processes and review• Talent in writing and editing• Sound knowledge of psychometric analysis,
validation, and statistics • Agree to sign CMAS certification security
agreement• Committed and willing to learn• Have the time …..• Have the “tools” to work from – PC at a secured place, appropriate software, compatible system
C’mon down!!
27Marilyn Balcita CMAS Council
Now let’s take a look at this item one more
timeNational Hospital Billing Audit Guidelines require that:
A. Auditors must be certifiedB. Hospitals must charge audit fees in excess of $275C. Auditors must conduct themselves in a
professional manner, adhere to ethical standards, confidentiality requirements, and objectivity
D. All of the above
28Marilyn Balcita CMAS Council
Ready to have fun???Let’s play ….
• Find your group --- introduce yourself• Choose a team lead, recorder, reporter, and
timekeeper• Choose one item to develop (15 minutes)
– Identify what is being measured: Application vs. Analysis
– Match with Test Blueprint (measure med. audit function)
– Follow checklist, “guiding principles”– Validate with official resource
• Each group: Be ready to present (5 min. ) and critique (5 min.)
• Good Luck!!!
29Marilyn Balcita CMAS Council
Thanks for coming !!!Hope to hear from you soon !!!
[email protected] – 686 -0180 (East Coast time)