Upload
timothy-nelson
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
CHAPTER OVERVIEW
• Deciding on a Method• Tests and Their Development• Types of Tests• Observational Techniques• Questionnaires
SOME IMPORTANT FIRST CONSIDERATIONS
• Is the tool you propose to use reliable and valid?
• Base your choice of research tools on how you have asked the research question
TESTS AND THEIR DEVELOPMENT
• A test should measure the nature and extent of individual differences
• A good test differentiates people based on true scores
WHY USE TESTS?
• Help determine outcomes of experiments
• Can be used to diagnose strengths and weaknesses
• Assist in placing individuals in appropriate programs
• Assist in selecting applicants• Used to evaluate a program’s
effectiveness
TYPES OF TESTSOverview
What It Does
Achievement Test Assesses an individual’s knowledge of a specific area
Attitude Test Assesses an individual’s feelings about an issue
Personality Test Assesses stable individual behavior patterns
ACHIEVEMENT TEST SOURCES
• Standardized– Commercially prepared for wide use– Scoring instructions included
• Researcher-made– Designed by user for specific purpose– Scoring instructions specific to test
ACHIEVEMENT TESTS REFERENCING
What Comparison Do Tests Make?
• Norm-referenced—Individual’s scores are compared to results from a larger group
• Criterion-referenced—Individual’s scores are compared to defined performance standards
MULTIPLE-CHOICE ACHIEVEMENT ITEMS
Anatomy of a Multiple-Choice Item
12. Intelligence tests that are given to preschool children a. favor middle-class children b. have questionable construct validity c. are based on motor skills d. are no fun at all
STEMClearly written
CORRECT ANSWER
DISTRACTERSShould be plausible (b & c), not easily ruled out
(d)
TO USE OR NOT TO USE?• Advantages
– Versatile– Easy to score– Simple to take– Poor writers not
penalized– Good items used
again– Good distracters are
diagnostic– Hard to fake correct
answer
• Disadvantages– Limit student’s
options– No opportunity to
practice writing– Some people don’t do
well on them– Limits content to be
assessed– Items must be well
written
ITEM ANALYSIS: HOW TO TELL IF YOUR ITEMS WORK• Questions should discriminate
those who know the material from those who don’t
• Item analysis provides two measures of a question’s ability to discriminate– Difficulty index– Discrimination index
COMPUTING INDICESFirst Steps
1. Rank scores from highest to lowest2. Choose top 27% of scores for “high” group3. Choose bottom 27% of scores for “low”
group
COMPUTING INDICES12. Intelligence tests that are given to preschool children
a. favor middle-class children b. have questionable construct validity c. are based on motor skills
d. are no fun at all
Item Alternative A B C D Total
High Group (n = 41)
23 12 4 2 41
Low Group (n = 41)
11 9 15 6 41
Total 34 21 19 8 82
COMPUTING INDICESDifficulty index• Proportion who
answered item correctly
• D = NCh + NCl
T
Discrimination index• Proportion in high
group who answered correctly – proportion in low group who answered correctly
• D = NCh - NCl
(.5)T
NCh
= number of people in high group answering correctly
NCl= number of people in low group answering correctly
T = total number of people in high and low groups
RELATIONSHIP BETWEEN ITEM DISCRIMINATION AND DIFFICULTY
0%
+1.00
50% 100%
-1.00
0
Difficulty Level
Dis
crim
inat
ion
Lev
el
Perfect Discrimination When: ½ gets item right, ½ gets item wrong
& ½ right is in upper half, ½ wrong is in lower half
ATTITUDE TESTSAssess an Individual’s Feelings About
a Topic
Item Agree No Strong Feeling
Disagree
The day before Thanksgiving should be a holiday.
____
____
____
Final exams should be elective. ____ ____ ____
The dining room should serve gourmet food.
____
____
____
My parents don’t appreciate how smart I am.
____
____
____
My professors don’t appreciate how smart I am, either.
____
____
____
THURSTONE SCALESMethod of Equal Appearing
Intervals
• Statements indicating an attitude toward a topic are written
• Judges rank the statements from least favorable to most favorable
• Statements receiving consistent ratings are given the average score
• A set of statements covering the entire range of attitudes is selected
THURSTONE SCALESAdministration
• Respondents check items with which they agree– Well-formed attitudes indicated by
consistently checking either high or low items
– Poorly-formed or inconsistent attitudes indicated by inconsistent pattern or by checking off many neutral items
LIKERT SCALES
• Statements indicating an attitude toward a topic are written
• Items with clearly positive or negative attitudes are selected
• Statements are listed with a space for respondent to indicate degree of agreement
Item Rating
Government has no business funding child care programs.
SD D U A SA
Child care should be supported by federal, state, and local tax dollars.
SD D U A SA
A LIKERT SCALE
Directions: Indicate to what extent you agree or disagree with the statements listed below by circling one of the following:
SA means that you strongly agree with the statementA means that you agree with the statementU means that you are undecided about the statementD means that you disagree with the statementSD means that you strongly disagree with the statement
SCORING LIKERT RESPONSESMethod of Summated Ratings
• Items are weighted• Weights of unfavorable items are reversed• Average score is computed
Item Rating
Government has no business funding child care programs.
SD D U A SA
Child care should be supported by federal, state, and local tax dollars.
SD D U A SA
PERSONALITY TESTS
• Projective tests– Present respondent with ambiguous
stimulus
• Structured tests– Questions are objective
OBSERVATIONAL TECHNIQUES
• Researcher observes and records• Does not interfere with behavior
TECHNIQUES FOR RECORDING BEHAVIOR
• Duration—how long the behavior lasts• Frequency—how frequently the
behavior occurs • Interval (time sampling)—target
behaviors of each subject during set period of time
• Continuous—all behaviors of the subject
TECHNIQUES FOR RECORDING BEHAVIOR
Technique How it Works Example
Duration recording The researcher records the length of time that a behavior occurs.
How much time is spent in verbal interaction between two children?
Frequency recording The researcher records the number of times a behavior occurs.
How often are questions asked?
Interval recording The researcher observes a subject for a fixed amount of time.
Within a 60-second period, how many times do members of the group talk to another person?
Continuous recording The researcher records everything that happens.
During a 1-hour period, all the behavior of a 6-year-old boy is recorded.
OBSERVATIONAL TECHNIQUES? BE CAREFUL!
Pitfalls to Avoid
• Observer effects• Observer bias• Fatigue• Changing definitions
QUESTIONNAIRES
• What they are– Paper and pencil tests with structured
questions– Self-administered
QUESTIONNAIRES
• Advantages– Can be mailed out– Survey broad geographic area– Cheaper than one-on-one interview– Respondents may be more honest– Data easy to share with other
researchers• Disadvantage
– Low return rate
BASIC ASSUMPTIONS OF A QUESTIONNAIRE
• Does not make unreasonable demands upon the respondent
• Does not have a hidden purpose• Requests information that
respondents presumably have• Contains interesting questions• Does not request information that
could be obtained by other means
THE QUESTIONS
• The questionnaire contains questions that can be answered
• The questionnaire contains questions that are straightforward
THE FORMAT• The presentation is attractive,
professional, and easy-to-understand• Questions and pages are clearly
numbered• Directions are clear and explicit• Questions are objective• Questions are ordered sensibly• Transitions are used from one topic to
the next
THE IMPORTANCE OF USING A COVER LETTER
• Informs the recipient about the research
• Establishes the importance of the research
• Makes the recipient a part of the research