Evaluating Research Claims Perfesser Plum 2012. Evaluating TV Ad Claims An ad on TV says, “USE Cocoa Creme. You’ll have youthful skin in 10 minutes. All

Evaluating Research Evaluating Research ClaimsClaims

Perfesser PlumPerfesser Plum

20122012

Evaluating TV Ad ClaimsEvaluating TV Ad Claims

An ad on TV says,An ad on TV says,

““USE Cocoa Creme. You’ll have USE Cocoa Creme. You’ll have youthful skin in 10 minutes. All youthful skin in 10 minutes. All natural ingredients.”natural ingredients.”

You say, “Yeah, sure! Youthful You say, “Yeah, sure! Youthful skin. skin.

I bet your face falls off in 10 I bet your face falls off in 10 minutes.”minutes.”


Another ad tells you to,Another ad tells you to,

““Use Motor Magician. Your Use Motor Magician. Your engine will purr like a kitten. And engine will purr like a kitten. And you’ll get 80 miles per gallon!”you’ll get 80 miles per gallon!”

You say, “I don’t want a motor You say, “I don’t want a motor that sounds like a cat. 80 miles that sounds like a cat. 80 miles per gallon? HA! Baloney.”per gallon? HA! Baloney.”


In other words…. In other words….

You aren’t You aren’t fooled fooled by ads by ads promising what is very promising what is very unlikelyunlikely, , and that give and that give no EVIDENCEno EVIDENCE to to support the inflated promises.support the inflated promises.

Evaluating Medical ClaimsEvaluating Medical Claims

The same goes with The same goes with medicinemedicine..

Let’s say your child is ill with an infection.Let’s say your child is ill with an infection.The physician says, “Here’s a prescription The physician says, “Here’s a prescription for Megabiotic.”for Megabiotic.”You ask, “Will it work, Doctor?”You ask, “Will it work, Doctor?”Physician. “Oh, yes. It will fix her right Physician. “Oh, yes. It will fix her right up.”up.”You. “Does it have bad side effects?”You. “Does it have bad side effects?”Physician. “Oh, no. No side effects.”Physician. “Oh, no. No side effects.”

You. You. “How do you know, “How do you know, Doctor?”Doctor?”


““How do you know, DoctorHow do you know, Doctor?”?”That is That is THE THE question. question. ““HowHow do you know?” do you know?”

““What What makes you so sure?”makes you so sure?”

““Do you have Do you have trustworthy trustworthy evidenceevidence to back up your claim?” to back up your claim?”


Will you let your child take the prescribed Will you let your child take the prescribed medication if these (below) are the medication if these (below) are the physician’s answers?....physician’s answers?....

• ““Well, I and my doctor friends Well, I and my doctor friends think it think it worksworks very well.” very well.”

• ““In a study of In a study of five personsfive persons, everyone got , everyone got well.”well.”

• ““In a study of 1000 persons, In a study of 1000 persons, many said many said they felt much betterthey felt much better.”.”

You wouldn’t be satisfied by these You wouldn’t be satisfied by these answers! answers!

In fact, you’d tell the physician, “See you In fact, you’d tell the physician, “See you later!”later!”


What What DODO you want the physician to tell you you want the physician to tell you??

How about these?...How about these?...• Megabiotic was Megabiotic was tested in 10 experimental tested in 10 experimental

studiesstudies..• The research was The research was notnot conducted by the drug conducted by the drug

company, but by company, but by independent researchersindependent researchers who had nothing to gain whether who had nothing to gain whether Megabiotic was effective or not effective.Megabiotic was effective or not effective.

• Each study lasted from Each study lasted from one to five yearsone to five years..• Each study had at least Each study had at least 1000 persons1000 persons in it. in it.

[More][More]


• Each study Each study samplesample had had a mix of different kinds a mix of different kinds of personsof persons in it: younger/older, more ill/less ill, in it: younger/older, more ill/less ill, male/female.male/female.

• Each study Each study compared the effects of Megabiotic compared the effects of Megabiotic with with other medications medications (alternative treatments) (alternative treatments) and with and with no no treatmentstreatments (control group). (control group).

• Each study Each study measured the exact amountmeasured the exact amount of of infection infection beforebefore treatment, each week treatment, each week duringduring treatment, and at the treatment, and at the endend of the course of of the course of treatment. The measurement was treatment. The measurement was objectiveobjective------the amount of infection in the blood.the amount of infection in the blood.

• Side effects were also measuredSide effects were also measured, such as blood , such as blood pressure, skin rash, and kidney function.pressure, skin rash, and kidney function.

[More][More]


• All of the studiesAll of the studies found that persons who found that persons who received Megabioticreceived Megabiotic had had NO NO serious side serious side effects. effects.

• Persons who received MegabioticPersons who received Megabiotic sshowedhowed directly observable directly observable (in the blood)(in the blood) reductionsreductions in infection each week until, in infection each week until, by the end of one year, by the end of one year, 95%95% of the of the persons were cured.persons were cured.

Now Now would you let your child take the would you let your child take the medication?medication?


Probably.Probably.

Why?Why?

Why is the second set of answers Why is the second set of answers from the doctor convincing?from the doctor convincing?

[Think about it.][Think about it.]


The second answers are more The second answers are more convincing because: convincing because:

• The second answers are The second answers are more more believablebelievable..

• The second answers give you The second answers give you solid solid REASONSREASONS to believe that the to believe that the medication is likely to cure your child’s medication is likely to cure your child’s infection and not cause harm.infection and not cause harm.

• The physician gave you The physician gave you evidenceevidence that is that is hard to dismiss as just baloney.hard to dismiss as just baloney.

[More][More]

Evaluating Medical ClaimsEvaluating Medical Claims• You can’t say, “It won’t work with You can’t say, “It won’t work with mymy child,” child,”

because the research on Megabiotic was because the research on Megabiotic was tested with thousandstested with thousands of persons---many of of persons---many of whom were whom were just like your childjust like your child..

• You can’t say, “Maybe persons got well by You can’t say, “Maybe persons got well by themselves. It wasn’t Megabiotic,” because themselves. It wasn’t Megabiotic,” because persons in the control grouppersons in the control group (who got no (who got no Megabiotic) Megabiotic) did did NOTNOT get wellget well..

• You can’t say, “They didn’t measure You can’t say, “They didn’t measure improvement,” because in fact they improvement,” because in fact they measured what can be SEENmeasured what can be SEEN (the amount of (the amount of infection in the blood)---an infection in the blood)---an objective objective measuremeasure of improvement, not an opinion. of improvement, not an opinion.

[More][More]


• You can’t say, “The effects are probably You can’t say, “The effects are probably temporarytemporary; persons will get sick again,” ; persons will get sick again,” because some of the because some of the studies lasted five studies lasted five yearsyears, and no one got sick again with , and no one got sick again with the infection.the infection.

• You can’t say, “The research was You can’t say, “The research was riggedrigged to get good results. For example, to get good results. For example, researchers dropped from the study all researchers dropped from the study all persons who were NOT getting well.” persons who were NOT getting well.” You can’t say this because the drug You can’t say this because the drug company (that wanted the drug to company (that wanted the drug to work) work) did not DO the research.did not DO the research.

Summary So FarSummary So Far

Let’s summarize. Let’s summarize.

What makes research What makes research believablebelievable? ?

What makes the What makes the conclusionsconclusions of of research…research…

””The (drug, treatment, or The (drug, treatment, or method) is method) is effective. Therefore, effective. Therefore, it’s a good idea to it’s a good idea to use it.”…use it.”…

……so so trustworthytrustworthy that that you can be you can be confident in your decisionconfident in your decision to use it? to use it?


Here is what makes research believable and Here is what makes research believable and the conclusions trustworthy.the conclusions trustworthy.

• The The effect effect of the “thing” being tested of the “thing” being tested (program, materials, treatment, or method) is (program, materials, treatment, or method) is defined and measured defined and measured objectivelyobjectively..

• The thing being tested is The thing being tested is tested against tested against alternativesalternatives (to see which works (to see which works betterbetter) and ) and against against nono treatment treatment (to see if the thing works (to see if the thing works at all-at all---that is, improvement is much greater --that is, improvement is much greater than when there is NO treatment).than when there is NO treatment).

• The research is The research is done many timesdone many times ( (replicatedreplicated), ), to ensure that results are not chance, not a to ensure that results are not chance, not a fluke.fluke.

• The research is done The research is done for a long period of timefor a long period of time ((longitudinallongitudinal), to see if the effects are long-), to see if the effects are long-lasting.lasting. [More][More]


• The research is done with The research is done with samples of personssamples of persons who have the who have the same featuressame features as persons in the as persons in the larger populationlarger population who might who might later receive the program, later receive the program, treatment, or method. treatment, or method. [[representative samplerepresentative sample]]

• The researchers do not have a The researchers do not have a stake in the outcomes. stake in the outcomes. [[independent researchindependent research]]

Let’s Apply This to EducationLet’s Apply This to Education• Just as you feel responsible for the care of Just as you feel responsible for the care of

persons you love, you also feel persons you love, you also feel responsible responsible for the children you teachfor the children you teach..

• You don’t want students to be taught with You don’t want students to be taught with curriculum materials or with teaching curriculum materials or with teaching methods that methods that don’t work and that cause don’t work and that cause harm-harm---for example, teaching students the --for example, teaching students the wrong way to read or to solve math problems.wrong way to read or to solve math problems.

• Therefore, you want to Therefore, you want to ask tough questionsask tough questions about researchabout research done done on curriculum on curriculum materials and teaching methods that are materials and teaching methods that are advertised, or that a school district wants you advertised, or that a school district wants you to use, or that your colleagues say are to use, or that your colleagues say are terrific.terrific.What tough questions should you ask about What tough questions should you ask about

research?research?

Questions About Questions About PurposePurpose.. Selling?Selling?

What tough questions should you ask about What tough questions should you ask about research?research?

Let’s say you are reading a research report, or a Let’s say you are reading a research report, or a description of a new program on a website, or an description of a new program on a website, or an advertisement for curriculum materials. advertisement for curriculum materials.

Does the ad, report, or program description Does the ad, report, or program description sound as if the authors are sound as if the authors are selling an idea or a selling an idea or a product—to persuade youproduct—to persuade you that they are right; that they are right; that they have the answer; that they can fix that they have the answer; that they can fix schools and raise achievement? schools and raise achievement?

””Just (believe us, join us, buy our Just (believe us, join us, buy our materials).”materials).”If selling is their purpose, this is NOT good.If selling is their purpose, this is NOT good. Why?Why?You can be almost certain that the You can be almost certain that the only only data data (evidence) the writers will show are data that (evidence) the writers will show are data that supportsupport their claims. They will not show data their claims. They will not show data that that do NOT support their claims. [This is called do NOT support their claims. [This is called disdishonest.]honest.]

Purpose:Purpose: SellingSellingIn other words, you should be In other words, you should be highly highly suspicioussuspicious of their claims about of their claims about effectivenesseffectiveness. .

““Our method works.” Our method works.” ““Our ideas are sound.” Our ideas are sound.” ““Use our ideas and program.”Use our ideas and program.”

These claims These claims do not come from objective do not come from objective scientific researchscientific research. They come from persons . They come from persons and groups who are making up a story (a and groups who are making up a story (a sales pitch) that will get you to buy. sales pitch) that will get you to buy. If they If they had solid research evidencehad solid research evidence that that their ideas, materials, or procedures were their ideas, materials, or procedures were effective, their writing would not sound like effective, their writing would not sound like a sales pitch. a sales pitch.

Purpose:Purpose: ReportingReportingHowever, the purpose of a journal article or a However, the purpose of a journal article or a program description may simply be to program description may simply be to REPORT REPORT the findings of research.the findings of research. For example, the For example, the authors are authors are telling telling you what they learned---you what they learned---not not what they want you to believe or to dowhat they want you to believe or to do..

For instance, authors may report which one of For instance, authors may report which one of three methods of peer tutoring produces the three methods of peer tutoring produces the highest achievement.highest achievement.

Or what percentage of fourth graders are Or what percentage of fourth graders are proficient in reading on grade level after a new proficient in reading on grade level after a new reading program was used.reading program was used.

Simple reporting is good.Simple reporting is good. If authors are If authors are notnot trying to sell you an idea or materials or trying to sell you an idea or materials or programs, then they are probably going to programs, then they are probably going to report honestlyreport honestly what they found; they have what they found; they have nothing to gain simply by passing on nothing to gain simply by passing on information.information.

Questions About the Questions About the Research StrategyResearch Strategy

There are different KINDS of There are different KINDS of research research questionsquestions; for instance, there are ; for instance, there are questions….questions….

• About persons’ About persons’ opinionsopinions. “How do . “How do teachers feel about….?teachers feel about….?

• About what persons About what persons dodo. “How do students . “How do students respond when teachers….?”respond when teachers….?”

• About About factors that cause or predict factors that cause or predict changeschanges. “Does timely correction of . “Does timely correction of errors reduce those same errors errors reduce those same errors happening in the future?”happening in the future?”

• About what is About what is effectiveeffective. “Which program . “Which program (A, B, or C) is more effective in raising the (A, B, or C) is more effective in raising the reading achievement of diverse learners?”reading achievement of diverse learners?”

Questions About the Questions About the Research StrategyResearch Strategy

There are different kinds of There are different kinds of research questionsresearch questions, and there are , and there are different kinds of different kinds of research research strategiesstrategies---or WAYS of trying to ---or WAYS of trying to answer the questions. For answer the questions. For instance, there are: instance, there are:

• Surveys.Surveys.• Observation.Observation.• Experimentation.Experimentation.

Questions About the Questions About the Research StrategyResearch StrategyHere’s a big ideaHere’s a big idea

Certain kinds of research Certain kinds of research questionsquestions require certain require certain research research strategiesstrategies, , or else the research will NOT give evidence or else the research will NOT give evidence (data, findings) that you can trust.(data, findings) that you can trust.

Do you have high blood pressure? Do you have high blood pressure? Which is the better research strategy to get an Which is the better research strategy to get an answer you will trust? answer you will trust?

• Do a Do a surveysurvey of the of the opinionsopinions of 20 of your of 20 of your friends. friends.

• Use a Use a blood pressure cuffblood pressure cuff (every day for a (every day for a week, at different times of day) week, at different times of day) to get exact to get exact measurement.measurement.

Blood pressure is Blood pressure is NOT an NOT an opinionopinion-kind-of--kind-of-thingthing! ! It is an It is an objective thingobjective thing. PRESSURE. . PRESSURE. Therefore, it requires an Therefore, it requires an OBJECTIVE OBJECTIVE measurement---measurement---with the cuff.with the cuff.

Questions About the Research StrategyQuestions About the Research StrategyHere are different kinds of research questions Here are different kinds of research questions and the proper research strategies.and the proper research strategies.

• What are teachers’ What are teachers’ opinionsopinions about a new math about a new math program that they are using?program that they are using?

• How well do students How well do students think think they read three they read three months after they’ve been taught with a new months after they’ve been taught with a new reading program?reading program?

• How How satisfiedsatisfied are teachers with the are teachers with the supervision and professional development supervision and professional development they receive?they receive?

• What are the What are the evaluationsevaluations of parents on the of parents on the helpfulness of their children’s teachers?helpfulness of their children’s teachers?

These are opinion questions.These are opinion questions.

Is the Research Strategy Appropriate Is the Research Strategy Appropriate for the Research Questions?for the Research Questions?

These are opinion questions.These are opinion questions.

The researcher wants The researcher wants information on how teachers, information on how teachers, students, and parents students, and parents SEE SEE things.things.

Therefore, a useful strategy Therefore, a useful strategy would be would be SURVEY researchSURVEY research. .

Example of Example of surveysurvey research… research…

Is the Research Strategy Appropriate Is the Research Strategy Appropriate for the Research Questions?for the Research Questions? Survey Survey

ResearchResearchConsider the question on parents’ Consider the question on parents’

satisfaction.satisfaction.• The researcher selects a The researcher selects a samplesample of families. of families. • This sample is This sample is representative of the whole representative of the whole

population of familiespopulation of families of children at of children at schools; for example, small and large schools; for example, small and large families; two-parent and single-parent families; two-parent and single-parent families; families with one child and families; families with one child and families with more children; poor families families with more children; poor families and affluent families. and affluent families.

• Then the researcher either uses a Then the researcher either uses a questionnaire questionnaire (that parents fill out); or uses (that parents fill out); or uses interviews (face-to-face) interviews (face-to-face) in which the in which the researcher asks the same kinds of questions researcher asks the same kinds of questions as on the questionnaire; or gives as on the questionnaire; or gives questionnaires to all families and questionnaires to all families and also also interviews a smaller sampleinterviews a smaller sample to obtain more to obtain more detailed information.detailed information.

Is the Research Strategy Appropriate Is the Research Strategy Appropriate for the Research Questions?for the Research Questions? Survey Survey

ResearchResearchThe researcher can examine (The researcher can examine (analyzeanalyze) the survey ) the survey

information (information (datadata) from the questionnaires or ) from the questionnaires or

interviews, and can then determine:interviews, and can then determine:• What kinds of help parents feel is What kinds of help parents feel is adequateadequate..• What kinds of help parents feel is What kinds of help parents feel is inadequateinadequate..• What What percentagepercentage of parents are satisfied vs. of parents are satisfied vs.

not satisfied with teachers’ helpfulness.not satisfied with teachers’ helpfulness.

These findings could assist school These findings could assist school administrators to administrators to

improve family-school relations.improve family-school relations.


Here is a different kind of research question.Here is a different kind of research question.• What are the different What are the different waysways that teachers that teachers

respond or try to handle students’ respond or try to handle students’ disruptive behavior? What are the disruptive behavior? What are the effects effects of each wayof each way??

• When students are doing peer tutoring, do When students are doing peer tutoring, do students learn to play the role of tutor and students learn to play the role of tutor and the role of tutee? If so, the role of tutee? If so, what are the what are the behaviors in these rolesbehaviors in these roles??

• When a class is organized into learning When a class is organized into learning centers (e.g., reading, math, science), and centers (e.g., reading, math, science), and students are allowed to come and go as students are allowed to come and go as they wish, they wish, what do students actually DO in what do students actually DO in the centersthe centers? ?


These are questions about what These are questions about what persons persons DODO---about observable ---about observable activityactivity---NOT about how persons ---NOT about how persons see see things---their opinions.things---their opinions.

Therefore, a useful research Therefore, a useful research strategy is strategy is OBSERVATIONAL OBSERVATIONAL researchresearch..

Example of Example of observationobservation research…research…

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions? Observational ResearchObservational Research

Consider the question of how teachers respond to Consider the question of how teachers respond to disruptive disruptive

behavior.behavior.• The researcher The researcher selects a sampleselects a sample of classrooms. of classrooms.• Standing out of the way, the researcher Standing out of the way, the researcher takes running takes running

notesnotes (like a sportscaster) on teacher-student (like a sportscaster) on teacher-student interaction. interaction.

• After several weeks, the researcher examines the notes After several weeks, the researcher examines the notes and and a. Identifies a. Identifies different kinds of different kinds of disruptive behaviordisruptive behavior; ; e.g., e.g., making noises, talking out of turn, making noises, talking out of turn, noncompliance with noncompliance with teacher requests.teacher requests.b. Identifies b. Identifies how the teacher typically how the teacher typically respondsresponds to to each kind each kind of disruptive behaviorof disruptive behavior; e.g., telling a ; e.g., telling a student to stop, student to stop, threatening to send the student to the threatening to send the student to the office, ignoring the office, ignoring the disruptive behavior but praising disruptive behavior but praising students when they are students when they are engaging in desirable engaging in desirable behavior.behavior.c. c. Determining what generally happensDetermining what generally happens when when teachers teachers respond in different ways; e.g., students do respond in different ways; e.g., students do the disruptive the disruptive behavior more often vs. less often.behavior more often vs. less often.

Is the Research Strategy Appropriate for Is the Research Strategy Appropriate for the Research Questions?the Research Questions?

Here is yet another Here is yet another kind kind of research of research questionquestion..

• Is peer tutoring Is peer tutoring more effectivemore effective (does it (does it yield higher achievement) for some yield higher achievement) for some subjectssubjects than other subjects? than other subjects?

• Are teachers’ Are teachers’ opinionsopinions about their new about their new math programs math programs affected byaffected by whether a whether a math program provides written math program provides written procedures for teachers to follow vs. procedures for teachers to follow vs. does not provide written procedures for does not provide written procedures for teachers to follow?teachers to follow?

[More][More]


• Under what conditionsUnder what conditions (e.g., size of (e.g., size of school, recent levels of student school, recent levels of student achievement, quality of teaching staff) is achievement, quality of teaching staff) is it better (there is faster change, less it better (there is faster change, less opposition) when school reform is rapid opposition) when school reform is rapid and extensive vs. slow and in small steps? and extensive vs. slow and in small steps? [In other words, [In other words, what factors predictwhat factors predict which kind of school reform works better.]which kind of school reform works better.]

• Which math programWhich math program yields the most yields the most math proficiency in students overall and math proficiency in students overall and students by subgroups, such as ethnicity students by subgroups, such as ethnicity and social class?and social class?

These are questions about These are questions about causationcausation..


These are questions about These are questions about causationcausation. The idea . The idea

is thatis that

Certain factors Certain factors Predict or Predict or produceproduce

or variables changes in or variables changes in other factorsother factors or variables or variables

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions?

Here are examples of questions about which factors or Here are examples of questions about which factors or variables predict or produce (cause) effects in other variables predict or produce (cause) effects in other

variables.variables.• Which Which features of a math programfeatures of a math program (Teaching (Teaching

procedures? Practice? How tasks are sequenced?) procedures? Practice? How tasks are sequenced?) have the largest effects on (cause the largest have the largest effects on (cause the largest changes in) changes in) learninglearning? ?

• Which Which features of a school features of a school affect (cause or inhibit) affect (cause or inhibit) the the success of effortssuccess of efforts to reform the school to reform the school curriculum?curriculum?

• How does How does subject mattersubject matter affect (cause or inhibit) affect (cause or inhibit) learning through peer tutoringlearning through peer tutoring??

• Do Do written procedureswritten procedures or the absence of written or the absence of written procedures in a math program affect (cause) procedures in a math program affect (cause) teachers’ opinionsteachers’ opinions of the program? of the program?


This kind of question…. This kind of question…. • Which factors cause what Which factors cause what

changeschanges??• Which curriculum, program, or Which curriculum, program, or

procedure is procedure is more effectivemore effective?.... ?....

…….requires the strategy of .requires the strategy of experimental experimental

researchresearch..

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions? ExperimentExperiment

The essential feature of the strategy of The essential feature of the strategy of experimental experimental

researchresearch is that you… is that you… CompareCompare two or more situations (e.g., schools) two or more situations (e.g., schools) that are that are as similaras similar as you can make them (e.g., % as you can make them (e.g., % of students who are poor, size of school, quality of of students who are poor, size of school, quality of teaching staff), but these schools are teaching staff), but these schools are different different in in the the few factorsfew factors (e.g., reading materials) (e.g., reading materials) whose whose effectseffects you are trying to find out, or you are trying to find out, or testtest..

IfIf the only main way that the schools differ is the the only main way that the schools differ is the reading materials used, and if one school has reading materials used, and if one school has

consistently consistently higher reading achievement than the others, then higher reading achievement than the others, then it it

isisvery likely that the very likely that the reading materialsreading materials make the make the

difference. difference. What ELSE What ELSE could could it be? After all…. it be? After all….


ExperimentExperiment• The factor that makes the difference in The factor that makes the difference in

reading achievement reading achievement can’t becan’t be class size-class size-----because this is because this is the samethe same across all the across all the schools.schools.

• The factor that makes the difference in The factor that makes the difference in reading achievement reading achievement can’t becan’t be teacher teacher qualityquality---because this is ---because this is the samethe same across all across all the schools.the schools.

• The factor that makes the difference in The factor that makes the difference in reading achievement reading achievement can’t be thecan’t be the percentage of students who are poorpercentage of students who are poor------because this is because this is the samethe same across all the across all the schools.schools.Therefore, the factor that makes the Therefore, the factor that makes the difference in reading achievement difference in reading achievement must be must be the ONE major way the schools are the ONE major way the schools are differentdifferent----reading ----reading curriculumcurriculum..


Not all experiments give information that you can Not all experiments give information that you can trusttrust.. The The validity (accuracy)validity (accuracy) of the information is of the information is

weakened by weakened by OTHER factors. So, you have to carry out experiments in a OTHER factors. So, you have to carry out experiments in a

way way that that RULES OUTRULES OUT these other factors. For example… these other factors. For example…1. 1. Maybe students in schools with the highest reading Maybe students in schools with the highest reading

achievement achievement were already reading at a higher levelwere already reading at a higher level.. Therefore, these students don’t have to learn very much Therefore, these students don’t have to learn very much in order to come out first. In other words, maybe in order to come out first. In other words, maybe students’ students’ pre-skills pre-skills (and (and notnot the curriculum materials) the curriculum materials) account for higher achievement.account for higher achievement.

To rule out this possibility, an experiment should have To rule out this possibility, an experiment should have pre-pre-test measurestest measures of the variables whose change we are of the variables whose change we are

interested interested in. For instance, we should measure reading skills in all of in. For instance, we should measure reading skills in all of

the the schools schools BEFOREBEFORE new reading materials are introduced, to new reading materials are introduced, to see if some classes are ahead of other classes.see if some classes are ahead of other classes.


So, if an article says that a program or So, if an article says that a program or method produced more change or is more method produced more change or is more effective than another program or method, effective than another program or method,

ask ask if the researchers did if the researchers did pre-testspre-tests to show that to show that students all students all started at the same levelstarted at the same level.. If they If they did not do pre-tests, then you cannot trust did not do pre-tests, then you cannot trust

the the claim about effectiveness. claim about effectiveness. The group with The group with

the the highest achievement may have started at a highest achievement may have started at a higher level.higher level.


2. 2. How do you know how much students learned?How do you know how much students learned? The The only way to find out is to do only way to find out is to do postpost-tests-tests (e.g., at the end (e.g., at the end of the semester and year) to see where students’ of the semester and year) to see where students’ reading achievement reading achievement ended upended up. And you have to see . And you have to see how much students how much students changedchanged (learned, achieved) (learned, achieved) betweenbetween the pre-test the pre-test (reading proficiency at the start) (reading proficiency at the start) and the post-testand the post-test (reading achievement at the end). (reading achievement at the end). And you need to And you need to report report how much differencehow much difference there is there is between the pre-test and post-test scoresbetween the pre-test and post-test scores..

If researchersIf researchers do NOT do post-tests, or do not REPORT do NOT do post-tests, or do not REPORT the the

size of differences between the pre-test and post-testsize of differences between the pre-test and post-test, , then you then you

don’t know don’t know HOW MUCHHOW MUCH students learned as a result of students learned as a result of the the

curriculum that was used curriculum that was used betweenbetween the pre-test and the the pre-test and the post-post-

test. test. Therefore, you can’t trust claims that one Therefore, you can’t trust claims that one curriculum curriculum

produced more achievement than the others.produced more achievement than the others.


3. 3. Were measures objective? Were measures objective? You don’t rely on a physician’s You don’t rely on a physician’s subjective opinionsubjective opinion about about

your your health.health.

““You look fine You look fine to meto me.”.”Maybe the physician is right.Maybe the physician is right.But what if the physician’s opinion is wrong?But what if the physician’s opinion is wrong?Just so, you want Just so, you want objective evidence ofobjective evidence of (in this (in this

experiment) experiment) reading achievementreading achievement. It is nice to know how much . It is nice to know how much

teachers teachers

think think their students learned, but their students learned, but you can’t judge you can’t judge effectiveness based on opinioneffectiveness based on opinion. Reading is a real . Reading is a real

thing. thing. It It cancan be measured be measured objectivelyobjectively---for example, how many ---for example, how many words students read correctly per minute. words students read correctly per minute.


So, if researchers So, if researchers do NOT use do NOT use objective measuresobjective measures of (for example) of (for example) reading and math achievement (e.g., reading and math achievement (e.g., counting correct words students read counting correct words students read or problems students solve per or problems students solve per minute), but instead use the minute), but instead use the subjective opinionssubjective opinions of teachers of teachers (“Students improved a great deal.”) (“Students improved a great deal.”) then then you cannot trust claims about you cannot trust claims about effectivenesseffectiveness----because the ----because the researchers did not give researchers did not give evidence evidence of of effectiveness. They only gave effectiveness. They only gave evidence of how teachers evidence of how teachers see thingssee things. . But But that that was not the question.was not the question.


4. 4. Were Were instruments instruments and and measurement methodsmeasurement methods tested tested for for validity validity (they measure what they are (they measure what they are supposed to measure) and supposed to measure) and reliability reliability (different (different persons measuring the same thing would come persons measuring the same thing would come up with about the same scores)? up with about the same scores)? Validity.Validity. Do you think a good measure of Do you think a good measure of reading proficiencyreading proficiency is whether students can is whether students can name the parts of a book and can turn the name the parts of a book and can turn the pages? pages?

If your school principal is selecting a new If your school principal is selecting a new reading program, is that the information you reading program, is that the information you want the principal to use?want the principal to use?

““Students taught with Weird Phonics Students taught with Weird Phonics learned to learned to turn the pages. So, I’ll use that turn the pages. So, I’ll use that program with 200program with 200 students in my school.”students in my school.”

I don’t think so.I don’t think so.


ExperimentExperiment

Turning pages is not what “READING” Turning pages is not what “READING” means.means.

You want to know if a program teaches You want to know if a program teaches students to: (1) say the sounds that go students to: (1) say the sounds that go with the letters; (2) accurately and with the letters; (2) accurately and quickly read single words; (3) quickly read single words; (3) accurately and quickly read passages; accurately and quickly read passages; (4) learn many new vocabulary words (4) learn many new vocabulary words each week; (5) answer both simple and each week; (5) answer both simple and hard comprehension questions about hard comprehension questions about what they read.what they read.


So, if researchers use instruments and So, if researchers use instruments and measures that measures that do not measuredo not measure what they what they OUGHTOUGHT to measure, then to measure, then the findings have the findings have nothing to do with the research question.nothing to do with the research question.

• The question is whether a curriculum The question is whether a curriculum teaches students to teaches students to readread. But the findings . But the findings have to do with have to do with turning pagesturning pages. . Invalid Invalid measure.measure.

• The question is whether a certain teaching The question is whether a certain teaching procedure increases math procedure increases math achievementachievement. . But the findings have to do with how much But the findings have to do with how much students students enjoyed enjoyed being taught with the being taught with the method. method. Invalid measure.Invalid measure.

• The question is whether peer tutoring is The question is whether peer tutoring is effective at helping students to effective at helping students to spell betterspell better. . But the findings have to do with changes But the findings have to do with changes in in self-esteemself-esteem. . Invalid measure.Invalid measure.


Instruments and measures may measure what they are Instruments and measures may measure what they are supposed to measure (e.g., they measure reading skills) supposed to measure (e.g., they measure reading skills)

but they but they do not provide accurate (valid) informationdo not provide accurate (valid) information on what they on what they measure. They are like a stop watch that gives the measure. They are like a stop watch that gives the

wrong time. wrong time. Therefore, the findings of research may Therefore, the findings of research may LOOKLOOK as if one as if one curriculum produces higher achievement than another, curriculum produces higher achievement than another,

when in when in fact the information (data, numbers) is incorrect. fact the information (data, numbers) is incorrect.

Therefore, it is essential to use instruments and Therefore, it is essential to use instruments and measures that measures that

have have already been tested and have been shown to provide already been tested and have been shown to provide accurate informationaccurate information. You cannot trust the claims of . You cannot trust the claims of

authors authors who do who do NOTNOT discuss how their tests, instruments, or discuss how their tests, instruments, or

measures measures were validated.were validated.


5.5. Is the sample size, composition (kinds of Is the sample size, composition (kinds of participants), and selection/allocation of participants), and selection/allocation of participants to groups appropriateparticipants to groups appropriate for the type for the type of study and for how the findings might be of study and for how the findings might be used? used? Poor.Poor. The researcher claims that a teaching method The researcher claims that a teaching method is effective. However, the method was used in is effective. However, the method was used in only only oneone class, or in one school. class, or in one school. There is no There is no way to tell if it would be effective way to tell if it would be effective anywhere anywhere elseelse..Poor.Poor. The researcher claims that a method is The researcher claims that a method is notnot effective. However, it was used in only effective. However, it was used in only one one class, or in one school. There is no way to tell if class, or in one school. There is no way to tell if it would have been effective it would have been effective elsewhereelsewhere..

[More][More]


Poor.Poor. The researcher claims that an instructional The researcher claims that an instructional procedure or curriculum is more effective procedure or curriculum is more effective than another. However, the than another. However, the comparison comparison groupsgroups (one class used one curriculum; the (one class used one curriculum; the other class used a different curriculum) other class used a different curriculum) were not created by were not created by random assignmentrandom assignment of of children to the groups, or by children to the groups, or by matchingmatching the the groups to make sure they had, for example, groups to make sure they had, for example, the same percentage of boys and girls. the same percentage of boys and girls.

Therefore, Therefore, the groups may have been the groups may have been different in different in other waysother ways besides the besides the curriculumcurriculum; e.g., more children in one ; e.g., more children in one group got reading instruction at home, and group got reading instruction at home, and this (not the curriculum) caused differences this (not the curriculum) caused differences in achievement. [More]in achievement. [More]


ExperimentExperiment

So, if research involves comparing (for example) So, if research involves comparing (for example)

achievement from one group/sample (class, achievement from one group/sample (class,

school) to another, but the researchers did NOT school) to another, but the researchers did NOT

make sure (through make sure (through random assignmentrandom assignment of of

students to the samples, or through students to the samples, or through matchingmatching) ) that that

the groups have the the groups have the same characteristicssame characteristics, then , then you you

can’t trust that differences in achievement are can’t trust that differences in achievement are

caused by the curriculum or teaching method.caused by the curriculum or teaching method.


Here is Here is one kindone kind of experimental research that you of experimental research that you DO want to see.DO want to see. Level-1 research.Level-1 research.The researcher conducts a The researcher conducts a pilot studypilot study to see if a method to see if a method seems to work well enough that it ought to be given a seems to work well enough that it ought to be given a

more more extensive testextensive test. The researcher uses a . The researcher uses a small samplesmall sample (one (one

class class or school). It is not a or school). It is not a representativerepresentative sample of the school sample of the school population, but that is alright. [Do you want to use a population, but that is alright. [Do you want to use a

method method that may not work on a lot of kids?] The researcher finds that may not work on a lot of kids?] The researcher finds

that that the new method is followed by significant student the new method is followed by significant student achievement. But since the sample is small and not achievement. But since the sample is small and not representative of the larger school population, and since representative of the larger school population, and since the research was done only once, the researcher cautions the research was done only once, the researcher cautions

that that the findings are very tentative and cannot be generalized the findings are very tentative and cannot be generalized anywhere else. Further research is needed. anywhere else. Further research is needed.

This is called honesty.This is called honesty.


Here is another kind of experimental research that you DO Here is another kind of experimental research that you DO want to see. want to see. Level-2 research.Level-2 research.The researcher The researcher repeats researchrepeats research on the same method in on the same method in different schools. Each new study is called a different schools. Each new study is called a replicationreplication. .

The The sample of schools sample of schools is representativeis representative of the population of of the population of schools---small/large, wealthy/poor, rural/urban. schools---small/large, wealthy/poor, rural/urban. The researcher claims that data from most of the studies The researcher claims that data from most of the studies show that the method is effective in most of the schools, show that the method is effective in most of the schools,

but but the researcher cannot figure out which factors hinder its the researcher cannot figure out which factors hinder its effectiveness or what additional factors may be needed to effectiveness or what additional factors may be needed to make it work better. The researcher advises caution in make it work better. The researcher advises caution in

using using the method. the method. [This is honest and morally responsible.] [This is honest and morally responsible.]


Here is yet another kind of experimental research Here is yet another kind of experimental research that you DO want to see.that you DO want to see. Level-3 research.Level-3 research. The researcher conducts a study in The researcher conducts a study in several school several school districts at oncedistricts at once. The sample of districts is . The sample of districts is representative representative of the larger population of schools and of the larger population of schools and school districts. Three new reading curricula are school districts. Three new reading curricula are tested against each other. The three curricula are tested against each other. The three curricula are randomly assignedrandomly assigned to the districts. to the districts. Objective and Objective and validated measuresvalidated measures of essential reading skills are of essential reading skills are made before the new curricula are introduced (pre-made before the new curricula are introduced (pre-test), every two weeks during the use of the curricula test), every two weeks during the use of the curricula (progress assessment). and at the end of the year (progress assessment). and at the end of the year (post test, or outcome assessment). [More](post test, or outcome assessment). [More]


At the end of the year, researchers At the end of the year, researchers examine the amount of change in students’ examine the amount of change in students’ skills skills from pre-test to post-testfrom pre-test to post-test. They find . They find that curriculum A produced significantly that curriculum A produced significantly more achievement (and objectively high more achievement (and objectively high achievement) in all of the reading skills achievement) in all of the reading skills (compared to achievement produced by (compared to achievement produced by curricula B and C) curricula B and C) in 8 out of the 10 in 8 out of the 10 schoolsschools in which it was used. The in which it was used. The researchers therefore claim that researchers therefore claim that curriculum A is more likely than the other curriculum A is more likely than the other curricula to produce high reading curricula to produce high reading achievement, and that it can be used with achievement, and that it can be used with at least modest confidenceat least modest confidence. However, they . However, they say that more research is needed to say that more research is needed to determine the school conditions in which determine the school conditions in which curriculum A works best.curriculum A works best.

Would you use curriculum A?Would you use curriculum A?

Validity ----> Credibility ----> Validity ----> Credibility ----> UsabilityUsability

For Making Decisions About: For Making Decisions About: • Instructional Methods: Effective?Instructional Methods: Effective?• Program Materials: Effective?Program Materials: Effective?• Assessment Instruments and Assessment Instruments and

Methods: Measure what they are Methods: Measure what they are supposed to, reliably?supposed to, reliably?

• Data, findings: Accurate? Data, findings: Accurate? Generalizable?Generalizable?

Adequacy of the QuestionAdequacy of the QuestionIs the question, hypothesis, problem, or Is the question, hypothesis, problem, or interest stated precisely enough that interest stated precisely enough that things to measure objectively can be things to measure objectively can be derived from it? derived from it? Poor.Poor. “Do students learn more when “Do students learn more when tasks are authentic? [What does tasks are authentic? [What does ‘authentic’ mean? What does ‘learn ‘authentic’ mean? What does ‘learn more’ mean? Amount? Speed?]more’ mean? Amount? Speed?]Better.Better. “Is there a difference in the “Is there a difference in the number of learning trials required for number of learning trials required for students to learn to decode words that students to learn to decode words that are in common use (such as fun, car, are in common use (such as fun, car, cat, is, run) vs. pseudowords (fis, nif, cat, is, run) vs. pseudowords (fis, nif, ris)?”ris)?”

Coverage of the TopicCoverage of the Topic Poor.Poor. A new theory of reading is presented. A new theory of reading is presented. The only literature cited was done by the The only literature cited was done by the author and by persons sympathetic to author and by persons sympathetic to the author’s position (that a new theory the author’s position (that a new theory is needed). The author criticizes other is needed). The author criticizes other approaches to reading instruction approaches to reading instruction (except his own), but presents none of (except his own), but presents none of the extensive research that DOES the extensive research that DOES support other approaches and/or that support other approaches and/or that criticizes the author’s. In other words, criticizes the author’s. In other words, the author is misleading readers into the author is misleading readers into thinking that his approach is the best thinking that his approach is the best one.one.

Coverage of the TopicCoverage of the Topic

Better.Better.

A new theory of reading is presented. A new theory of reading is presented. The author identifies the other major The author identifies the other major approaches to reading instruction, and approaches to reading instruction, and presents a representative sample of presents a representative sample of the research on each approach—both the research on each approach—both research that supports and research research that supports and research that challenges these approaches. The that challenges these approaches. The author also presents research that author also presents research that supports and that challenges his own supports and that challenges his own approach.approach.

RelevanceRelevanceDid the literature review examine Did the literature review examine literature that is both directly literature that is both directly relevant to the topic at hand and to relevant to the topic at hand and to larger relevant issues? For example, larger relevant issues? For example, did a literature review on effective did a literature review on effective mathematics instruction include mathematics instruction include mathematics research and research mathematics research and research on instruction in general?on instruction in general?

Poor.Poor.

The literature review focuses narrowly The literature review focuses narrowly on the topic.on the topic.

RelevanceRelevance

Better.Better.

The literature review focuses on The literature review focuses on the topic and on other relevant the topic and on other relevant issues, such as the proper issues, such as the proper distribution of practice. distribution of practice.

Therefore, the reader can see that Therefore, the reader can see that the topic at hand is important on the topic at hand is important on both a smaller and larger scale.both a smaller and larger scale.

Useful SummaryUseful Summary

Did the writer summarize the literature Did the writer summarize the literature with a set of generalizations stated as with a set of generalizations stated as propositions and facts, and diagrams? propositions and facts, and diagrams? Did the writer identify possible gaps in Did the writer identify possible gaps in what is known?what is known?

Poor.Poor.

The literature review is mostly a cascade The literature review is mostly a cascade of supporting citations that gives the of supporting citations that gives the appearance of serious intent to conduct appearance of serious intent to conduct unbiased work. The literature does not unbiased work. The literature does not lead lead obviouslyobviously to particular research to particular research questions.questions.








Better.Better.



Poor.Poor.


RelevanceRelevance

Better.Better.





Poor.Poor.



Better.Better.

The literature review ends with a The literature review ends with a summary of what is known summary of what is known empiricallyempirically, partly known but not , partly known but not confirmed, and what is unknown. confirmed, and what is unknown. These These empiricalempirical findings are stated findings are stated such that it is clear exactly what the such that it is clear exactly what the writer is talking about and what writer is talking about and what needs to be studied.needs to be studied.


Poor.Poor.

““The more engaged students are The more engaged students are the more they participate.” [Vague]the more they participate.” [Vague]

Good.Good.

““The higher the rate of The higher the rate of opportunities to respond, the fewer opportunities to respond, the fewer times students become inattentive.” times students become inattentive.” [Objective, observable] [Objective, observable]

FeasibilityFeasibility

Is the project do-able, or feasible? Can Is the project do-able, or feasible? Can sufficient data be collected and analyzed?sufficient data be collected and analyzed?Poor.Poor. “We wish to determine the most “We wish to determine the most effective method for teaching effective method for teaching comprehension.” [It is impossible to comprehension.” [It is impossible to determine what IS most effective because determine what IS most effective because you cannot study everything that IS and you cannot study everything that IS and will be.]will be.]Better.Better. “We wish to determine which of “We wish to determine which of three methods for teaching three methods for teaching comprehension is associated with the comprehension is associated with the highest gains in scores of comprehension highest gains in scores of comprehension on the Especially Fine Test of Reading on the Especially Fine Test of Reading Comprehension.”Comprehension.”

FeasibilityFeasibilityPoor.Poor. ““Our objective is to determine whether Our objective is to determine whether violent images in the media cause violent violent images in the media cause violent behavior in schools. [You may learn that behavior in schools. [You may learn that students who commit more violent acts in students who commit more violent acts in school also watch more violent events on school also watch more violent events on TV, but you can never know if seeing those TV, but you can never know if seeing those events events causes causes the violence. Besides, there the violence. Besides, there are so many intervening variables and are so many intervening variables and contributing conditions that you could contributing conditions that you could never study all the connections.]never study all the connections.]Better.Better. ““Our objective is to determine whether Our objective is to determine whether students who commit more violent acts in students who commit more violent acts in school also watch more violent events on school also watch more violent events on TV.”TV.”

Null HypothesisNull Hypothesis

Is the research or report based on the Is the research or report based on the null hypothesis—an intellectually null hypothesis—an intellectually honest position? honest position?

““I believe program X works, but I’m I believe program X works, but I’m going to assume that it going to assume that it doesn’tdoesn’t work work and I’m going to collect data to try to and I’m going to collect data to try to show that it doesn’t workshow that it doesn’t work. If the data . If the data do NOT show that X does not work, do NOT show that X does not work, then I will conclude that then I will conclude that maybe it does maybe it does workwork. Maybe.” . Maybe.”


Or, on the contrary, is the Or, on the contrary, is the author is trying to persuade author is trying to persuade readers to join him in readers to join him in accepting a method, or accepting a method, or already believes he knows the already believes he knows the truth or knows what works, or truth or knows what works, or is not very interested in the is not very interested in the possibility that he is wrong?possibility that he is wrong?

Null HypothesisNull HypothesisPoor.Poor. ““Good readers do NOT sound out words. Good readers do NOT sound out words. They use context to They use context to predictpredict what words what words say. I will say. I will support support this statement with this statement with examples.” examples.”

The point is not to support what you The point is not to support what you believe but to TEST it—believe but to TEST it—to try to to try to FALSIFY it. Look for examples that do FALSIFY it. Look for examples that do NOT support the belief or hypothesis. NOT support the belief or hypothesis. Look for good readers who do NOT Look for good readers who do NOT guess. Anyone can find evidence to guess. Anyone can find evidence to support the stupidest ideas. Hitler support the stupidest ideas. Hitler could “prove” he was a kind person by could “prove” he was a kind person by reporting how often he petted his dog.reporting how often he petted his dog.


Poor.Poor.

An author only reports studies that show An author only reports studies that show one thing (that Program X is associated one thing (that Program X is associated with high achievement). The author with high achievement). The author concludes that Program X produces high concludes that Program X produces high achievement. However, there is some achievement. However, there is some research showing that Program X did not research showing that Program X did not produce high achievement. The author produce high achievement. The author did not report THAT research. [This is did not report THAT research. [This is called “cherry picking.” It is also called called “cherry picking.” It is also called not telling the whole truth.]not telling the whole truth.]

Null HypothesisNull HypothesisGood.Good. An author conducts research on Math An author conducts research on Math Program X in one school (suburban, middle Program X in one school (suburban, middle class). Kids who got Program X learned a lot class). Kids who got Program X learned a lot of math. The author says, “It probably won’t of math. The author says, “It probably won’t work again.” She tests the program again in work again.” She tests the program again in a very similar school. Same results. “Okay, a very similar school. Same results. “Okay, but it won’t work in a poor, urban school.” but it won’t work in a poor, urban school.” Same results. Okay, that was a fluke. It Same results. Okay, that was a fluke. It won’t happen again.” She tests it in another won’t happen again.” She tests it in another poor, urban school. Same results. “Okay, poor, urban school. Same results. “Okay, but it won’t work with Hispanic kids.” It but it won’t work with Hispanic kids.” It does. “Okay, but it won’t work with POOR does. “Okay, but it won’t work with POOR Hispanic kids.” It does. “Okay, but it won’t Hispanic kids.” It does. “Okay, but it won’t work in rural areas. It does. “Okay, maybe work in rural areas. It does. “Okay, maybe it’s wrong to say it doesn’t work. It makes it’s wrong to say it doesn’t work. It makes sense for a principal to try it.” [Replication]sense for a principal to try it.” [Replication]

Appropriate DesignAppropriate Design

Is the design (survey, experiment, Is the design (survey, experiment, ethnography) appropriate to the ethnography) appropriate to the question? question?

Let’s say the research question has to do Let’s say the research question has to do with what works better (what is with what works better (what is associated with better outcomes), or associated with better outcomes), or whether something worked. For example, whether something worked. For example, is one reading program better than is one reading program better than another? Does a math program produce another? Does a math program produce more skilled math students?more skilled math students?

Appropriate DesignAppropriate DesignPoor.Poor. Do a survey of teachers. Ask their Do a survey of teachers. Ask their opinions. [If so, you are measuring opinions. [If so, you are measuring teacher perception. You WANT to measure teacher perception. You WANT to measure student achievement. That requires that student achievement. That requires that you measure what students DO, not merely you measure what students DO, not merely what teachers think.]what teachers think.]

Poor.Poor. Have teachers make up their own Have teachers make up their own instruments to measure progress and instruments to measure progress and outcomes. [How will you know that these outcomes. [How will you know that these instruments measure what they are instruments measure what they are supposed to measure? Are reliable? Are supposed to measure? Are reliable? Are predictive? Are as useful as standardized predictive? Are as useful as standardized instruments? You won’t.]instruments? You won’t.]


Poor.Poor. Conduct ethnographic observations in Conduct ethnographic observations in classes. Observe student-teacher classes. Observe student-teacher interaction and students working together interaction and students working together and individually. and individually. [This may be interesting information, but it [This may be interesting information, but it tells you little about WHAT math or reading tells you little about WHAT math or reading skills students learned. It tells you about skills students learned. It tells you about the conditions in which they learned or the conditions in which they learned or didn’t learn. didn’t learn. You can’t use ethnographic You can’t use ethnographic research as the sole source of evaluative research as the sole source of evaluative data on methods and curricula.data on methods and curricula. You need You need to know what students learned and how to know what students learned and how much they learned.] much they learned.]

Appropriate DesignAppropriate DesignBetter.Better. You have an You have an experimental designexperimental design. You give . You give pre-tests, post-tests, and possibly pre-tests, post-tests, and possibly measures of progress in between. You use measures of progress in between. You use instruments that have already been instruments that have already been validated by research, and that measure validated by research, and that measure student learning—specifically, the accuracy student learning—specifically, the accuracy and speed of solving math problems, or and speed of solving math problems, or reading connected text. If you are reading connected text. If you are comparing one program with another, you comparing one program with another, you have comparison classes that are as have comparison classes that are as similar as possible so that other variables similar as possible so that other variables (such as help from parents, differences in (such as help from parents, differences in students’ abilities) do not contribute to the students’ abilities) do not contribute to the outcomes. Observers are trained to use outcomes. Observers are trained to use the instruments and the reliability of their the instruments and the reliability of their measures is determined.measures is determined.








Better.Better.



Poor.Poor.


RelevanceRelevance

Better.Better.





Poor.Poor.



Better.Better.

The literature review ends with a The literature review ends with a summary of what is known summary of what is known empiricallyempirically, partly known but not , partly known but not confirmed, and what is unknown. confirmed, and what is unknown. These These empiricalempirical findings are stated findings are stated such that it is clear exactly what the such that it is clear exactly what the writer is talking about and what writer is talking about and what needs to be studied.needs to be studied.


Poor.Poor.

““The more engaged students are The more engaged students are the more they participate.” [Vague]the more they participate.” [Vague]

Good.Good.

““The higher the rate of The higher the rate of opportunities to respond, the fewer opportunities to respond, the fewer times students become inattentive.” times students become inattentive.” [Objective, observable] [Objective, observable]

FeasibilityFeasibility

Is the project do-able, or feasible? Can Is the project do-able, or feasible? Can sufficient data be collected and analyzed?sufficient data be collected and analyzed?Poor.Poor. “We wish to determine the most “We wish to determine the most effective method for teaching effective method for teaching comprehension.” [It is impossible to comprehension.” [It is impossible to determine what IS most effective because determine what IS most effective because you cannot study everything that IS and you cannot study everything that IS and will be.]will be.]Better.Better. “We wish to determine which of “We wish to determine which of three methods for teaching three methods for teaching comprehension is associated with the comprehension is associated with the highest gains in scores of comprehension highest gains in scores of comprehension on the Especially Fine Test of Reading on the Especially Fine Test of Reading Comprehension.”Comprehension.”

FeasibilityFeasibilityPoor.Poor. ““Our objective is to determine whether Our objective is to determine whether violent images in the media cause violent violent images in the media cause violent behavior in schools. [You may learn that behavior in schools. [You may learn that students who commit more violent acts in students who commit more violent acts in school also watch more violent events on school also watch more violent events on TV, but you can never know if seeing those TV, but you can never know if seeing those events events causes causes the violence. Besides, there the violence. Besides, there are so many intervening variables and are so many intervening variables and contributing conditions that you could contributing conditions that you could never study all the connections.]never study all the connections.]Better.Better. ““Our objective is to determine whether Our objective is to determine whether students who commit more violent acts in students who commit more violent acts in school also watch more violent events on school also watch more violent events on TV.”TV.”


Is the research or report based on the Is the research or report based on the null hypothesis—an intellectually null hypothesis—an intellectually honest position? honest position?

““I believe program X works, but I’m I believe program X works, but I’m going to assume that it going to assume that it doesn’tdoesn’t work work and I’m going to collect data to try to and I’m going to collect data to try to show that it doesn’t workshow that it doesn’t work. If the data . If the data do NOT show that X does not work, do NOT show that X does not work, then I will conclude that then I will conclude that maybe it does maybe it does workwork. Maybe.” . Maybe.”


Or, on the contrary, is the Or, on the contrary, is the author is trying to persuade author is trying to persuade readers to join him in readers to join him in accepting a method, or accepting a method, or already believes he knows the already believes he knows the truth or knows what works, or truth or knows what works, or is not very interested in the is not very interested in the possibility that he is wrong?possibility that he is wrong?

Null HypothesisNull HypothesisPoor.Poor. ““Good readers do NOT sound out words. Good readers do NOT sound out words. They use context to They use context to predictpredict what words what words say. I will say. I will support support this statement with this statement with examples.” examples.”

The point is not to support what you The point is not to support what you believe but to TEST it—believe but to TEST it—to try to to try to FALSIFY it. Look for examples that do FALSIFY it. Look for examples that do NOT support the belief or hypothesis. NOT support the belief or hypothesis. Look for good readers who do NOT Look for good readers who do NOT guess. Anyone can find evidence to guess. Anyone can find evidence to support the stupidest ideas. Hitler support the stupidest ideas. Hitler could “prove” he was a kind person by could “prove” he was a kind person by reporting how often he petted his dog.reporting how often he petted his dog.


Poor.Poor.

An author only reports studies that show An author only reports studies that show one thing (that Program X is associated one thing (that Program X is associated with high achievement). The author with high achievement). The author concludes that Program X produces high concludes that Program X produces high achievement. However, there is some achievement. However, there is some research showing that Program X did not research showing that Program X did not produce high achievement. The author produce high achievement. The author did not report THAT research. [This is did not report THAT research. [This is called “cherry picking.” It is also called called “cherry picking.” It is also called not telling the whole truth.]not telling the whole truth.]

Null HypothesisNull HypothesisGood.Good. An author conducts research on Math An author conducts research on Math Program X in one school (suburban, middle Program X in one school (suburban, middle class). Kids who got Program X learned a lot class). Kids who got Program X learned a lot of math. The author says, “It probably won’t of math. The author says, “It probably won’t work again.” She tests the program again in work again.” She tests the program again in a very similar school. Same results. “Okay, a very similar school. Same results. “Okay, but it won’t work in a poor, urban school.” but it won’t work in a poor, urban school.” Same results. Okay, that was a fluke. It Same results. Okay, that was a fluke. It won’t happen again.” She tests it in another won’t happen again.” She tests it in another poor, urban school. Same results. “Okay, poor, urban school. Same results. “Okay, but it won’t work with Hispanic kids.” It but it won’t work with Hispanic kids.” It does. “Okay, but it won’t work with POOR does. “Okay, but it won’t work with POOR Hispanic kids.” It does. “Okay, but it won’t Hispanic kids.” It does. “Okay, but it won’t work in rural areas. It does. “Okay, maybe work in rural areas. It does. “Okay, maybe it’s wrong to say it doesn’t work. It makes it’s wrong to say it doesn’t work. It makes sense for a principal to try it.” [Replication]sense for a principal to try it.” [Replication]


Is the design (survey, experiment, Is the design (survey, experiment, ethnography) appropriate to the ethnography) appropriate to the question? question?

Let’s say the research question has to do Let’s say the research question has to do with what works better (what is with what works better (what is associated with better outcomes), or associated with better outcomes), or whether something worked. For example, whether something worked. For example, is one reading program better than is one reading program better than another? Does a math program produce another? Does a math program produce more skilled math students?more skilled math students?

Appropriate DesignAppropriate DesignPoor.Poor. Do a survey of teachers. Ask their Do a survey of teachers. Ask their opinions. [If so, you are measuring opinions. [If so, you are measuring teacher perception. You WANT to measure teacher perception. You WANT to measure student achievement. That requires that student achievement. That requires that you measure what students DO, not merely you measure what students DO, not merely what teachers think.]what teachers think.]

Poor.Poor. Have teachers make up their own Have teachers make up their own instruments to measure progress and instruments to measure progress and outcomes. [How will you know that these outcomes. [How will you know that these instruments measure what they are instruments measure what they are supposed to measure? Are reliable? Are supposed to measure? Are reliable? Are predictive? Are as useful as standardized predictive? Are as useful as standardized instruments? You won’t.]instruments? You won’t.]


Poor.Poor. Conduct ethnographic observations in Conduct ethnographic observations in classes. Observe student-teacher classes. Observe student-teacher interaction and students working together interaction and students working together and individually. and individually. [This may be interesting information, but it [This may be interesting information, but it tells you little about WHAT math or reading tells you little about WHAT math or reading skills students learned. It tells you about skills students learned. It tells you about the conditions in which they learned or the conditions in which they learned or didn’t learn. didn’t learn. You can’t use ethnographic You can’t use ethnographic research as the sole source of evaluative research as the sole source of evaluative data on methods and curricula.data on methods and curricula. You need You need to know what students learned and how to know what students learned and how much they learned.] much they learned.]

Appropriate DesignAppropriate DesignBetter.Better. You have an You have an experimental designexperimental design. You give . You give pre-tests, post-tests, and possibly pre-tests, post-tests, and possibly measures of progress in between. You use measures of progress in between. You use instruments that have already been instruments that have already been validated by research, and that measure validated by research, and that measure student learning—specifically, the accuracy student learning—specifically, the accuracy and speed of solving math problems, or and speed of solving math problems, or reading connected text. If you are reading connected text. If you are comparing one program with another, you comparing one program with another, you have comparison classes that are as have comparison classes that are as similar as possible so that other variables similar as possible so that other variables (such as help from parents, differences in (such as help from parents, differences in students’ abilities) do not contribute to the students’ abilities) do not contribute to the outcomes. Observers are trained to use outcomes. Observers are trained to use the instruments and the reliability of their the instruments and the reliability of their measures is determined.measures is determined.

Experimental DesignsExperimental DesignsClassic Design: Pre-test, Post-Classic Design: Pre-test, Post-Test, Control Group, with Random Test, Control Group, with Random AllocationAllocation

EE

C C Pre-test Post-testPre-test Post-test

However, randomization may not be However, randomization may not be possible. Some students are denied possible. Some students are denied effective instruction.effective instruction.

Experimental DesignsExperimental DesignsTime SeriesTime Series

99 00 01 02 03 04 05 06 07 0899 00 01 02 03 04 05 06 07 08

Eclectic Reading First Eclectic Reading First Percentage PassingPercentage Passing

No no-treatment control group. No no-treatment control group. However, samples will vary from year to However, samples will vary from year to year.year.

Experimental DesignsExperimental Designs

Interrupted Time Series. Same Interrupted Time Series. Same Group, Reversal of Treatment Group, Reversal of Treatment and Baselineand Baseline Eclectic Eclectic RF Eclectic RF Eclectic RFRF

A1 B1 A2 A1 B1 A2 B2B2

Appropriate DesignAppropriate DesignRule.Rule. If you want to If you want to describedescribe what is what is happening at one point in time (for happening at one point in time (for example, the percentage of students example, the percentage of students in each grade level that pass state in each grade level that pass state tests in reading—a tests in reading—a cross-sectioncross-section) you ) you could do a SURVEY (literally, over could do a SURVEY (literally, over view) of a sample of schools that view) of a sample of schools that represent the population of schools represent the population of schools in the state (e.g., some urban, some in the state (e.g., some urban, some suburban, some rural; some wealthy, suburban, some rural; some wealthy, some middle class, some poor). some middle class, some poor).


These survey data might These survey data might show show relationshipsrelationships; for example, the higher ; for example, the higher the social class, the higher the the social class, the higher the percentage of students who pass. But percentage of students who pass. But that does not show that social class is that does not show that social class is the the causecause of achievement. It could be of achievement. It could be that parental education is part of the that parental education is part of the cause; educated parents can teach cause; educated parents can teach their own children, or wealthier their own children, or wealthier schools have better teachers. schools have better teachers.


If you want to find out what people think If you want to find out what people think and feel, ask them, using interviews or and feel, ask them, using interviews or questionnaires. This information may be questionnaires. This information may be useful in making certain decisions. For useful in making certain decisions. For example, teachers using a new reading example, teachers using a new reading program may say that they do not like it, or program may say that they do not like it, or say that they need more assistance. Still, say that they need more assistance. Still, these opinions cannot be used to judge these opinions cannot be used to judge whether a program is well-designed or is whether a program is well-designed or is effective.effective. That is a different variable and That is a different variable and would have to be measured in terms of would have to be measured in terms of student achievement by teachers who used student achievement by teachers who used the program properly. the program properly.

Causal RelationshipsCausal Relationships

If the research is assessing If the research is assessing causal relationships (e.g., causal relationships (e.g., program effectiveness) did the program effectiveness) did the researchers satisfy the researchers satisfy the conditions for making credible conditions for making credible causal inferences? Did they:causal inferences? Did they:

Causal RelationshipsCausal Relationships• Determine that change in the outcome Determine that change in the outcome

variables (e.g., achievement)variables (e.g., achievement) followed followed change in the input variables change in the input variables (introduction of the program)? This (introduction of the program)? This requires a requires a pre-test pre-test and perhapsand perhaps repeatedrepeated or or alternative measuresalternative measures..

• Collect data on both the outcome Collect data on both the outcome variables (e.g., achievement) and the variables (e.g., achievement) and the input variables (e.g., whether, how often, input variables (e.g., whether, how often, and how well teachers used the and how well teachers used the program)? Otherwise, if data show that program)? Otherwise, if data show that achievement is low, you might conclude achievement is low, you might conclude the program is ineffective when in fact the program is ineffective when in fact teachers did not use it properly.teachers did not use it properly.

• Rule out the effects of extraneous Rule out the effects of extraneous factors, such as maturation, other factors, such as maturation, other sources of instruction, unreliable sources of instruction, unreliable measurement?measurement?

DefinitionsDefinitionsOften, Often,

• When When definitions and measures definitions and measures are vagueare vague (you don’t know exactly (you don’t know exactly what you are supposed to see), what you are supposed to see),

• When When measures are subjectivemeasures are subjective (students enjoy, appreciate, (students enjoy, appreciate, demonstrate, understand), and demonstrate, understand), and

• When When data collection involves data collection involves writing narrative and impressions writing narrative and impressions rather than countingrather than counting what what students and teacher do…students and teacher do…it serves the function of disguising it serves the function of disguising bias, ignorance, and agenda bias, ignorance, and agenda behind a façade of “science.”behind a façade of “science.”

DefinitionsDefinitions

Weak, vague, imprecise, and too-broad Weak, vague, imprecise, and too-broad definitions foster or support definitions foster or support unaccountability.unaccountability.

How can you be held accountable if you How can you be held accountable if you never say exactly what you do and what never say exactly what you do and what students are supposed to learn?students are supposed to learn?

It ensures that beliefs and biases will It ensures that beliefs and biases will always be supported by evidence because always be supported by evidence because when definitions are vague and general, when definitions are vague and general, and data are subjective, and data are subjective, almost anything almost anything can count as supporting evidencecan count as supporting evidence..


Were adequate Were adequate conceptual definitionsconceptual definitions (from which measures were derived) (from which measures were derived) used? That is, did conceptual definitions used? That is, did conceptual definitions cover all relevant features of a concept cover all relevant features of a concept and exclude irrelevant features? and exclude irrelevant features?

Poor.Poor.

Reading is defined as a psycholinguistic Reading is defined as a psycholinguistic guessing game. Therefore, researchers guessing game. Therefore, researchers measure student guessing as evidence of measure student guessing as evidence of reading. Is guessing what is meant by reading. Is guessing what is meant by reading?reading?


Poor.Poor.

Reading is defined as making sense of Reading is defined as making sense of text, or “meaning-making.” [In other text, or “meaning-making.” [In other words, reading is wrongly defined not by words, reading is wrongly defined not by what readers DO but by what reading what readers DO but by what reading accomplishes.] Therefore, researchers will accomplishes.] Therefore, researchers will only measure comprehension. If a teacher only measure comprehension. If a teacher is not yet working on comprehension (but is not yet working on comprehension (but IS working on other reading skills) IS working on other reading skills) students will by that narrow definition be students will by that narrow definition be considered nonreaders.considered nonreaders.

DefinitionsDefinitionsBetter.Better.Reading is a cognitive routine for Reading is a cognitive routine for accurately and rapidly decoding written accurately and rapidly decoding written text into words and connected statements, text into words and connected statements, and then comprehending the definitions and then comprehending the definitions and propositions communicated by the and propositions communicated by the text. text. [This definition covers all the reading [This definition covers all the reading skills (the correspondence between letters skills (the correspondence between letters and sounds, sounding out words, reading and sounds, sounding out words, reading words and sentences fluently, and knowing words and sentences fluently, and knowing vocabulary and comprehension vocabulary and comprehension strategies). Research based on this strategies). Research based on this definition would be obliged either to study definition would be obliged either to study all of what is meant by reading or to all of what is meant by reading or to explicitly limit the research to certain explicitly limit the research to certain subskills.] subskills.]


Where adequate Where adequate operational operational definitionsdefinitions used? Operational used? Operational definitions are supposed to be definitions are supposed to be derived from conceptual derived from conceptual definitions. In addition, definitions. In addition, operational definitions should operational definitions should provide clear examples that cover provide clear examples that cover the range of what is implied by the the range of what is implied by the conceptual definition, and should conceptual definition, and should exclude what is not relevant.exclude what is not relevant.

DefinitionsDefinitionsPoor.Poor. Reading is (properly) conceptually defined Reading is (properly) conceptually defined as a cognitive routine for accurately and as a cognitive routine for accurately and rapidly decoding written text into words rapidly decoding written text into words and connected statements and then and connected statements and then comprehending the definitions and comprehending the definitions and propositions communicated by the text. propositions communicated by the text. However, the operational definition of However, the operational definition of reading includes how children handle reading includes how children handle books (upright, turn pages), name the books (upright, turn pages), name the parts of a book, and recognize parts of a book, and recognize environmental print. These may be environmental print. These may be important, but important, but they are NOT part of the they are NOT part of the conceptual definition of readingconceptual definition of reading. . Therefore, kids whose reading is measured Therefore, kids whose reading is measured according to this definition may get high according to this definition may get high scores even though they are not reading.scores even though they are not reading.


Better.Better.

The researchers state that they will not The researchers state that they will not measure all aspects of reading. They are measure all aspects of reading. They are interested (in this study) only in students interested (in this study) only in students accurately and rapidly decoding words. accurately and rapidly decoding words. They define decoding words this way. “The They define decoding words this way. “The student says each sound in a word, does student says each sound in a word, does not stop between the sounds, and says the not stop between the sounds, and says the word as a unit (blends sounds into a word as a unit (blends sounds into a whole).”whole).” This operational definition yields This operational definition yields objective, observable measures.objective, observable measures.


Were Were objective measuresobjective measures derived derived from the operational definitions? from the operational definitions? Decisions about which variables Decisions about which variables affect other variables, or about affect other variables, or about whether a program or method is whether a program or method is effective, are so important to effective, are so important to children’s lives that measures must children’s lives that measures must be objective; that is, be objective; that is, any any observer observer can SEE what is being measured. can SEE what is being measured.

DefinitionsDefinitionsPoor measures.Poor measures.

• ““Students enjoyment of reading.” [How Students enjoyment of reading.” [How many times they smile?]many times they smile?]

• ““Students’ appreciation of literary Students’ appreciation of literary genres.” [Students say Thanks to a genres.” [Students say Thanks to a poem?]poem?]

• ““Students’ demonstrating understanding Students’ demonstrating understanding of the sounding out routine?” [They use of the sounding out routine?” [They use hand puppets?]hand puppets?]

If a researcher doesn’t define variables in If a researcher doesn’t define variables in terms of what persons DO, then the terms of what persons DO, then the researcher probably doesn’t know WHAT researcher probably doesn’t know WHAT he’s talking about.he’s talking about.

DefinitionsDefinitionsBetter measures.Better measures.

• Enjoyment of reading might be Enjoyment of reading might be measured objectively (indirectly) by measured objectively (indirectly) by HOW MANY books or how much time HOW MANY books or how much time students read on their own. students read on their own.

• Students’ appreciation of literacy Students’ appreciation of literacy genres might be measured genres might be measured objectivelyobjectively by how many samples of literacy genres by how many samples of literacy genres (fiction, biography, poetry) they (fiction, biography, poetry) they correctly name. correctly name.

• Students demonstrating understanding Students demonstrating understanding of the sounding out routine might be of the sounding out routine might be measured measured objectively objectively by the number of by the number of words in a paragraph students correctly words in a paragraph students correctly read within two minutes.read within two minutes.

Tested InstrumentsTested Instruments

Were instruments and measurement Were instruments and measurement methods tested for validity and reliability? methods tested for validity and reliability? For example, were the scores obtained For example, were the scores obtained from a new instrument compared to scores from a new instrument compared to scores obtained from another instrument known obtained from another instrument known to be valid and reliable? And were the to be valid and reliable? And were the scores highly correlated? If so, then the scores highly correlated? If so, then the new instrument is probably valid and new instrument is probably valid and reliable. reliable. [Criterion validity][Criterion validity] If a new If a new instrument is not validated there is no way instrument is not validated there is no way to know if it measures what it is supposed to know if it measures what it is supposed to measure and provides accurate to measure and provides accurate information.information.

SamplingSamplingIs the sample size, composition, and Is the sample size, composition, and selection/allocation of participants to selection/allocation of participants to groups appropriate for the type of study groups appropriate for the type of study and for how the findings might be used? and for how the findings might be used? Poor.Poor. The author claims that a teaching The author claims that a teaching method is effective. However, it was method is effective. However, it was used on only one class, or in one school. used on only one class, or in one school. There is no way to tell if it would be There is no way to tell if it would be effective anywhere else.effective anywhere else.Poor.Poor. The author claims that a method was not The author claims that a method was not effective. However, it was used on only effective. However, it was used on only one class, or in one school. There is no one class, or in one school. There is no way to tell if it would have been effective way to tell if it would have been effective elsewhere.elsewhere.

SamplingSampling

Poor.Poor.

The author claims that one method The author claims that one method worked better than another. worked better than another. However, the two comparison groups However, the two comparison groups were not created by random were not created by random assignment or by matching. assignment or by matching. Therefore, the groups may have been Therefore, the groups may have been different in other ways besides the different in other ways besides the method that is associated with method that is associated with higher achievement. higher achievement.

SamplingSampling

Better.Better. The researcher conducted a pilot study to The researcher conducted a pilot study to see if a method seems to work well enough see if a method seems to work well enough that it ought to be given a more valid test. that it ought to be given a more valid test. The author used a small sample (one class The author used a small sample (one class or school). This is not a representative or school). This is not a representative sample of the school population but that is sample of the school population but that is alright. [Do you want to use a method alright. [Do you want to use a method that may be harmful on a lot of kids?] The that may be harmful on a lot of kids?] The author claims that the findings are very author claims that the findings are very tentative and cannot be generalized tentative and cannot be generalized anywhere else. Further research is anywhere else. Further research is needed. needed.

SamplingSampling

Better.Better. The author has conducted research on the The author has conducted research on the same method in many different settings. same method in many different settings. Each new study is called a Each new study is called a replicationreplication. The . The sample of places is representative of the sample of places is representative of the population of schools. The author claims population of schools. The author claims that the data from most of the studies that the data from most of the studies suggest that the method is effective most suggest that the method is effective most of the time, but the author cannot figure of the time, but the author cannot figure out what factors hinder its effectiveness or out what factors hinder its effectiveness or what additional factors may be needed to what additional factors may be needed to make it work. The author advises caution make it work. The author advises caution in using the method. [This is honest and in using the method. [This is honest and morally responsible.] morally responsible.]

LongtitudinalLongtitudinalWas the research conducted for long Was the research conducted for long enough to reveal important processes?enough to reveal important processes?Poor.Poor. A project on a new teaching method was A project on a new teaching method was used for (two weeks, a month, a semester). used for (two weeks, a month, a semester). Students made significant gains in Students made significant gains in achievement between pre-test and post-achievement between pre-test and post-test. The author says that the findings are test. The author says that the findings are evidence that the method works. [The evidence that the method works. [The introduction of something new is often introduction of something new is often followed by beneficial change, especially followed by beneficial change, especially because students and teachers expect because students and teachers expect something to happen.] Do YOU plan to use something to happen.] Do YOU plan to use the method for such a short time? the method for such a short time?

LongtitudinalLongtitudinal

Better.Better.

A project on a new teaching was used for A project on a new teaching was used for a year. [Longitudinal research] For the a year. [Longitudinal research] For the first month or so, students made first month or so, students made significant gains. But progress slowed. significant gains. But progress slowed. By the end of the year, students receiving By the end of the year, students receiving the new method had gained no more than the new method had gained no more than students in control class. The authors students in control class. The authors conclude that there is no evidence that conclude that there is no evidence that the program works better than the the program works better than the current method.current method.

Level of MeasurementLevel of Measurement

Was the proper level of Was the proper level of measurement used? measurement used? It is best to It is best to measure with the highest degree of measure with the highest degree of precision or detail.precision or detail.Least detail.Least detail. You can measure at the You can measure at the nominal nominal levellevel. These are categories, such as . These are categories, such as student who is a “Visual learning student who is a “Visual learning style reader” “Auditory learning style reader” “Auditory learning style reader” or “Tactile/kinesthetic style reader” or “Tactile/kinesthetic learning style reader. learning style reader.


A little more detail. You can A little more detail. You can measure at the measure at the ordinal levelordinal level, , which implies which implies rank orderrank order..

• Proficient readerProficient reader• Emergent readerEmergent reader• Struggling readerStruggling reader• NonreaderNonreader


Most detail.Most detail. You can measure at the You can measure at the ratio levelratio level: real : real numbers. Objective data. Count numbers. Objective data. Count something everyone can see.something everyone can see.

• Some students in grade 3 read connected Some students in grade 3 read connected text 120 correct words per minute.text 120 correct words per minute.

• Other students in grade 3 read connected Other students in grade 3 read connected text 100 correct words per minute.text 100 correct words per minute.

• Still other students in grade 3 read Still other students in grade 3 read connected text less than 40 correct words connected text less than 40 correct words per minute.per minute.

Level of MeasurementLevel of MeasurementObviously you want to Obviously you want to measure what you measure what you are teachingare teaching. That’s the only information . That’s the only information that tells you if the program is doing what that tells you if the program is doing what it is supposed to do.it is supposed to do.If a program is teaching kids to read If a program is teaching kids to read connected text fluently, which measure connected text fluently, which measure would be more useful? [ratio level]would be more useful? [ratio level]

• The number of correct words per minute.The number of correct words per minute.• A child is ranked as a Proficient, A child is ranked as a Proficient,

Emergent, Struggling, or Nonreader. Emergent, Struggling, or Nonreader. [ordinal level][ordinal level]

• A child is labeled with one or another A child is labeled with one or another learning style? [nominal level]learning style? [nominal level]Hint: If reading benchmarks are Hint: If reading benchmarks are numerical (120 wcpm) then what kind of numerical (120 wcpm) then what kind of data do you need?data do you need?

Level of MeasurementLevel of MeasurementPoor.Poor. Researcher gives students a large sample Researcher gives students a large sample of math problems covered over the year, of math problems covered over the year, which ended with long division. She which ended with long division. She reports the data on each student as (1) reports the data on each student as (1) Little or No Retention; (2) Modest Little or No Retention; (2) Modest Retention; (3) Substantial Retention. Retention; (3) Substantial Retention. Then she presents the percentage of Then she presents the percentage of students in each category.students in each category.Does this tell readers what the students Does this tell readers what the students DO? DO? Does it tell the teacher how to increase Does it tell the teacher how to increase retention? [No. It’s pretty useless.]retention? [No. It’s pretty useless.]

Level of MeasurementLevel of MeasurementBetterBetter..

Researcher gives students a large sample of Researcher gives students a large sample of math problems covered over the year, which math problems covered over the year, which ended with long division. She reports the data ended with long division. She reports the data on each student as: (1) percentage of long on each student as: (1) percentage of long division problems correctly solved; (2) division problems correctly solved; (2) percentage of multiplication problems (a percentage of multiplication problems (a component of long division) correctly solved; (3) component of long division) correctly solved; (3) percentage of estimation problems (a component percentage of estimation problems (a component of long division) correctly solved; (4) percentage of long division) correctly solved; (4) percentage of subtraction problems (a component of long of subtraction problems (a component of long division) correctly solved; (5) percentage of division) correctly solved; (5) percentage of addition problems (a component of addition problems (a component of multiplication) correctly solved. She reports multiplication) correctly solved. She reports class data for each category. [This tells the class data for each category. [This tells the researcher exactly how much students retained researcher exactly how much students retained of the last thing taught, and all of its component of the last thing taught, and all of its component skills. She can determine which of these skills. She can determine which of these component skills are weak and therefore whether component skills are weak and therefore whether they might have been reviewed more often.] they might have been reviewed more often.]

Documents

Evaluating Research Claims Perfesser Plum 2012. Evaluating TV Ad Claims An ad on TV says, “USE Cocoa Creme. You’ll have youthful skin in 10 minutes. All