Assessing the Quality of Research Plans and Publications Assessing the Quality of Research Plans and Publications

Assessing the Quality Assessing the Quality of Research Plans and of Research Plans and

PublicationsPublications

Validity ----> Credibility ----> Validity ----> Credibility ----> UsabilityUsability

For Making Decisions About: For Making Decisions About: • Instructional Methods: Effective?Instructional Methods: Effective?• Program Materials: Effective?Program Materials: Effective?• Assessment Instruments and Assessment Instruments and

Methods: Measure what they are Methods: Measure what they are supposed to, reliably?supposed to, reliably?

• Data, findings: Accurate? Data, findings: Accurate? Generalizable?Generalizable?

Adequacy of the QuestionAdequacy of the QuestionIs the question, hypothesis, problem, or Is the question, hypothesis, problem, or interest stated precisely enough that interest stated precisely enough that things to measure objectively can be things to measure objectively can be derived from it? derived from it? Poor.Poor. “Do students learn more when “Do students learn more when tasks are authentic? [What does tasks are authentic? [What does ‘authentic’ mean? What does ‘learn ‘authentic’ mean? What does ‘learn more’ mean? Amount? Speed?]more’ mean? Amount? Speed?]Better.Better. “Is there a difference in the “Is there a difference in the number of learning trials required for number of learning trials required for students to learn to decode words that students to learn to decode words that are in common use (such as fun, car, are in common use (such as fun, car, cat, is, run) vs. pseudowords (fis, nif, cat, is, run) vs. pseudowords (fis, nif, ris)?”ris)?”

Coverage of the TopicCoverage of the Topic Poor.Poor. A new theory of reading is presented. A new theory of reading is presented. The only literature cited was done by the The only literature cited was done by the author and by persons sympathetic to author and by persons sympathetic to the author’s position (that a new theory the author’s position (that a new theory is needed). The author criticizes other is needed). The author criticizes other approaches to reading instruction approaches to reading instruction (except his own), but presents none of (except his own), but presents none of the extensive research that DOES the extensive research that DOES support other approaches and/or that support other approaches and/or that criticizes the author’s. In other words, criticizes the author’s. In other words, the author is misleading readers into the author is misleading readers into thinking that his approach is the best thinking that his approach is the best one.one.

Coverage of the TopicCoverage of the Topic

Better.Better.

A new theory of reading is presented. A new theory of reading is presented. The author identifies the other major The author identifies the other major approaches to reading instruction, and approaches to reading instruction, and presents a representative sample of presents a representative sample of the research on each approach—both the research on each approach—both research that supports and research research that supports and research that challenges these approaches. The that challenges these approaches. The author also presents research that author also presents research that supports and that challenges his own supports and that challenges his own approach.approach.

RelevanceRelevanceDid the literature review examine Did the literature review examine literature that is both directly relevant literature that is both directly relevant to the topic at hand and to larger to the topic at hand and to larger relevant issues? For example, did a relevant issues? For example, did a literature review on effective literature review on effective mathematics instruction include mathematics instruction include mathematics research and research on mathematics research and research on instruction in general?instruction in general?

Poor.Poor.

The literature review focuses narrowly The literature review focuses narrowly on the topic.on the topic.

RelevanceRelevance

Better.Better.

The literature review focuses on The literature review focuses on the topic and on other relevant the topic and on other relevant issues, such as the proper issues, such as the proper distribution of practice. distribution of practice.

Therefore, the reader can see that Therefore, the reader can see that the topic at hand is important on the topic at hand is important on both a smaller and larger scale.both a smaller and larger scale.

Useful SummaryUseful Summary

Did the writer summarize the literature Did the writer summarize the literature with a set of generalizations stated as with a set of generalizations stated as propositions and facts, and diagrams? propositions and facts, and diagrams? Did the writer identify possible gaps in Did the writer identify possible gaps in what is known?what is known?

Poor.Poor.

The literature review is mostly a cascade The literature review is mostly a cascade of supporting citations that gives the of supporting citations that gives the appearance of serious intent to conduct appearance of serious intent to conduct unbiased work. The literature does not unbiased work. The literature does not lead lead obviouslyobviously to particular research to particular research questions.questions.


Better.Better.

The literature review ends with a The literature review ends with a summary of what is known summary of what is known empiricallyempirically, partly known but not , partly known but not confirmed, and what is unknown. confirmed, and what is unknown. These These empiricalempirical findings are stated findings are stated such that it is clear exactly what the such that it is clear exactly what the writer is talking about and what writer is talking about and what needs to be studied.needs to be studied.


Poor.Poor.

““The more engaged students are the The more engaged students are the more they participate.” [Vague]more they participate.” [Vague]

Good.Good.

““The higher the rate of The higher the rate of opportunities to respond, the fewer opportunities to respond, the fewer times students become inattentive.” times students become inattentive.” [Objective, observable] [Objective, observable]

FeasibilityFeasibility

Is the project do-able, or feasible? Can Is the project do-able, or feasible? Can sufficient data be collected and analyzed?sufficient data be collected and analyzed?Poor.Poor. “We wish to determine the most “We wish to determine the most effective method for teaching effective method for teaching comprehension.” [It is impossible to comprehension.” [It is impossible to determine what IS most effective because determine what IS most effective because you cannot study everything that IS and you cannot study everything that IS and will be.]will be.]Better.Better. “We wish to determine which of “We wish to determine which of three methods for teaching three methods for teaching comprehension is associated with the comprehension is associated with the highest gains in scores of comprehension highest gains in scores of comprehension on the Especially Fine Test of Reading on the Especially Fine Test of Reading Comprehension.”Comprehension.”

FeasibilityFeasibilityPoor.Poor. ““Our objective is to determine whether Our objective is to determine whether violent images in the media cause violent violent images in the media cause violent behavior in schools. [You may learn that behavior in schools. [You may learn that students who commit more violent acts in students who commit more violent acts in school also watch more violent events on school also watch more violent events on TV, but you can never know if seeing those TV, but you can never know if seeing those events events causes causes the violence. Besides, there the violence. Besides, there are so many intervening variables and are so many intervening variables and contributing conditions that you could contributing conditions that you could never study all the connections.]never study all the connections.]Better.Better. ““Our objective is to determine whether Our objective is to determine whether students who commit more violent acts in students who commit more violent acts in school also watch more violent events on school also watch more violent events on TV.”TV.”

Null HypothesisNull Hypothesis

Is the research or report based on the Is the research or report based on the null hypothesis—an intellectually null hypothesis—an intellectually honest position? honest position?

““I believe program X works, but I’m I believe program X works, but I’m going to assume that it going to assume that it doesn’tdoesn’t work work and I’m going to collect data to try to and I’m going to collect data to try to show that it doesn’t workshow that it doesn’t work. If the data . If the data do NOT show that X does not work, do NOT show that X does not work, then I will conclude that then I will conclude that maybe it does maybe it does workwork. Maybe.” . Maybe.”


Or, on the contrary, is the Or, on the contrary, is the author is trying to persuade author is trying to persuade readers to join him in readers to join him in accepting a method, or accepting a method, or already believes he knows the already believes he knows the truth or knows what works, or truth or knows what works, or is not very interested in the is not very interested in the possibility that he is wrong?possibility that he is wrong?

Null HypothesisNull HypothesisPoor.Poor. ““Good readers do NOT sound out words. Good readers do NOT sound out words. They use context to They use context to predictpredict what words what words say. I will say. I will support support this statement with this statement with examples.” examples.”

The point is not to support what you The point is not to support what you believe but to TEST it—believe but to TEST it—to try to to try to FALSIFY it. Look for examples that do FALSIFY it. Look for examples that do NOT support the belief or hypothesis. NOT support the belief or hypothesis. Look for good readers who do NOT Look for good readers who do NOT guess. Anyone can find evidence to guess. Anyone can find evidence to support the stupidest ideas. Hitler support the stupidest ideas. Hitler could “prove” he was a kind person by could “prove” he was a kind person by reporting how often he petted his dog.reporting how often he petted his dog.


Poor.Poor.

An author only reports studies that show An author only reports studies that show one thing (that Program X is associated one thing (that Program X is associated with high achievement). The author with high achievement). The author concludes that Program X produces high concludes that Program X produces high achievement. However, there is some achievement. However, there is some research showing that Program X did not research showing that Program X did not produce high achievement. The author produce high achievement. The author did not report THAT research. [This is did not report THAT research. [This is called “cherry picking.” It is also called called “cherry picking.” It is also called not telling the whole truth.]not telling the whole truth.]

Null HypothesisNull HypothesisGood.Good. An author conducts research on Math An author conducts research on Math Program X in one school (suburban, middle Program X in one school (suburban, middle class). Kids who got Program X learned a lot class). Kids who got Program X learned a lot of math. The author says, “It probably won’t of math. The author says, “It probably won’t work again.” She tests the program again in work again.” She tests the program again in a very similar school. Same results. “Okay, a very similar school. Same results. “Okay, but it won’t work in a poor, urban school.” but it won’t work in a poor, urban school.” Same results. Okay, that was a fluke. It Same results. Okay, that was a fluke. It won’t happen again.” She tests it in another won’t happen again.” She tests it in another poor, urban school. Same results. “Okay, poor, urban school. Same results. “Okay, but it won’t work with Hispanic kids.” It but it won’t work with Hispanic kids.” It does. “Okay, but it won’t work with POOR does. “Okay, but it won’t work with POOR Hispanic kids.” It does. “Okay, but it won’t Hispanic kids.” It does. “Okay, but it won’t work in rural areas. It does. “Okay, maybe work in rural areas. It does. “Okay, maybe it’s wrong to say it doesn’t work. It makes it’s wrong to say it doesn’t work. It makes sense for a principal to try it.” [Replication]sense for a principal to try it.” [Replication]

Appropriate DesignAppropriate Design

Is the design (survey, experiment, Is the design (survey, experiment, ethnography) appropriate to the ethnography) appropriate to the question? question?

Let’s say the research question has to do Let’s say the research question has to do with what works better (what is with what works better (what is associated with better outcomes), or associated with better outcomes), or whether something worked. For example, whether something worked. For example, is one reading program better than is one reading program better than another? Does a math program produce another? Does a math program produce more skilled math students?more skilled math students?

Appropriate DesignAppropriate DesignPoor.Poor. Do a survey of teachers. Ask their Do a survey of teachers. Ask their opinions. [If so, you are measuring opinions. [If so, you are measuring teacher perception. You WANT to measure teacher perception. You WANT to measure student achievement. That requires that student achievement. That requires that you measure what students DO, not merely you measure what students DO, not merely what teachers think.]what teachers think.]

Poor.Poor. Have teachers make up their own Have teachers make up their own instruments to measure progress and instruments to measure progress and outcomes. [How will you know that these outcomes. [How will you know that these instruments measure what they are instruments measure what they are supposed to measure? Are reliable? Are supposed to measure? Are reliable? Are predictive? Are as useful as standardized predictive? Are as useful as standardized instruments? You won’t.]instruments? You won’t.]


Poor.Poor. Conduct ethnographic observations in Conduct ethnographic observations in classes. Observe student-teacher classes. Observe student-teacher interaction and students working together interaction and students working together and individually. and individually. [This may be interesting information, but it [This may be interesting information, but it tells you little about WHAT math or reading tells you little about WHAT math or reading skills students learned. It tells you about skills students learned. It tells you about the conditions in which they learned or the conditions in which they learned or didn’t learn. didn’t learn. You can’t use ethnographic You can’t use ethnographic research as the sole source of evaluative research as the sole source of evaluative data on methods and curricula.data on methods and curricula. You need You need to know what students learned and how to know what students learned and how much they learned.] much they learned.]

Appropriate DesignAppropriate DesignBetter.Better. You have an You have an experimental designexperimental design. You give . You give pre-tests, post-tests, and possibly pre-tests, post-tests, and possibly measures of progress in between. You use measures of progress in between. You use instruments that have already been instruments that have already been validated by research, and that measure validated by research, and that measure student learning—specifically, the accuracy student learning—specifically, the accuracy and speed of solving math problems, or and speed of solving math problems, or reading connected text. If you are reading connected text. If you are comparing one program with another, you comparing one program with another, you have comparison classes that are as have comparison classes that are as similar as possible so that other variables similar as possible so that other variables (such as help from parents, differences in (such as help from parents, differences in students’ abilities) do not contribute to the students’ abilities) do not contribute to the outcomes. Observers are trained to use outcomes. Observers are trained to use the instruments and the reliability of their the instruments and the reliability of their measures is determined.measures is determined.

Experimental DesignsExperimental DesignsClassic Design: Pre-test, Post-Classic Design: Pre-test, Post-Test, Control Group, with Random Test, Control Group, with Random AllocationAllocation

EE

C C Pre-test Post-testPre-test Post-test

However, randomization may not be However, randomization may not be possible. Some students are denied possible. Some students are denied effective instruction.effective instruction.

Experimental DesignsExperimental DesignsTime SeriesTime Series

99 00 01 02 03 04 05 06 07 0899 00 01 02 03 04 05 06 07 08

Eclectic Reading First Eclectic Reading First Percentage PassingPercentage Passing

No no-treatment control group. No no-treatment control group. However, samples will vary from year to However, samples will vary from year to year.year.

Experimental DesignsExperimental Designs

Interrupted Time Series. Same Interrupted Time Series. Same Group, Reversal of Treatment Group, Reversal of Treatment and Baselineand Baseline Eclectic Eclectic RF Eclectic RF Eclectic RFRF

A1 B1 A2 A1 B1 A2 B2B2

Appropriate DesignAppropriate DesignRule.Rule. If you want to If you want to describedescribe what is what is happening at one point in time (for happening at one point in time (for example, the percentage of students example, the percentage of students in each grade level that pass state in each grade level that pass state tests in reading—a tests in reading—a cross-sectioncross-section) you ) you could do a SURVEY (literally, over could do a SURVEY (literally, over view) of a sample of schools that view) of a sample of schools that represent the population of schools represent the population of schools in the state (e.g., some urban, some in the state (e.g., some urban, some suburban, some rural; some wealthy, suburban, some rural; some wealthy, some middle class, some poor). some middle class, some poor).


These survey data might These survey data might show show relationshipsrelationships; for example, the higher ; for example, the higher the social class, the higher the the social class, the higher the percentage of students who pass. But percentage of students who pass. But that does not show that social class is that does not show that social class is the the causecause of achievement. It could be of achievement. It could be that parental education is part of the that parental education is part of the cause; educated parents can teach cause; educated parents can teach their own children, or wealthier their own children, or wealthier schools have better teachers. schools have better teachers.


If you want to find out what people think If you want to find out what people think and feel, ask them, using interviews or and feel, ask them, using interviews or questionnaires. This information may be questionnaires. This information may be useful in making certain decisions. For useful in making certain decisions. For example, teachers using a new reading example, teachers using a new reading program may say that they do not like it, or program may say that they do not like it, or say that they need more assistance. Still, say that they need more assistance. Still, these opinions cannot be used to judge these opinions cannot be used to judge whether a program is well-designed or is whether a program is well-designed or is effective.effective. That is a different variable and That is a different variable and would have to be measured in terms of would have to be measured in terms of student achievement by teachers who used student achievement by teachers who used the program properly. the program properly.

Causal RelationshipsCausal Relationships

If the research is assessing If the research is assessing causal relationships (e.g., causal relationships (e.g., program effectiveness) did the program effectiveness) did the researchers satisfy the researchers satisfy the conditions for making credible conditions for making credible causal inferences? Did they:causal inferences? Did they:

Causal RelationshipsCausal Relationships• Determine that change in the outcome Determine that change in the outcome

variables (e.g., achievement)variables (e.g., achievement) followed followed change in the input variables change in the input variables (introduction of the program)? This (introduction of the program)? This requires a requires a pre-test pre-test and perhapsand perhaps repeatedrepeated or or alternative measuresalternative measures..

• Collect data on both the outcome Collect data on both the outcome variables (e.g., achievement) and the variables (e.g., achievement) and the input variables (e.g., whether, how often, input variables (e.g., whether, how often, and how well teachers used the and how well teachers used the program)? Otherwise, if data show that program)? Otherwise, if data show that achievement is low, you might conclude achievement is low, you might conclude the program is ineffective when in fact the program is ineffective when in fact teachers did not use it properly.teachers did not use it properly.

• Rule out the effects of extraneous Rule out the effects of extraneous factors, such as maturation, other factors, such as maturation, other sources of instruction, unreliable sources of instruction, unreliable measurement?measurement?

DefinitionsDefinitionsOften, Often,

• When When definitions and measures definitions and measures are vagueare vague (you don’t know exactly (you don’t know exactly what you are supposed to see), what you are supposed to see),

• When When measures are subjectivemeasures are subjective (students enjoy, appreciate, (students enjoy, appreciate, demonstrate, understand), and demonstrate, understand), and

• When When data collection involves data collection involves writing narrative and impressions writing narrative and impressions rather than countingrather than counting what what students and teacher do…students and teacher do…it serves the function of disguising it serves the function of disguising bias, ignorance, and agenda bias, ignorance, and agenda behind a façade of “science.”behind a façade of “science.”

DefinitionsDefinitions

Weak, vague, imprecise, and too-broad Weak, vague, imprecise, and too-broad definitions foster or support definitions foster or support unaccountability.unaccountability.

How can you be held accountable if you How can you be held accountable if you never say exactly what you do and what never say exactly what you do and what students are supposed to learn?students are supposed to learn?

It ensures that beliefs and biases will It ensures that beliefs and biases will always be supported by evidence because always be supported by evidence because when definitions are vague and general, when definitions are vague and general, and data are subjective, and data are subjective, almost anything almost anything can count as supporting evidencecan count as supporting evidence..


Were adequate Were adequate conceptual definitionsconceptual definitions (from which measures were derived) (from which measures were derived) used? That is, did conceptual definitions used? That is, did conceptual definitions cover all relevant features of a concept cover all relevant features of a concept and exclude irrelevant features? and exclude irrelevant features?

Poor.Poor.

Reading is defined as a psycholinguistic Reading is defined as a psycholinguistic guessing game. Therefore, researchers guessing game. Therefore, researchers measure student guessing as evidence of measure student guessing as evidence of reading. Is guessing what is meant by reading. Is guessing what is meant by reading?reading?


Poor.Poor.

Reading is defined as making sense of Reading is defined as making sense of text, or “meaning-making.” [In other text, or “meaning-making.” [In other words, reading is wrongly defined not by words, reading is wrongly defined not by what readers DO but by what reading what readers DO but by what reading accomplishes.] Therefore, researchers will accomplishes.] Therefore, researchers will only measure comprehension. If a teacher only measure comprehension. If a teacher is not yet working on comprehension (but is not yet working on comprehension (but IS working on other reading skills) IS working on other reading skills) students will by that narrow definition be students will by that narrow definition be considered nonreaders.considered nonreaders.

DefinitionsDefinitionsBetter.Better.Reading is a cognitive routine for Reading is a cognitive routine for accurately and rapidly decoding written accurately and rapidly decoding written text into words and connected statements, text into words and connected statements, and then comprehending the definitions and then comprehending the definitions and propositions communicated by the and propositions communicated by the text. text. [This definition covers all the reading [This definition covers all the reading skills (the correspondence between letters skills (the correspondence between letters and sounds, sounding out words, reading and sounds, sounding out words, reading words and sentences fluently, and knowing words and sentences fluently, and knowing vocabulary and comprehension vocabulary and comprehension strategies). Research based on this strategies). Research based on this definition would be obliged either to study definition would be obliged either to study all of what is meant by reading or to all of what is meant by reading or to explicitly limit the research to certain explicitly limit the research to certain subskills.] subskills.]


Where adequate Where adequate operational operational definitionsdefinitions used? Operational used? Operational definitions are supposed to be definitions are supposed to be derived from conceptual derived from conceptual definitions. In addition, definitions. In addition, operational definitions should operational definitions should provide clear examples that cover provide clear examples that cover the range of what is implied by the the range of what is implied by the conceptual definition, and should conceptual definition, and should exclude what is not relevant.exclude what is not relevant.

DefinitionsDefinitionsPoor.Poor. Reading is (properly) conceptually defined Reading is (properly) conceptually defined as a cognitive routine for accurately and as a cognitive routine for accurately and rapidly decoding written text into words rapidly decoding written text into words and connected statements and then and connected statements and then comprehending the definitions and comprehending the definitions and propositions communicated by the text. propositions communicated by the text. However, the operational definition of However, the operational definition of reading includes how children handle reading includes how children handle books (upright, turn pages), name the books (upright, turn pages), name the parts of a book, and recognize parts of a book, and recognize environmental print. These may be environmental print. These may be important, but important, but they are NOT part of the they are NOT part of the conceptual definition of readingconceptual definition of reading. . Therefore, kids whose reading is measured Therefore, kids whose reading is measured according to this definition may get high according to this definition may get high scores even though they are not reading.scores even though they are not reading.


Better.Better.

The researchers state that they will not The researchers state that they will not measure all aspects of reading. They are measure all aspects of reading. They are interested (in this study) only in students interested (in this study) only in students accurately and rapidly decoding words. accurately and rapidly decoding words. They define decoding words this way. “The They define decoding words this way. “The student says each sound in a word, does student says each sound in a word, does not stop between the sounds, and says the not stop between the sounds, and says the word as a unit (blends sounds into a word as a unit (blends sounds into a whole).”whole).” This operational definition yields This operational definition yields objective, observable measures.objective, observable measures.


Were Were objective measuresobjective measures derived derived from the operational definitions? from the operational definitions? Decisions about which variables Decisions about which variables affect other variables, or about affect other variables, or about whether a program or method is whether a program or method is effective, are so important to effective, are so important to children’s lives that measures must children’s lives that measures must be objective; that is, be objective; that is, any any observer observer can SEE what is being measured. can SEE what is being measured.

DefinitionsDefinitionsPoor measures.Poor measures.

• ““Students enjoyment of reading.” [How Students enjoyment of reading.” [How many times they smile?]many times they smile?]

• ““Students’ appreciation of literary Students’ appreciation of literary genres.” [Students say Thanks to a genres.” [Students say Thanks to a poem?]poem?]

• ““Students’ demonstrating understanding Students’ demonstrating understanding of the sounding out routine?” [They use of the sounding out routine?” [They use hand puppets?]hand puppets?]

If a researcher doesn’t define variables in If a researcher doesn’t define variables in terms of what persons DO, then the terms of what persons DO, then the researcher probably doesn’t know WHAT researcher probably doesn’t know WHAT he’s talking about.he’s talking about.

DefinitionsDefinitionsBetter measures.Better measures.

• Enjoyment of reading might be Enjoyment of reading might be measured objectively (indirectly) by measured objectively (indirectly) by HOW MANY books or how much time HOW MANY books or how much time students read on their own. students read on their own.

• Students’ appreciation of literacy genres Students’ appreciation of literacy genres might be measured might be measured objectivelyobjectively by how by how many samples of literacy genres (fiction, many samples of literacy genres (fiction, biography, poetry) they correctly name. biography, poetry) they correctly name.

• Students demonstrating understanding Students demonstrating understanding of the sounding out routine might be of the sounding out routine might be measured measured objectively objectively by the number of by the number of words in a paragraph students correctly words in a paragraph students correctly read within two minutes.read within two minutes.

Tested InstrumentsTested Instruments

Were instruments and measurement Were instruments and measurement methods tested for validity and methods tested for validity and reliability? For example, were the scores reliability? For example, were the scores obtained from a new instrument obtained from a new instrument compared to scores obtained from compared to scores obtained from another instrument known to be valid and another instrument known to be valid and reliable? And were the scores highly reliable? And were the scores highly correlated? If so, then the new correlated? If so, then the new instrument is probably valid and reliable. instrument is probably valid and reliable. [Criterion validity][Criterion validity] If a new instrument is If a new instrument is not validated there is no way to know if it not validated there is no way to know if it measures what it is supposed to measure measures what it is supposed to measure and provides accurate information.and provides accurate information.

SamplingSamplingIs the sample size, composition, and Is the sample size, composition, and selection/allocation of participants to selection/allocation of participants to groups appropriate for the type of study groups appropriate for the type of study and for how the findings might be used? and for how the findings might be used? Poor.Poor. The author claims that a teaching The author claims that a teaching method is effective. However, it was method is effective. However, it was used on only one class, or in one school. used on only one class, or in one school. There is no way to tell if it would be There is no way to tell if it would be effective anywhere else.effective anywhere else.Poor.Poor. The author claims that a method was not The author claims that a method was not effective. However, it was used on only effective. However, it was used on only one class, or in one school. There is no one class, or in one school. There is no way to tell if it would have been effective way to tell if it would have been effective elsewhere.elsewhere.

SamplingSampling

Poor.Poor.

The author claims that one method The author claims that one method worked better than another. worked better than another. However, the two comparison groups However, the two comparison groups were not created by random were not created by random assignment or by matching. assignment or by matching. Therefore, the groups may have been Therefore, the groups may have been different in other ways besides the different in other ways besides the method that is associated with method that is associated with higher achievement. higher achievement.

SamplingSampling

Better.Better. The researcher conducted a pilot study to The researcher conducted a pilot study to see if a method seems to work well enough see if a method seems to work well enough that it ought to be given a more valid test. that it ought to be given a more valid test. The author used a small sample (one class The author used a small sample (one class or school). This is not a representative or school). This is not a representative sample of the school population but that is sample of the school population but that is alright. [Do you want to use a method alright. [Do you want to use a method that may be harmful on a lot of kids?] The that may be harmful on a lot of kids?] The author claims that the findings are very author claims that the findings are very tentative and cannot be generalized tentative and cannot be generalized anywhere else. Further research is anywhere else. Further research is needed. needed.

SamplingSampling

Better.Better. The author has conducted research on the The author has conducted research on the same method in many different settings. same method in many different settings. Each new study is called a Each new study is called a replicationreplication. The . The sample of places is representative of the sample of places is representative of the population of schools. The author claims population of schools. The author claims that the data from most of the studies that the data from most of the studies suggest that the method is effective most suggest that the method is effective most of the time, but the author cannot figure of the time, but the author cannot figure out what factors hinder its effectiveness or out what factors hinder its effectiveness or what additional factors may be needed to what additional factors may be needed to make it work. The author advises caution make it work. The author advises caution in using the method. [This is honest and in using the method. [This is honest and morally responsible.] morally responsible.]

LongtitudinalLongtitudinalWas the research conducted for long Was the research conducted for long enough to reveal important processes?enough to reveal important processes?Poor.Poor. A project on a new teaching method was A project on a new teaching method was used for (two weeks, a month, a semester). used for (two weeks, a month, a semester). Students made significant gains in Students made significant gains in achievement between pre-test and post-achievement between pre-test and post-test. The author says that the findings are test. The author says that the findings are evidence that the method works. [The evidence that the method works. [The introduction of something new is often introduction of something new is often followed by beneficial change, especially followed by beneficial change, especially because students and teachers expect because students and teachers expect something to happen.] Do YOU plan to use something to happen.] Do YOU plan to use the method for such a short time? the method for such a short time?

LongtitudinalLongtitudinal

Better.Better.

A project on a new teaching was used for A project on a new teaching was used for a year. [Longitudinal research] For the a year. [Longitudinal research] For the first month or so, students made first month or so, students made significant gains. But progress slowed. significant gains. But progress slowed. By the end of the year, students receiving By the end of the year, students receiving the new method had gained no more than the new method had gained no more than students in control class. The authors students in control class. The authors conclude that there is no evidence that conclude that there is no evidence that the program works better than the the program works better than the current method.current method.

Level of MeasurementLevel of Measurement

Was the proper level of Was the proper level of measurement used? measurement used? It is best to It is best to measure with the highest degree of measure with the highest degree of precision or detail.precision or detail.Least detail.Least detail. You can measure at the You can measure at the nominal nominal levellevel. These are categories, such as . These are categories, such as student who is a “Visual learning student who is a “Visual learning style reader” “Auditory learning style reader” “Auditory learning style reader” or “Tactile/kinesthetic style reader” or “Tactile/kinesthetic learning style reader. learning style reader.


A little more detail. You can A little more detail. You can measure at the measure at the ordinal levelordinal level, , which implies which implies rank orderrank order..

• Proficient readerProficient reader• Emergent readerEmergent reader• Struggling readerStruggling reader• NonreaderNonreader


Most detail.Most detail. You can measure at the You can measure at the ratio levelratio level: real : real numbers. Objective data. Count numbers. Objective data. Count something everyone can see.something everyone can see.

• Some students in grade 3 read connected Some students in grade 3 read connected text 120 correct words per minute.text 120 correct words per minute.

• Other students in grade 3 read connected Other students in grade 3 read connected text 100 correct words per minute.text 100 correct words per minute.

• Still other students in grade 3 read Still other students in grade 3 read connected text less than 40 correct words connected text less than 40 correct words per minute.per minute.

Level of MeasurementLevel of MeasurementObviously you want to Obviously you want to measure what you measure what you are teachingare teaching. That’s the only information . That’s the only information that tells you if the program is doing what that tells you if the program is doing what it is supposed to do.it is supposed to do.If a program is teaching kids to read If a program is teaching kids to read connected text fluently, which measure connected text fluently, which measure would be more useful? [ratio level]would be more useful? [ratio level]

• The number of correct words per minute.The number of correct words per minute.• A child is ranked as a Proficient, A child is ranked as a Proficient,

Emergent, Struggling, or Nonreader. Emergent, Struggling, or Nonreader. [ordinal level][ordinal level]

• A child is labeled with one or another A child is labeled with one or another learning style? [nominal level]learning style? [nominal level]Hint: If reading benchmarks are Hint: If reading benchmarks are numerical (120 wcpm) then what kind of numerical (120 wcpm) then what kind of data do you need?data do you need?

Level of MeasurementLevel of MeasurementPoor.Poor. Researcher gives students a large sample Researcher gives students a large sample of math problems covered over the year, of math problems covered over the year, which ended with long division. She which ended with long division. She reports the data on each student as (1) reports the data on each student as (1) Little or No Retention; (2) Modest Little or No Retention; (2) Modest Retention; (3) Substantial Retention. Retention; (3) Substantial Retention. Then she presents the percentage of Then she presents the percentage of students in each category.students in each category.Does this tell readers what the students Does this tell readers what the students DO? DO? Does it tell the teacher how to increase Does it tell the teacher how to increase retention? [No. It’s pretty useless.]retention? [No. It’s pretty useless.]

Level of MeasurementLevel of MeasurementBetterBetter..

Researcher gives students a large sample of Researcher gives students a large sample of math problems covered over the year, which math problems covered over the year, which ended with long division. She reports the data ended with long division. She reports the data on each student as: (1) percentage of long on each student as: (1) percentage of long division problems correctly solved; (2) division problems correctly solved; (2) percentage of multiplication problems (a percentage of multiplication problems (a component of long division) correctly solved; (3) component of long division) correctly solved; (3) percentage of estimation problems (a component percentage of estimation problems (a component of long division) correctly solved; (4) percentage of long division) correctly solved; (4) percentage of subtraction problems (a component of long of subtraction problems (a component of long division) correctly solved; (5) percentage of division) correctly solved; (5) percentage of addition problems (a component of addition problems (a component of multiplication) correctly solved. She reports multiplication) correctly solved. She reports class data for each category. [This tells the class data for each category. [This tells the researcher exactly how much students retained researcher exactly how much students retained of the last thing taught, and all of its component of the last thing taught, and all of its component skills. She can determine which of these skills. She can determine which of these component skills are weak and therefore whether component skills are weak and therefore whether they might have been reviewed more often.] they might have been reviewed more often.]

Documents

Assessing the Quality of Research Plans and Publications Assessing the Quality of Research Plans and Publications