Reviewing Related Research (R3) Workbook - …meaningfulevidence.com/.../Reviewing-Related-Research-R3-Workbo… · Reviewing Related Research (R3) Workbook © Meaningful Evidence,

© Meaningful Evidence, LLC, 2016. Reprint with permission only | www.meaningfulevidence.com

Bernadette Wright, PhD & Ladel Lewis, PhD MEANINGFUL EVIDENCE, LLC | WWW.MEANINGFULEVIDENCE.COM

OCTOBER, 2016

Reviewing Related Research (R3) Workbook

http://www.meaningfulevidence.com/

© Meaningful Evidence, LLC, 2016. Reprint with permission only | www.meaningfulevidence.com i

CONTENTS

About this Workbook ...................................................................................... 1

Step 1. Develop Relevant R3 Questions ............................................................. 2

Develop Relevant R3 Questions Worksheet ........................................................ 3

Step 2: Determine Inclusion Criteria ................................................................. 4

Inclusion Criteria Worksheet ............................................................................ 8

Step 3. Determine Places to Search .................................................................. 9

Places to Search Worksheet .......................................................................... 13

Step 4. Develop Search Terms ....................................................................... 14

Search Terms Worksheet .............................................................................. 15

Step 5. Search and Select Studies .................................................................. 16

Study Selection Worksheet ............................................................................ 18

Step 6. Extract Relevant Information from Studies ........................................... 19

Study Information Worksheet ........................................................................ 23

Step 7. Evaluate the Relevance and Quality of Studies ...................................... 24

Study Quality and Relevance Assessment Worksheet ........................................ 26

Step 8. Synthesize Relevant Information from Across Studies ............................ 27

Step 9. Clearly Report How You Analyzed the Studies ....................................... 28

© Meaningful Evidence, LLC, 2016. Reprint with permission only | www.meaningfulevidence.com 1

ABOUT THIS WORKBOOK

This workbook is a tool that you can use to plan a rigorous and useful review of

related research (R3). You can use the step-by-step, fill-in-the-blank model we

provide in this workbook to conduct a R3 as part of a larger evaluation project or as

a stand-alone R3 study.

You can use this workbook for any research activity that involves looking at what

others have done, using existing related studies and materials. Reviews of related

research (R3) go by many terms, such as literature review, systematic review,

review, evaluation synthesis, just to name a few. R3s also include research that is

not always clearly labeled as a review, such as…

Introduction or background sections of academic papers—the part where the

authors talk about what past related studies have found and why their new

research is needed

College papers that involve finding and using existing studies that provide

information on your topic and summarizing your results

Issue papers produced by think tanks that describe a public policy issue

based on the latest available research

For best results, we recommend that you collaborate closely with evaluation

customers and other stakeholders, such as an advisory group or evaluation

workgroup, in every step of completing this workbook.

If possible, it is also a good idea to work with a librarian, for assistance with your

search.


STEP 1. DEVELOP RELEVANT R3 QUESTIONS

The first step in planning an effective review of related research is to develop

relevant questions. Because all the other steps of your review—from searching and

selecting studies to analyzing and reporting the data—are driven by your review

questions.

A review of related research can address any evaluation question—any question

that managers and funders want answered.

Your review questions are driven by your evaluation purpose, by what you want to

know. To develop your review questions, look at your evaluation purpose and break

it down into specific topics that a review could explore. Each question should

address one specific topic that you will explore using existing research and

materials.

Your review questions are not written in stone. You can (and should) make changes

to your questions when needed to get useful results. Your initial search results may

suggest new sub-questions to explore. Or you may find that you need to narrow

your question to find more relevant studies, or broaden your question to get

enough studies. To be systematic, document and report your initial questions and

any changes you make to your questions and why.

R3 QUESTIONS EXAMPLE

Evaluation Purpose

You’re involved with a community organization in Charlotte, North Carolina that is

working to help improve race relations and promote racial equity in the city. At a

board meeting, someone suggests the idea of an ”interracial dialogue group”

programs (also called “study circles”) in which small groups meet to discuss and

take action to address race relations in their community. The board wants to know

what the research evidence for these programs is. Have they shown evidence that

they help improve race relations and increase racial equity? What are the effective

strategies to make them successful?

Example Review of Related Research Questions

Based on this purpose, two R3 questions might be:

1) What effects on race relations and racial equity have interracial dialogue

group programs shown?

2) What are the effective strategies for making interracial dialogue group

programs successful?


DEVELOP RELEVANT R3 QUESTIONS WORKSHEET

Look at the purpose of your specific research project. Develop one or more review

questions from your research purpose. Enter your specific R3 questions in the

spaces below, one question per number. The number of questions your review

needs to address will depend on your purpose—what you want to know.

Add a new page as needed if you have more questions.

1) _____________________________________________________________

____________________________________________________________?

2) _____________________________________________________________

____________________________________________________________?

3) _____________________________________________________________

____________________________________________________________?

4) _____________________________________________________________

____________________________________________________________?

5) _____________________________________________________________

____________________________________________________________?

6) _____________________________________________________________

____________________________________________________________?

7) _____________________________________________________________

____________________________________________________________?

8) _____________________________________________________________

____________________________________________________________?


STEP 2: DETERMINE INCLUSION CRITERIA

The next step is to determine your criteria for including or excluding studies. Below

are some criteria that are important in many reviews of related research.

STUDY ABILITY TO ADDRESS YOUR RESEARCH QUESTIONS

Include studies that provide information that can address your review questions

that you formulated in step 1.

STUDY POPULATION

You might select studies that focus on the specific population that you’re interested

in, such as studies involving older persons with disabilities living at home, women

small business owners, or high school students in public schools. For our dialogue

group example, we might focus on studies involving participants in dialogue groups

on race in the United States. For a broader review, we might look at related studies

involving other populations; for example, studies about Jewish-Palestinian dialogue

groups in Israel might provide useful lessons for dialogue groups on race in the

United States. For a more narrow review, we might target studies specifically

involving participants in community-based dialogue groups (rather than school-

based dialogues limited to students).

STUDY GEOGRAPHIC AREA

You might select studies that focus on a specific geographic area, such as a

particular city, state, or country, or urban, suburban, or rural areas. For our

dialogue group example, we might focus on dialogue groups based in the U.S., as

they are most relevant to our situation.

TYPE OF INTERVENTION

When your question is about a specific type of intervention (e.g. dialogue

groups/study circles), you may want to narrow your review to studies about that

type of intervention. For comparison, you might also review studies on other

alternative interventions to achieve the same goal, to see how the intervention

you’re considering compares with other options. On the other hand, when your

question is to explore the effective ways to accomplish a certain goal (e.g. what are

the effective interventions to promote racial equity?), then you would keep your

selection of studies open to studies of any type of intervention.


INTERVENTION SETTING

Another inclusion criteria that is often important is the type of setting where the

intervention takes place. For a review with the purpose of informing decisions about

dialogue groups hosted by your community organization, for example, you might

examine studies of programs based in community organizations and open to

community members, excluding studies of programs based in schools and limited to

students.

On the other hand, the experiences of related interventions in other settings may

provide relevant lessons for your situation. How narrowly or broadly you focus your

review will depend on the scope, timeline, and budget of your review and how

many studies you find specific to your situation.

DATE OF PUBLICATION

The importance of the date of publication of studies depends on your specific

question and how fast the information is changing. For example, if you’re looking at

studies on programs that involve online meetings, studies from 15 years ago may

not apply because technologies have changed and internet use has increased.

Decades old studies about race relations interventions may also not be

generalizable to today, because social attitudes and awareness about racial issues

have likely changed. However, older studies can often provide some valuable

insights from previous attempts to solve the problem.

TYPE OF MATERIAL

Some reviews of related research include only articles from scholarly peer-reviewed

journals. However, a review can include relevant materials of any kind—scholarly

journal articles, books, conference presentations, and research reports. New

technologies have provided researchers with more opportunities to disseminate

their research results in webinars, blogs, online videos, and other formats. Don’t

exclude sources by the type of publication unless you have a good reason to do so

(e.g. if your research question is specifically to understand trends in what’s being

published in peer-reviewed journals).

TYPE OF STUDY DESIGN

A review of related research can include relevant studies of any type of design and

methods. Often, you’ll need to include studies of many designs to answer all your

R3 questions. These may include, for example, studies using interviews and focus

groups with stakeholders and experts, surveys, statistical analyses of existing

datasets, previous reviews of related research, comparison group studies, and other


designs. Do not exclude studies by the type of design unless your research question

is specific to a particular type of design.

PUBLICATION LANGUAGE

Another consideration is whether you will include only publications written in one

language (e.g. English), or studies published in more than one language. The

language of studies that are important to include will depend on your R3 scope and

questions.


INCLUSION CRITERIA EXAMPLE

Below are example inclusion criteria for the example R3 scenario described on page

2.

Inclusion

Criteria

Types of Studies We Will Include

Study ability

to address R3

questions

Studies that can provide information that addresses either or

both of our review of related research questions:

1) What effects on race relations and racial equity have interracial dialogue group programs shown?

2) What effective strategies for making interracial dialogue

group programs successful have studies identified?

Study

population

Participants in dialogue groups/study circles on race in the

United States

Study

geographic

area

Studies based in the United States, especially any studies

based in the Charlotte or in North Carolina.

Date of

publication

We’ll search for studies published within the past 10 years.

However, if we come across any particularly noteworthy or influential older studies, we’ll include those as well.

Also, if we find that few studies have been published on the topic in the past 10 years, then we’ll expand our search to

include older studies (e.g. from the past 20 years).

Type of

material

Any type of material (journal articles, reports, conference

presentations, books, etc.)

Type of study design

Studies of any type of design

Publication

language

English


INCLUSION CRITERIA WORKSHEET

In the second column of the table below, enter your criteria for selecting materials

to include in your review of related research. For some rows, you might narrow

your review to meet certain criteria (e.g. studies in English), and for other rows you

might include any study regardless of that criteria (e.g. studies of any design).

Inclusion

Criteria Types of Studies We Will Include

Study ability to

address R3 questions

Study

population

Type of

intervention

Intervention

setting

Study

geographic area

Date of

publication

Type of

material

Type of study

design

Publication

language

Other (specify)


STEP 3. DETERMINE PLACES TO SEARCH

Below are eight good places to look for related research and materials. Many of

these places are free or low-cost to search (some materials may cost money to

access full-text).

1. YOUR OWN BOOKSHELF

For some questions, you (or others at your organization) may already know of

some sources. These may include related professional magazine articles,

conference presentations, books, and other sources you’ve come across. Don’t

overlook what might be right under your nose.

Tip: Ask your project team members, staff, and colleagues with knowledge of the

topic what sources they think may be helpful.

2. THE INTERNET

In searching the internet, consider using more than one search engine, such as

Google (https://www.google.com/), Yahoo Search (https://search.yahoo.com/),

Bing (http://www.bing.com/) , Duck Duck Go (https://duckduckgo.com/), and

Baidu (http://www.baidu.com/). Different search engines may yield different

results.

Tips: I

Google’s advanced search

(https://support.google.com/websearch/answer/35890) lets you apply

various filters to narrow your search. You can require that certain words be

included, or an exact phrase, or certain words not included. You can also

narrow results to a particular language or region and apply other filters.

To master Google search techniques, check out Google’s Search Tips

(https://support.google.com/websearch/answer/134479?hl=en).

3. KEY WEBSITES

Another technique is to browse or search the websites of select leading professional

associations, government agencies, and other organizations that are active on your

topic. Organizations and agencies often provide publications and research on their

websites.

https://www.google.com/

https://search.yahoo.com/

http://www.bing.com/

https://duckduckgo.com/

http://www.baidu.com/

https://support.google.com/websearch/answer/35890

https://support.google.com/websearch/answer/134479?hl=en


4. SCHOLARLY LITERATURE DATABASES

Google Scholar (https://scholar.google.com/) is a handy tool to search for academic

research articles. Searching Google Scholar is free to anyone with an internet

connection. Many of the indexed papers require a subscription or cost to download,

and many of them are available to download at no charge.

A few additional, free-to-search sites for finding scholarly publications are

Academia.edu (https://www.academia.edu/), ResearchGate

(https://www.researchgate.net/), and SSRN (Social Science Research Network,

https://www.ssrn.com/en/).

Tip: Similar to Google’s advanced search, Google Scholar’s advanced search lets

you apply various filters to narrow your search results, to focus on what’s most

relevant. Check out Google Scholar’s search tips here:

http://scholar.google.com/intl/en/scholar/help.html

For a more comprehensive search, consider subscription databases and materials

(articles, books) that may require a paid subscription to search. Examples include

Scopus, CINAHL (Cumulative Index to Nursing and Allied Health), Social Science

Citation Index, and ProQuest Dissertations & Theses Database, just to name a few.

Tips:

If you’re affiliated with a college or university, as a student, professor, or

employee, you probably have access to subscription databases available at

your institution’s library.

Some universities also have programs where anyone can pay to become a

member of the university library. That may be a worthwhile if you’ll need a

large number of articles that are not available for free, or if you want to

search scholarly databases that cost money to search.

5. YOUR SOURCES’ SOURCES

Once you’ve gathered a few helpful articles, reports, and other items, looking at

what sources those sources mention can lead to more relevant materials.

Tip: As you review related publications, check to see what sources they mention in

their discussion of background information, footnotes/endnotes, and references or

bibliographies. Look up any sources that sound relevant to your project.

https://scholar.google.com/

https://www.academia.edu/

https://www.researchgate.net/

https://www.ssrn.com/en/

http://scholar.google.com/intl/en/scholar/help.html


6. ASKING RESEARCHERS AND EXPERTS

You can contact study authors and other researchers and experts in the field and

ask them about what materials they recommend, studies underway, and any

studies underway or planned new research. Or you might ask many researchers at

once by posting a question via an email discussion list or online forum for

researchers in your field. Researchers are often glad to be asked about their

research and happy to answer questions. This can be a great way to find out about

studies that are not yet published or hard-to-find studies. Asking other researchers

what studies they recommend is not a substitute for doing your own research.

7. ONLINE REPOSITORIES ON SPECIFIC TOPICS

Many online repositories provide opportunities to search for information on a

specific topic. Examples include the ERIC (http://eric.ed.gov) collection of research

and scholarship related to education and Health Systems Evidence

(https://www.healthsystemsevidence.org/), a repository of syntheses of research

evidence about health systems, both of which are free to search. Many subscription

databases of scholarly literature also focus on specific topics.

8. KEY JOURNALS

Another approach is to browse or search key journals that are likely to have

published relevant studies on your topic. Once you’ve found some initial studies to

include in your review, you can look at what journals they were published in, then

explore those journals to look for additional studies.

http://eric.ed.gov/

https://www.healthsystemsevidence.org/


PLACES TO SEARCH EXAMPLE

The table below lists some example places we might look to search for information

to address the example review questions shown in step 1.

Place to search Details

Our own bookshelf Sources known to the reviewers and others on the

project team

The internet Google search https://www.google.com/

Key websites Browse/search websites of the National Coalition for

Dialogue and Deliberation (http://ncdd.org/) and Initiatives of Change (http://us.iofc.org/)

Scholarly literature

databases

Google Scholar search https://scholar.google.com/

Our sources’ sources As we review relevant studies we find, scan the

sources they cite to identify potentially useful additional sources

Key Journals Search the journals Race and Social Problems and

Journal of Dialogue Studies

https://www.google.com/

http://ncdd.org/

http://us.iofc.org/

https://scholar.google.com/


PLACES TO SEARCH WORKSHEET

In the second column of the table below, enter details for one or more places that

you plan to search. Leave blank the details for any places that you do not plan to

search. The number of places you search will depend on the scope of your review

(whether your goal is to find all publications on your topic or to provide a brief

overview based on readily available information).

Place to search Details of Where We Will Search

Our own bookshelf

The internet

Key websites

Scholarly literature

databases

Our sources’ sources

Asking researchers and

experts

Online repositories on

specific topics

Key journals

Other (specify)


STEP 4. DEVELOP SEARCH TERMS

Your search terms flow from your review questions. To develop search terms,

identify the concepts in each of your questions and come up with terms related to

each concept. Use those terms to surf.

Experiment with different combinations of search terms, and refine your search

terms based on your preliminary search results. Add to and delete terms as needed

to narrow or broaden your search, to find the most relevant information.

You will likely need to construct and run more than one search query in order to

search all relevant combinations of terms.

To be systematic and rigorous, keep track of all iterations of search terms you use

for each place that you search and how many results you get from each search.

Tips:

Try different forms of words, such as “dialogue group” and “dialogue groups”

(with and without the plural “s”).

Try different words that refer to the same thing, such as “dialogue groups”

and “study circles,” or “community organizations” and “nonprofit

organizations.”

Use your inclusion criteria to refine your search, such as restricting your

search to studies within a certain date range, or studies written in English.

SEARCH TERMS EXAMPLE

The table below shows example concepts and search terms for our example R3

question, “What effects on race relations and racial equity have interracial dialogue

group programs shown?”

Concepts in R3 Question(s) Search Terms

Dialogue groups dialogue group, dialogue groups, study circles

race relations/racial

equity/interracial

interracial, race, racism, ethnicity, racial

justice, racial equity


SEARCH TERMS WORKSHEET

In the first column of the table below, enter concepts in your R3 questions. In the

second column, enter your search terms for each concept. Use as few or as many

rows as you need to include relevant concepts in your questions.

Concepts in R3 Question(s) Search Terms


STEP 5. SEARCH AND SELECT STUDIES

The next step is to search and select studies. This involves three steps:

1. Scan titles and select potentially relevant items for abstract review

2. Scan descriptions or abstracts of selected titles and select potentially relevant

items for full-text review

3. Scan full-text papers and select materials to include in your R3

For a faster and easier approach, track and report:

Your inclusion criteria (from step 2)

What sources you searched (from step 3)

The total number of studies you select

For a more systematic and rigorous R3, use the worksheet on the following page to

track the number of studies screened and selected or rejected in each step

(scanning titles, scanning abstracts, and scanning full-text), for each search that

you conduct. Also, identify duplicate search results (when you find the same study

in more than one search). Track the number of search “hits” before and after

deleting duplicates.

Tip:

To increase the rigor of your R3, have two or more reviewers independently scan

and screen studies. In cases of uncertainty, discuss and resolve differences

between you. Consult other member(s) of the research team as needed to resolve

any remaining uncertainties about which items to include.


STUDY SELECTION EXAMPLE

The table below shows some example search results and selection decisions.

Place searched: Google Scholar

Date: 10/27/2016

Search terms: interracial OR intergroup /AND race AND dialogue AND group

AND "United States"

Search result (title/citation of each “hit”

in your search results)

Scan abstract? (yes/no)

Scan full text?

(yes/no)

Include in R3?

(yes/no)

Reason for excluding

(if no)

[Book] Intergroup dialogue: Deliberative democracy in school,

college, community, and workplace.

DL Schoem, S Hurtado,

2001

Yes Yes Yes -

[Book] Streetwise: Race, class, and change in an urban community.

E Anderson, 2013

No - -

Does not seem to be about interracial

dialogue groups

Restorative justice dialogue: The impact of mediation and

conferencing on juvenile recidivism.

W Bradshaw, D Roseborough, Fed.

Probation, 2005

No - -

About juvenile recidivism – not about improving

race relations in the community

Critical Ethnography for Communication Studies:

Dialogue and Social Justice in Service Learning.

L Artz, Southern Journal of Communication, 2001

Yes No -

About service learning – not

about race relations in the community

Using intergroup dialogue to promote social justice and change.

A Dessel, ME Rogge, SB

Garlington, Social work, 2006

Yes Yes Yes -

https://scholar.google.com/citations?user=t3ybRTUAAAAJ&hl=en&oi=sra


STUDY SELECTION WORKSHEET

Complete the worksheet below for each place that you search. Add more rows as

needed.

Place searched: ___________________

Date: ___________________

Search terms: _________________________________________________

_________________________________________________

Search result

(citation for each “hit” in your search results)

Scan

abstract? (yes/no)

Scan full

text? (yes/no)

Include

in R3? (yes/no)

Reason for

excluding (if no)


STEP 6. EXTRACT RELEVANT INFORMATION FROM STUDIES

The next step is to read and extract relevant information—study bibliographic information, study details, and study

results and recommendations for each of your R3 questions—from the studies that you selected. Below is an

example for a few studies.

EXAMPLE TABLE OF STUDIES

Table of Studies: Study Bibliographic Information

Study Author Title

Publisher/

journal and issue/pages

Date of publication Web Link

1 B. M. Judkins Intergroup

Dialogues: Building Community and

Relational Justice

Catalyst: A

Social Justice Forum, Volume 2,

Issue 1

2012 http://trace.tennessee.edu/cgi/viewco

ntent.cgi?article=1032&context=catalyst

2 E. Nagai-

Rothe

Dialogue as a

Tool for Racial

Reconciliation: Examining Racialised

Frameworks

Journal of

Dialogue

Studies, 3:1

Spring 2015 http://www.dialoguestudies.org/wp-

content/uploads/2015/05/Journal_of_

Dialogue_Studies_Vol_3_No_1_Dialogue_as_a_Tool_for_Racial_Reconciliation_Examining_Racialised_Framework

s.pdf

3 J.L. Miller &

S. Donner

More Than Just

Talk: The Use of

Racial Dialogues to Combat Racism

Social Work

With Groups

23(1):31-53

December

2000

https://www.researchgate.net/publica

tion/232873101_More_Than_Just_Tal

k_The_Use_of_Racial_Dialogues_to_Combat_Racism

http://trace.tennessee.edu/cgi/viewcontent.cgi?article=1032&context=catalyst



http://www.dialoguestudies.org/wp-content/uploads/2015/05/Journal_of_Dialogue_Studies_Vol_3_No_1_Dialogue_as_a_Tool_for_Racial_Reconciliation_Examining_Racialised_Frameworks.pdf






https://www.researchgate.net/publication/232873101_More_Than_Just_Talk_The_Use_of_Racial_Dialogues_to_Combat_Racism





Table of Studies: Study Details

Study Study Population Study Geographic Area Study Methods

1 Review of previous studies on dialogue groups, both in communities and on

college campuses

United States Analyzed current research on intergroup dialogue

2 Participants in a study circle with American

University students (graduate and undergraduate)

American University

(Washington, DC)

Draws on author’s

experience facilitating a study circle; also discusses

related literature

3 A racial dialogue at a school of social work.

The dialogue was open to master’s

students, doctoral students, faculty, and staff.

In the United States

(location not specified)

Tape-recorded and

transcribed a racial

dialogue; participants completed questionnaires


Table of Studies: Study Results Stu

dy Study Results on

What effects on race relations and racial equity have interracial

dialogue group programs shown? (question 1)

Study Results on

What are the effective strategies for making interracial

dialogue group programs successful? (question 2) Study Recommendations

1 The study reported, the research on study circles found that

organizers and participants of all races/ethnicities generally viewed the circles as effective for fostering

changes in attitudes and behaviors. Participants expressed

less agreement, especially among young Black participants, on whether the study circles were

effective for addressing institutional racism.

-

Suggested that inter-racial dialogue seems an effective way to educate

participants about discrimination, inequality, and oppression, providing a step toward racial justice. Suggested

working with community organizations or sectors to organization dialogue

events or groups. Also suggested bringing together participants from across dialogue groups to discuss what

they learned and how that shared knowledge might influence public

policies.

2 Contended that dialogue is

important to racial reconciliation at an interpersonal level, particularly in the context of race relations in

the U.S.; however, dialogue alone is not enough to reach a racial

reconciliation, symmetry and equality.

-

Stated, “Mechanisms to address

structural inequality and power disparities at the societal level must therefore be in place in addition to the

interpersonal reconciliation that takes place within dialogue settings.”


Stu

dy Study Results on

What effects on race relations and racial equity have interracial

dialogue group programs shown? (question 1)

Study Results on

What are the effective strategies for making interracial

dialogue group programs successful? (question 2) Study Recommendations

3 Nearly 100 percent of participants strongly agreed that the racial

dialogue was helpful to them. However, white participants were more likely than participants of

color to say that they gained increased understanding or heard

perspectives that they had not considered before.

Participants showed openness, which the researchers said

could be attributable to several factors: the opening dialogue by the bi-racial team of

facilitators; the commitment of the participants; participants’

confidence in the skill of the facilitators; “the hunger participants expressed for a

forum in which to meaningfully discuss race and racism;” prior

race-related events and courses on the campus; and the relationships that the

student participants had with each other before the event.

Provided detailed recommendations for dialogue preparation, facilitation, and

follow-up (see study).


STUDY INFORMATION WORKSHEET

Complete the worksheets below to extract relevant information from each study in

your review of related research. Add more lines as needed if you have more

questions.

STUDY BIBLIOGRAPHIC INFORMATION

Author(s)___________________________________________________________

Title_______________________________________________________________

Publisher/journal and

issue/pages_________________________________________________________

Date of

publication__________________________________________________________

Web

link_________________________________________________________

STUDY DETAILS

Study

Population__________________________________________________________

Study Geographic

Area_______________________________________________________________

Study

Methods______________________________________________________

STUDY RESULTS AND RECOMMENDATIONS

Study results for R3 question

Study recommendations


STEP 7. EVALUATE THE RELEVANCE AND QUALITY OF STUDIES

Below are three ways to evaluate the relevance and quality of studies from any

type of study design. Which approach is right for you will depend on your study

information needs, timeline, and budget.

TRUST AUTHORITY (FASTEST)

This is a quick and easy approach that we often use in our everyday lives. For

example, say you’re curious about whether drinking orange juice helps heal a cold.

So you search for that information online. You’ll probably be more likely to trust

information from authoritative sources like the Mayo Clinic, the Centers for Disease

Control and Prevention, or the World Health Organization, than to trust inform

Similarly, for our review of related research for Strong Communities, we focused on

reasonably reliable sources--like government agencies, leading national

organizations n the field, universities, peer-reviewed journal articles.

Similarly, in a review of related research, it’s a good idea to focus on reasonably

reliable sources—like government agencies, leading national organizations in the

field, universities, and peer-reviewed journal articles. Less reliable sources (e.g.

Wikipedia, which anyone can edit) may be useful for leading you to other sources,

but are not a reliable source to cite directly.

Assess the relevance of each study based on how closely the population, geographic

area, intervention type, and intervention setting included in the study match the

context of your project.

One problem with this approach is that we might overlook valuable information that

comes from someone who doesn’t have “the right credentials.” Such as

independent researchers and smaller and lesser-known research organizations, or

someone who doesn’t have a PhD.

The flip side problem is that information coming from a trusted source is no

guarantee that the information won’t be misleading or have limitations. So

sometimes we need to “question authority.”

EXAMPLE STUDIES’ SELF-REPORTED LIMITATIONS (MORE RIGOROUS)

A somewhat more rigorous approach is to examine the studies’ self-reported

limitations. You can do this on your study information sheet or Table of Studies

(from step 6) as you’re extracting information from the studies. Record the study

strengths and limitations that the study authors reported. Then, synthesize results

across studies to assess strengths and limitations of the overall collection evidence

from the studies.


Look for limitations that the study authors mention in the discussion sections of the

papers.

This approach better than just trusting authority (“It’s from a peer-reviewed

journal, so I believe it.”) However, a problem with relying on this approach is that

researchers don’t always report all the limitations of their research.

EVALUATE STUDY QUALITY FOR YOURSELF (MOST RIGOROUS, ADVANCED)

Selecting only studies from peer-reviewed journals and other leading authoritative

sources is no guarantee that a study has no methodological flaws. To be more

rigorous, scrutinize the quality of evidence from each study yourself. Use the

worksheet below to determine criteria that you will use for assessing the quality of

the studies.

This is the most advanced and rigorous approach to assessing study relevance and

quality. It requires that you know when the designs and methods that the study

used are appropriate to use and how to properly apply those methods. So that you

can judge whether the study used appropriate methods for the types of data and

questions and whether the researchers followed appropriate procedures for the

types of methods used.

We created the Study Quality and Relevance Assessment Worksheet below based

on our review of several tools for assessing study relevance and quality that we

found in the evaluation and research literature. For more information, see our

Review of Related Research (R3) Reading List.


STUDY QUALITY AND RELEVANCE ASSESSMENT WORKSHEET

Complete the table below for each study in your R3. In the second column of the

table, enter information about the quality and relevance of the study. Feel free to

modify the relevance and quality considerations to fit your specific situation.

Relevance and Quality Considerations

Yes/No (and reasons, if applicable)

Good

Uses appropriate designs/ methods for the study questions and context?

Clearly explains methods (what the researchers did to collect, analyze, and interpret the data)?

Reports limitations and how they were addressed?

Uses appropriate system for collecting, analyzing, and interpreting data for the type of designs and methods used?

Explains potential sources of bias or conflicts of interest?

Findings are generalizable beyond the specific study?

Better Findings are supported by good quality data from multiple sources?

Best People have used knowledge gained from the study to make decisions that have worked as the study predicted?

Relevance / Meaningfulness of Findings

Good Researchers collaborated with all stakeholders to ensure study relevance?

Study context (e.g. participant population,

geographic area, type of intervention, types of organizations involved) is similar to our context?

Better Independent experts/stakeholders (e.g. peer-reviewers) reviewed and agreed with the study?

Our stakeholders agree with the study?

Best Study has broad consensus among stakeholders and experts in the field?

Quality of Explanation/Theory

Good Study includes causal explanations that make sense to you?

Better Study considers all relevant potential causes and alternative explanations?

Best Study improves upon existing explanations or theories from past related studies?


STEP 8. SYNTHESIZE RELEVANT INFORMATION FROM ACROSS

STUDIES

There are many specific techniques for analyzing information from reviews of

related research:

Realist review

Narrative synthesis

Quantitative meta-analysis

And many others

For additional information, see our Review of Related Research (R3) Reading List,

which provides links to information about many techniques.

A basic approach is to use your Study Information Worksheet or Table of Studies

from step 6—look at where you entered the study findings for your review

questions. Then summarize the overall findings across studies for each question.

To make it more systematic, count and report how many studies say each thing, as

in the example below.

EXAMPLE

Below is an example synthesis of the study information from the example Table of

Studies in step 6.

Effects of Interracial Dialogue Groups/Study Circles on Race Relations and Racial

Equity

Three studies reported study circles as being helpful for fostering individual-level

change, such as changes in attitudes and behaviors (Judkins, 2012), interpersonal

relations (Nagai-Rothe, 2015), and understanding and awareness of other

perspectives (Miller & Donner, 2000). In one study, white participants were more

likely than participants of color to report learning something new (Miller & Donner,

2000).

Two studies found that study circles alone did not appear effective for addressing

institutional-level change, in terms of promoting institutional racism or promoting

racial equity (Judkins, 2012; Nagai-Rote, 2015).


STEP 9. CLEARLY REPORT HOW YOU ANALYZED THE STUDIES

Completing this workbook will give you the details of your review of related

research methods that you’ll need to clearly report exactly what you did to conduct

your review.

Track and report what changes, if any, you make to your original review plan, and

why.

For more information, our Review of Related Research Reading List provides links to

sources that provide detailed guidance on reporting about a literature review study.

Use the R3 steps you learned in this workbook to transform “Little is known” to

“This is known”!

Documents

Reviewing Related Research (R3) Workbook - …meaningfulevidence.com/.../Reviewing-Related-Research-R3-Workbo… · Reviewing Related Research (R3) Workbook © Meaningful Evidence,