ARS Peer Review Process Making scientific quality review work for you

Ed ClevelandEd ClevelandScientific Quality Review Officer (SQRO)Scientific Quality Review Officer (SQRO)

Office of Scientific Quality Review (OSQR)Office of Scientific Quality Review (OSQR)

1998 Farm BillARS research peer-

reviewed every 5 years

Most review panelists external to ARS

Satisfactory reviewbefore beginning

research

Creation of OSQR

National Program Action Plan

Peer Review

Program Assessment

Stakeholder Workshop

Input

Implement

PlanAssess

InputPDRAM

Research Project Plan

Research initiated

Annual review

Congressional Mandate

Project Plan Outline

Certification

INTERNAL REVIEW

AREA OFFICE NATIONAL PROGRAMSTAFF

RESEARCH TEAM(individual scientists)

COOPERATION BETWEEN ALL LEVELSDURING PROJECT PLAN PREPARATION PROCESS

EXTERNAL REVIEW

OSQR OUTSIDE REVIEW PANELSInternal dialogue

and cooperation is essential

Review is a dialogue:Panel Recommendations and ARS Responses

National Program Action Plan

Peer Review

Program Assessment

Stakeholder Workshop

Input

Implement

PlanAssess

InputPDRAM

Research Project Plan

Research initiated

Annual review

Congressional Mandate

Project Plan Outline

Certification

Goal of Peer Review

Enhance research through independent, expert examination of PROSPECTIVE plans for scientific and technical merit.

Not an evaluation of the ARS, its mission, National Programs, project budgets, or personnel management

OSQROSQR

Context of Peer Review

OSQROSQR

Research must be relevant to an ARS National Research must be relevant to an ARS National Program Action PlanProgram Action Plan

Primary driver is the need to solve a problem, not Primary driver is the need to solve a problem, not investigator curiosity or idea noveltyinvestigator curiosity or idea novelty

Projects are not in competition for funding Projects are not in competition for funding

Evaluation generates an “Action Class” and Evaluation generates an “Action Class” and recommendations for improving.recommendations for improving.

Research plan must receive a passing Action Research plan must receive a passing Action Class in order to proceed.Class in order to proceed.

Peer Reviews

Provide external review by peers of the quality of a prospective project plan

Identify potential areas where impact of scientific effort can be increased

Increase the awareness of the quality and extent of ARS research programs

What is Reviewed?

Adequacy of Approach and Procedures

Probability of Successfully Accomplishing the Project’s Objectives

Merit and Significance

OSQROSQR

Action Classes

No revisionNo revisionExcellent, no change needed

Minor revisionMinor revisionvery good, a few modifications required

Moderate revisionModerate revisionGood, but has some important areas to

address

Major revision requiredMajor revision required Requires significant changes or additions

Not feasibleNot feasibleMajor flaws or not possible to assessOSQROSQR

OSQR Review Like Review of a Paper for Publication(strong advisory component)

- “Editor” = SQRO

- Two outcomes1. Publish after revision as monitoredby the “editor” (SQRO). Reviewers clear on whatresearchers are planning (minor gaps in info).(no, minor, moderate revision)

2. Publish after revision and reexamination by both reviewers and SQRO. Reviewers not at allclear about what researchers are planning (majorgaps in info). (major revision, not feasible)

• Hypotheses are poorly constructed and Hypotheses are poorly constructed and unclearunclear

• Lack of appropriate detail in Approach, Lack of appropriate detail in Approach, Procedure and Experimental DesignProcedure and Experimental Design

• Weak or no connection between project Weak or no connection between project objectivesobjectives

• Readability, narrative flow, rationale for Readability, narrative flow, rationale for research not developedresearch not developed

• Role of project team members, including Role of project team members, including collaborators, not well-definedcollaborators, not well-defined

• Milestones and timelines vague Milestones and timelines vague

• Perfunctory Contingency PlansPerfunctory Contingency Plans

• Project management and progress Project management and progress evaluation not documentedevaluation not documented

Frequent Panel Comments

OSQROSQR

See Statistician

Provide Diagram

Tie Contingenciesto Milestones

Most research in ARS is hypothesis-driven. Make sure these are credible, scientifically testable (i.e., falsifiable) hypotheses related to the objectives.

One of the most frequent comments OSQR receives from reviewers is that the plans do not contain real, testable, hypotheses.

Get advice from Statistician.

What Is a Real Hypothesis?

Definitions:

“A hypothesis is a tentative statement that proposes a possible explanation to some phenomenon or event.”

“It is an assumption written in a clear, concise manner about what you think will happen in your project”

“A hypothesis is a logical supposition, a reasonable guess, an educated conjecture”

The value of a well constructed hypothesis is to provide direction for your project, keep your investigation focused, and forces one to think about what results to look for in an experiment.

The development of a good hypothesis is not always an easy task, but without it, you may collect aimless data. Take the time to refine your hypothesis so you collect pertinent data. Remember the hypothesis keeps you a seeker of pertinent knowledge.

Debbie Boykin, Statistician MSA

Hypotheses that are too complex, i.e., these are statements with “and” and “or” that essentially make the hypothesis a compound hypothesis, rendering it very difficult if not impossible to really test and reject because part might be rejected and part might not.

Wiggle words. A hypothesis with “may” or “might” or “could” cannot be rejected; it’s true no matter what result you get.

Misdirected hypotheses about the researchers themselves. These say things like “Discovering the mechanism behind X will enable us to…….” This tests the abilities of the researchers to take information and do something with it. Instead, the hypothesis should focus on the experimental system itself.

Hypotheses that are statements of the obvious, or are scientifically trivial. “Disease results from expression of genes for virulence in the pathogen and genes for susceptibility in the host.”

Too global. “Quantifying X will provide significant increases in income for the industry.” Can any 5-year project plan in ARS really test this?

Some research is not hypothesis-driven.This is acceptable.

Examples are some types of engineering work and model development (Even in these, however, there may be a basis for hypothesis testing, e.g., testing whether a particular modification in a model provides a quantifiable improvement in how well the model predicts some real phenomenon).

If stating a hypothesis is not appropriate, be sure the goal or target of the work is clear.

• Hypotheses are poorly constructed and Hypotheses are poorly constructed and unclearunclear

• Lack of appropriate detail in Approach, Lack of appropriate detail in Approach, Procedure and Experimental DesignProcedure and Experimental Design

• Weak or no connection between project Weak or no connection between project objectivesobjectives

• Readability, narrative flow, rationale for Readability, narrative flow, rationale for research not developedresearch not developed

• Role of project team members, including Role of project team members, including collaborators, not well-definedcollaborators, not well-defined

• Milestones and timelines vague Milestones and timelines vague

• Perfunctory Contingency PlansPerfunctory Contingency Plans

• Project management and progress Project management and progress evaluation not documentedevaluation not documented

Frequent Panel Comments

OSQROSQR

INOCULATION OF DIFFERENTIALLY RESISTAN T (R) MAIZE GER MPL ASM

2. Plant Tran sformation, Tissu e Culture/Reg enerationTobacco mod el, Cotton (Rajasekaran)

1. Vector Design (Cary)

***BREEDING, RELEASE AFL ATOXIN RESISTANT GERMPL ASM, COMMERCIALIZ ATION PROCESS

PROTEOMICS; N ATIVE RESISTANCE TR AIT (PROTEIN/GENE)

IDENTIFICATION AND D ATAB ASE

INTEGRATION OF PROTEOMICS, PLANT BREEDING AND PLANT GENETIC ENGINEERING FOR ENHANCEMENT OF HOST PLANT RESISTANCE TO AFLATOXIN CONTAMINATION

3. Protein/DNA/mRNA Analysis (Cary)

Cooperating CRIS 6435- 41420-004.Assess plant R factors--Aflatoxinreduction relationship s; use fungal genomics to ID bioch emical targ ets for aflatoxin bio synthetic path way inhibition (Bhatnagar, Yu)

ID “R” Proteins Correlat ed with Either: 1. Gro wth or 2.

Aflatoxin Inhibition

Microsequen cing, GenBan k Search, ID Genes, Clone Genes, use Genes as Selectable Markers or in Plant Transformation (Brown, Cleveland, Chen** )

OBJECTIVE 1. Id entif y corn var ieties sho wing redu ction in fungal gro wth OR aflatoxin contamination levels (Brown, Cleveland)

OBJECTIVE 2. Identif y R related proteins in resistant corn through proteomics; ARS (Bro wn, Cleveland), LSU Staff/Facilities (**Chen/Damann)

OBJECTIVE 4. H yperspectral Imaging; Rapid Detection–spectral signatures of fungus/ aflatoxin & resistance traitsARS (Bro wn, Cleveland), ITD* (Hruska)

4. Greenhouse testing for Aflatoxin Resistan ce (Klich) Field testing (Cooperating CRIS 6430-42000-018, Cotty)

MARKER-ASSISTED BREEDING (corn)*** ARS & Institute for International

Tropical Agriculture (IITA) Breeders

*SPECIFIC COOPERATIVE AGREEMENT (SCA) INSTITUTE FOR TECHNOLOGY DEVELOPMENT (ITD)

**SCA WITH LSU

OBJECTIVE 3.ID“Foreign genes” (Cary, Rajasekaran, Cleveland)

PL ANT GENETIC ENGINEERING (cotton)

……does the plan credibly describe a cohesive, integrated project, or does it look “stove piped” with respect to how the objectives and personnel interact? It is important to describe a multi-personnel project in which the work hangs together into an integrated whole. Your plan should reflect how the work all comes together to accomplish the overall goals and objectives of the project.

• Hypotheses are poorly constructed and Hypotheses are poorly constructed and unclearunclear

• Lack of appropriate detail in Approach, Lack of appropriate detail in Approach, Procedure and Experimental DesignProcedure and Experimental Design

• Weak or no connection between project Weak or no connection between project objectivesobjectives

• Readability, narrative flow, rationale for Readability, narrative flow, rationale for research not developedresearch not developed

• Role of project team members, including Role of project team members, including collaborators, not well-definedcollaborators, not well-defined

• Milestones and timelines vague Milestones and timelines vague

• Perfunctory Contingency PlansPerfunctory Contingency Plans

• Project management and progress Project management and progress evaluation not documentedevaluation not documented

Frequent Panel Comments

OSQROSQR

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Sco

re/n

No

Rev ision

Minor

Rev ision

Moderate

Rev ision

Major

Rev ision

Not

Feasible

Action Class

Science Missing Information Writing or Organization

Writing a Clear PlanA well-done plan presents the “take home” message

from its opening pages.

What is the problem?

Why is it important?

Where are you going with it?

How are you going to get there?

And how will you know you have arrived?

This should be in brief on the opening pages

Correct Grammar and Spelling are Important…but not enough

Be sure your plan presents a clear, logical, path to success…at the outset and through the document.

Have scientific peers outside your project and Unit read the plan for understandability.

• Hypotheses are poorly constructed and Hypotheses are poorly constructed and unclearunclear

• Lack of appropriate detail in Approach, Lack of appropriate detail in Approach, Procedure and Experimental DesignProcedure and Experimental Design

• Weak or no connection between project Weak or no connection between project objectivesobjectives

• Readability, narrative flow, rationale for Readability, narrative flow, rationale for research not developedresearch not developed

• Role of project team members, including Role of project team members, including collaborators, not well-definedcollaborators, not well-defined

• Milestones and timelines vague Milestones and timelines vague

• Perfunctory Contingency PlansPerfunctory Contingency Plans

• Project management and progress Project management and progress evaluation not documentedevaluation not documented

Frequent Panel Comments

OSQROSQR

Complete a database on …….

Determine the accuracy and bounds of uncertainty of a model…….

Complete all work for a paper on…..

Complete the second year of a two-year experiment on…..

Complete the laboratory analyses for field samples collected last summer…

Deliver data from resistance trials to a breeder who will………

A Milestone is a MARKER that allows you to measure or assess your progress.

Used in the ancient world to gauge distance from Rome

Continue studies on… Cannot tell what threshold would determine success on this

Develop understanding of… Understanding is a fleeting goal easily overturned by new information

Plan a study that… Planning is an ongoing activity for all scientists.

Initiate experiment on… Could be as simple as a dated entry in a notebook.

One good approach to Contingencies is to link the section explicitly with Milestones that you specify in the Milestones table that comes later in the Plan. The Milestone might be acquiring either positive or negative data/results. If you create good Milestones that serve as decision points along the way, then Contingencies are the decisions that come as a result of achieving those Milestones.

Lead Scientist and Scientist

• Responsible for plan development and implementation (Lead scientist)

• Evaluate and document progress through the five-year cycle

• Interface with stakeholders providing information on impacts

• Prepare research papers and summaries of findings

Roles and Responsibilities

INTERNAL REVIEW

AREA OFFICE NATIONAL PROGRAMSTAFF

RESEARCH TEAM(individual scientists)

COOPERATION BETWEEN ALL LEVELSDURING PROJECT PLAN PREPARATION PROCESS

EXTERNAL REVIEW

OSQR OUTSIDE REVIEW PANELSInternal dialogue

and cooperation is essential

Review is a dialogue:Panel Recommendations and ARS Responses

Everyone has responsibility for quality of Project Plans

Research Team

Lead Scientist

Research Leader

OSQR Panel

Area Leadership

Laboratory Director

NPS--Program Direction (Dialogue, Coordination, Synthesize Objectives)

Roles and Responsibilities

Ensure Quality, Science Input

Ensure Quality, Management Viewpoint

Project Plans are Linked to Team and Individual Performance

Project Project PlansPlans

AnnualAnnualReports (421’s)Reports (421’s)

AnnualAnnualPerformancePerformance

Impact of Impact of Science,Science,

RPESRPES

Project Plan development forces thoughtful attention to project planning (hypotheses, experimental design, statistics, milestones and contingencies) which can enhance research and career success.

Another way to look at it…

You may enjoy a 30-year career with ARS.

Over that time the government may support project research with $20-30 million.

Six times in that career you will be asked what you are doing with the government’s money.

OSQR RESOURCES

Training Focusing More on What to Look For During INTERNAL REVIEW to Increase Quality of Plans

Training of NP Scientists after PDRAMs Issued

New SY Training

Leadership Training

New Research Leader Training

Stakeholder Workshop Training

Area SY Training (SAA done, MWA future)

Training on Web-site…new items added regularly

NEW: Workshop for all the Area Offices’ Project Plan reviewers (continue with periodic on-line training)

Project Plans

The foundation of ARS research

Link to performance and impact of individual and team

Reflect project team’s scientific expertise

OSQROSQR

The new OSQR Manual1. Shorter by 30-35 percent (main part >25 pages)

2. More guidance on writing and presentation

3. Format not quite a rigid(we care more about readability than if you use Times Roman)

4. PPO replaces prospectus

5. Reflects lessons learned over the years

6. Information on areas of concern to reviewers highlighted

Formal Agency review will begin shortly.(copies will be reviewed by all Areas and NPS)

Anticipate release by FY08 (October 1).

Foundation of Project Plan Development

NationalProgram

Researchteam

Project Plan

External Review

Stakeholder Input

Scientific Impact

Roles and ResponsibilitiesProgram Direction

National Program LeadersSet the objectives

(in dialogue with research team)

Project TeamScientists

Lead Scientists Research Leaders

Prepare Project Plan

Management Research Leaders

Center/Laboratory DirectorsArea DirectorsEnsure quality

OSQROSQR