View
121
Download
0
Embed Size (px)
DESCRIPTION
Presentation about Project Review strategies. Look at my profile at: http://www.pongratzconsulting.com/?page_id=33 or http://www.pongratzconsulting.se/?page_id=26 and don't hesitate to contact me if you have questions or comments IP
Citation preview
PEER REVIEW STRATEGYWarzaw, May 2013Sofia Kuhn and Ingemar PongratzMuninn: Managing and promoting science
OUTLINE OF TALK
1. Setting up a proposal review strategy
2. General Considerations
3. Planning the call
4. Publication of the call text
5. How to manage an open call
6. Review process
7. Publication of Results
8. Closing the call
WWW.MUNINNSCIENCE.COM
1. SETTING UP A REVIEW STRATEGY
• Plan in advance• Develop an realistic time-frame
• Include ample time to identify the key players and key actions
• Reviewers
• Profile
• Check for conflicts of interest
• links
• Training needs for personnel
• Communication strategy
• Strategy need to be developed together with financing provider
• In general advisor(s) could support this phase
• Identify problems in advance
WWW.MUNINNSCIENCE.COM
GENERAL CONSIDERATIONS
• Take into account:• Timing
• Dissemination strategy
• Where to provide information about the call
• How to provide information
• Which is the audience
• How many proposals do you expect
• Yearly (similar) calls
• Develop and publish guidelines
• Evaluate your guidelines
• Inform
WWW.MUNINNSCIENCE.COM
PREPARING THE CALL
• What are you going to fund• Research Projects (consumables, salaries etc)
• Career development (courses, training, schools etc)
• Fellowships (PhD student, post-doc, researchers)
• Conferences meetings
• Who is your target• Academia
• Industry
• Translational research
WWW.MUNINNSCIENCE.COM
PREPARING THE CALL
• Focus of the call• Open or theme based call
• Applied of frontline research
• Outcomes of the research
• Knowledge or Innovation
• Dissemination and communication• Internal communication (own organization)
• Communicate to possible recipients
• Communicate to scientists/organizations that will not receive funding
• Media, trade organizations, general public
WWW.MUNINNSCIENCE.COM
PREPARING THE CALL
• Dissemination/Communication strategy• Timeline
• Communication plan
• Prepare online documentation
• Administrative planning• Several calls
• Use a dedicated solution
• Open and secure intra-section
• Test in advance
• Guidelines to staff (written mandate)
• Adhoc experts
WWW.MUNINNSCIENCE.COM
PREPARING THE CALL
• How to handle applications• Webb-based or paper based
• Register (file) all documents
• Inform the applicants
• Develop a web-based application portal• Several calls
• Use a dedicated solution
• Open and secure intra-section
• Test in advance
WWW.MUNINNSCIENCE.COM
THE REVIEW PROCESS
• Core principles of the review
WWW.MUNINNSCIENCE.COM
Core
PrinciplesIntegrityQuality
Admin
Method
THE REVIEW PROCESS
• The participants• Applicants
• Evaluators
• Programme officers and staff
• Proper communication• Applicants
• Staff
• Review duties• Objective
• Impartial
• Fair
WWW.MUNINNSCIENCE.COM
KEY DUTIES
• Avoid/declare conflicts of interest• Perceived
• Public trust
• Real• Identify early
• Staff duty, define clear guidelines
• Inform reviewers
• Search in advance
• Confidentiality of the review• Open
• Blind (double, single)
• Legislation may be important
WWW.MUNINNSCIENCE.COM
KEY DUTIES
• Communication of the results• Reviewers/advisors
• Non-disclosure agreement
• Code of Conduct
• Administrative staff
• Explicit mandate to communicate
• Other
WWW.MUNINNSCIENCE.COM
INTERVENTION BY APPLICANTS
• Applicant right to to intervene is critical for the integrity of the process• Stress this point
• Critical for the integrity and reputation of the call
• Redress• No interference with the review process
• Factual errors need to be corrected (Advisors)
• New review?
• Multistep process• Allows for more interaction between applicants and reviewers
• Information to all in advance
WWW.MUNINNSCIENCE.COM
SUMMARY OF CALL CYCLE
WWW.MUNINNSCIENCE.COM
Monitoring
and
Evaluation
Communi-
cation
Preparation
Call publication
Preliminary check
Expert selection
Review and decision
Rebutal
PREPARATORY PHASE
• Mandate• Launch or Re-launch
• Communication and PR
• Programme:• Needs
• Objectives
• Target groups
• Performance monitoring
• Administration and implementation• Work plan and logistics
• Responsibilities
• Budget (Review process)
• Tech Resources
WWW.MUNINNSCIENCE.COM
PREPARATORY PHASE
• Staff and Resources• Contact points for applicants, journalists, other
• Eligibility checks, Conflicts of Interests, etc
• KEY ROLE IN THE REVIEW
• Training
• Formal mandate
WWW.MUNINNSCIENCE.COM
PREPARATORY PHASE
• Peer Review• Main features of the call
• Full proposal or two stage
• Communication between Reviewers and applicants
• Peer review model
• Remote
• Meetings
WWW.MUNINNSCIENCE.COM
PREPARATORY PHASE
• Administrative issues• Support for the panels
• Information flow • Reviewers
• Reviewers-administration
• Monitoring• Administrative
• Advisors
• Manuals• Call manuals, scope information, information about eligibility etc
• Other documents
WWW.MUNINNSCIENCE.COM
CALL PREPARATION
• Documentation
• Call for proposals (call text)• Scientific
• Eligibility criteria
• Format
• Review process
• Guidelines to applicants• Mandatory templates, page length etc
WWW.MUNINNSCIENCE.COM
PREPARATORY PHASE
• Reference material• Guidelines for applicants
• Guidelines for reviewers
• Code of conduct
• Redress
• FAQ
• Webb-
WWW.MUNINNSCIENCE.COM
LAUNCH OF THE PROGRAMME
• Dissemination• Web-based promotion
• Scientific media
• Mailing lists, newsletters
• Meetings
• Opening of the call• Stated time
• Closing• Communicated to all participants
• Receipts of submitted application
• Avoid extensions
WWW.MUNINNSCIENCE.COM
PROCESSING
• Validate the status of the application• Readable
• All the submitted material is present
• Pre-screening• Screen for complete proposals
• Submitted in time
• Affiliation
• Ethical concerns
• Inform early
• Acknowledgement• Full Review
WWW.MUNINNSCIENCE.COM
EXPERT SELECTION
• Very difficult task
• Expert Identification• Keyword matching
• Remote evaluation and consensus meeting
• Panel Rapporteur
• Monitoring
• Selection criteria• Excellence and experience
• Active in the field
• International / National
WWW.MUNINNSCIENCE.COM
REVIEW
• Incentives• Increases participation but mot quality
• Briefing of reviewers• Opportunity to provide formal advice
• Conducted in place
• Provide equal opportunities for all applicants• Message should be the same
• Communication• Timely
• Correct
WWW.MUNINNSCIENCE.COM
FINAL DECISION
• Consensus report• Provided by the panel rapporteurs
• Minutes of the meetings (including ranking) should be recorded
• With clear explanations for the applicants
• Funding decision• Taken by the organizations
• Based on the consensus report
• Information to applicants and other stakeholders• Fast
• Clear information on how to appeal
• Redress
WWW.MUNINNSCIENCE.COM
COMMUNICATION
• General• Timely
• Effective
• Efficient
• Planned in advance
• Train for leaks
WWW.MUNINNSCIENCE.COM
COMMUNICATION
• Applicants• Receipts and validation
• Inform of possible technical problems
• Eligibility
• Review outcome
• Right to answer
• Reviewers• Necessary and useful information
• Complimentary
• Stakeholders and media• Post-results
• guidelines (in advance)
WWW.MUNINNSCIENCE.COM
AFTER THE CALL CYCLE
• Evaluate the process • Evaluate the different steps individually
• Identify strengths and weakness
• Possibly by outside experts
• Quantify the problems
• Identify corrective actions
• Evaluate the views of all the participants• Administrative staff
• Funders
• Reviewers
• Researchers
WWW.MUNINNSCIENCE.COM
AFTER THE CALL CYCLE
• Identify weakness after every call• Even if no follow up is planned
• Corrective action before new cycle is planned/decided• Specially if weakness is technical or structural
• Update Guidelines
• Publish with care
WWW.MUNINNSCIENCE.COM
FUNDING INSTRUMENTS
• Solicited or unsolicited research• Solicited: Clear guidelines (preparatory phase)
• Solicited: Expert panels can be established in advance
• Unsolicited: Experts identified after the applications have been received
• Pre-registration is an option
• Multi / Inter disciplinary research• One topic but several approaches
• Cross-disciplinary• Topics at the intersection of different research areas
• Review panels are more difficult to establish• Expert selection need to be discussed in advance
WWW.MUNINNSCIENCE.COM
Dis
c 3
T
FUNDING INSTRUMENTS
WWW.MUNINNSCIENCE.COM
T
Disc 1
Disc 2
Disc 3
T
Disc 1
IndividualProjects
MultidisciplinaryProjects
T
Disc 1Disc 2
Disc 3
InterdisciplinaryProjects
Disc 1
CrossdisciplinaryProjects
T
Disc 1Disc 2
Disc 3
TransdisciplinaryProjects
Disc 1
FUNDING INSTRUMENTS
• Career Development• PhD students grants
• Post-doc grants
• Advance Career Grants
• Many proposals
• Conflict of interests are common
• Project Applications• One researcher/research group
• Often unsolicited
• Experts panels need to be set up after applications have been received
• Many panels available in advance (continuous calls)
WWW.MUNINNSCIENCE.COM
FUNDING INSTRUMENTS
• Collaborative Research• Can be mono, multi or intradisciplinary
• Involve several partners
• Eligibility criteria
• Networking• No research funding
• Workshops and meetings
• Thematic
• National or international
• Eligibility and commitment
WWW.MUNINNSCIENCE.COM
FUNDING INSTRUMENTS
• Center of Excellence• Very Very competitive
• Political
• International review panels
• Infrastructure grants• Replace or Acquire new equipment
• Often the applicants contact funders in advance
• Application can also be endorsed by the University
WWW.MUNINNSCIENCE.COM