View
23
Download
0
Category
Tags:
Preview:
DESCRIPTION
Workshop on research assessment in CERIF. Stephen Grace, Brigitte Jörg, Aija Kaitera, Maximilian Stempfhuber. Workshop on research assessment in CERIF. Three ten-minute presentations on systems in UK, Germany and Finland - PowerPoint PPT Presentation
Citation preview
Workshop on research assessment in CERIF
Stephen Grace, Brigitte Jörg, Aija Kaitera, Maximilian Stempfhuber
2
Workshop on research assessment in CERIF
• Three ten-minute presentations on systems in UK, Germany and Finland
• One hour discussion on common data elements across different national boundaries
• Map them to CERIF
• Submit to CERIF Task Force
Questions to ask
• WHO does it? Is it local or national?
• WHEN and WHERE it takes place? One-off or regular activity?
• WHY is it done? What is the purpose of the exercise – to allocate money, to improve performance, to encourage co-operation etc.?
• WHAT information/data is needed? The types of information needed?
• HOW universities handle the process? Systems, staff involved?
3
4
Assessment in the UK Stephen Grace
Research Assessment Exercise (RAE):•WHO does it? Funding Councils•WHEN? Every six years•WHERE? From University to Funding Council•WHY? To identify Research Excellence in order to
allocate funding•WHAT? Information on Researchers, their Outputs,
Sources of Funding, Studentships, Measures of Esteem•HOW? Gather data from internal systems to submit to
the Funding Council using XML Schema or online form filling
5
Research Excellence Framework (REF)
• Research Excellence Framework (REF) aims to be simpler to administer than RAE
• Introduces Impact element (25% of marks) to cover benefits to economy, society, culture, public policy and services, health, environment, international development and the quality of life
• Still working out what will be asked, and the form of
submissions in the area of Impact
6
Research Rating:
• WHO does it? German Research Council (Wissenschaftsrat)
• WHEN? Since 2007, pilot phase for selected disciplines
• WHERE? At Universities and research centres jointly funded by federal ministries and states
• WHY? To rate and compare research performance
• WHAT? Information on Research Groups: Quality, Impact / Effectiveness, Promotion of young Scientists, Transfer
• HOW? Questionnaire for all information except publications; self reporting system for publications or use of commercial reference database (depending on discipline)
Assessment in Germany Maximilian Stempfhuber
7
Assessment in Germany Maximilian Stempfhuber
• Rating (not ranking)
• Pilot studies in Chemistry and Sociology (2007/08), different modes of data collection
• Current: Electrical engineering & Information technology, planned: Humanities
• Currently not used for performance based funding
• In parallel:• University Excellence Initiative (Wissenschaftsrat & DFG)• CHE University Ranking (CHE & Die Zeit)
1: Quality of Research
• Citations• Per selected publication• Per selected publication, normalized to journal
• No. of reviewed publications
• Publications• List of publications per publication type• 5 most important publications• List of patents
1: Quality of Research (cont.)
• Total of external funding spent• Public funding• Commercial funding
• List of research awards etc.
• Management of national / international research networks
2. Impact / Effectiveness
• Publications: No. of reviewed publications
• Patents: No. of patents, total of earned licensing fees
• Total of external funding spent• Public funding• Commercial funding
• Percentage of external funded staff
• Management of national / international research networks
2. Impact / Effectiveness (cont.)
• Citations of selected (important) publications
• Citations of selected (important) publications normalized to journal
• No. of funded visiting researchers
• No. of distinguished positions in external scientific organizations, learned societies etc.
• Chair / co-chair at conferences
• Ratio: reviewed publications / scientific staff
3. Promotion of young scientists• PhDs
• No. of successful PhD studies / female applicants
• List of PhD (structured) programmes• No. of advised and finished PhD thesis
• No. of persons receiving grants
• List of first-time professorships
• No. of habilitations / female applicants
4. Knowledge / Technology Transfer
• No. of patents, total of earned license fees
• Total of external funding from industry
• List of spin-offs
• Involvement in standardizing bodies
• List of positions relevant for transfer
• List of activities in knowledge transfer
14
Assessment in FinlandAija Kaitera
• WHO does it? Academy of Finland / Ministry of Education
• WHEN? Every three years• WHERE? from individual commitees, the Universities• WHY? The state and quality of scientific research in
Finland Funding Allocation
• WHAT? publications, international mobility, develop research training research careers, interaction with society
• HOW? bibliometrics, targeted questionnaires; evaluation of disciplines and research fields (more continuous)
15
Assessment in FinlandAija Kaitera
16
Assessment Activities in FinlandAija Kaitera
• National level research system evaluations by Academy of Finland
• The state and quality of scientific research in Finland (every 3 years; bibliometrics, targeted questionnaires)
• Evaluation of disciplines and research fields (more continuous)
• Organisation Level by Scientific Advisory Boards • Intl evaluation of organisation’s activities (every 3 or 6
years)• Operational level by organisation’s internal
committees• Allocation of basic funding (from ministry to
universities every three years; universities to faculties every year)
• Allocation of quality funding• Evaluation of grant applications (continuous)• Evaluation of research programmes (continuous)
17
Assessment Activities in FinlandAija KaiteraData Collecting Processes
• From researcher to funding bodies• From researcher to university• From university to funding body• From university to ministry
18
Assessment Activities in FinlandAija KaiteraMethods• New national data warehouse (2013?)
• Research: Project, Grant decision, Person, Person’s education, University or Polytechnic, Address, Research group, Space, Publications, Innovations, Other research activities, Results
• Also person, employment, academic data• New national publication database (2012?)
• New national publication classification• Open access is included in national reporting (is/is
not), separate service for full texts • Will include publication quality assessment
“Norwegian model
National annual reportingQuestionnairesGrant reportsSABs
19
Assessment Activities in FinlandAija KaiteraPublication classification in Finland 2010->
20
CERIF4REF schema – 3:Tables generated in CERIF
cfOrgUnit-CORE
cfPers-CORE
cfFundProg-2ND
cfPersName-ADD
cfOrgUnitId_OrgUnit-LINK
cfPersName-OrgUnit-LINK
cfOrgUnit_ResPubl-LINK
cfPers_ResPubl-LINK
cfResPubl_Class-LINK
cfOrgUnit_FundProg-LINK
cfResPublBiblNote-LANG
cfResPublAbstr-LANG
cfClassTerm-LANG
cfOrgUnitResAct-LANG
cfOrgUnitName-LANG
cfPers_Pers-LINK
cfPers_ExpSkills-LINK
cfExpSkillsDescr-LANG
cfResPubl-RES
21
Expert Group on Assessment of University-Based Research
EU, 2010
http://ec.europa.eu/research/science-society/document_library/pdf_06/assessing-europe-university-based-research_en.pdf
22
Purpose Research productivity Quality and scholarly impact
Innovation and social benefit
Sustainability and scale
Research infrastructure
Allocate resources Research output/ bibliometric data
Citation dataPeer reviewKeynote, awards, etc.
Research income ‘research active’ as percentage of total academic staff; libraries, equipment etc.
Drive research mission Differentiation
Research output/ bibliometric dataOutput per research academic
Peer reviewSelf-evaluation
Ration of research income: teaching incomeExternal research income
Ratio of undergraduate: master/PhD students
Increase regional/ community engagement
Publications, Policy, Reports etc.
End user reviewsKeynote, Media awards etc.
Percentage funding from end-usersPatents, licences, spin-offs
Number of collaborations and partnerships
Improve research performance
Research output/ bibliometrics data
Citation dataNumber and percentage publication in top-ranked, high impact journalsPeer review
Assess value-for-money or cost-benefit of research
Research output/ bibliometrics dataOutput per research academic
Peer review and/or citation dataCommercialisation data
Social, economic, cultural and environmental impacts/ benefits indicatorsExternal research incomeEmployability of PhD graduates
Number of collaboration and partnerships
Encourage international co-operation
Research output/ bibliometrics data with focus on European & international collaborations
Percentage of research income from international sources
Number of collaboration and partnerships
Increase multi-disciplinary research
Research output/ bibliometrics data with focus on interdisciplinary fields
Peer reviewSelf-evaluation
New research fields, interdisciplinary teaching programmes, etc.
Research conducted by people from different disciplines
Multidimensional research assessment matrix
23
Purpose of research assessment • Allocate resources
• Drive research mission differentiation
• Increase regional/ community engagement
•Improve research performance
•Assess VFM or cost-benefits of research
•Encourage international co-operation
•Increase multidisciplinary research
24
Next steps
• Report back to this conference
• Report to CERIF Task Group
• etc
25
Thank you
Stephen Grace
Centre for e-Research
King’s College London
http://www.kcl.ac.uk/iss/cerch/
stephen.grace@kcl.ac.uk
Workshop Results
- Many countries deal with the same problems
- Different states of progress to implementation
- Currently, most go for Output Measurement rather than Impact (because measurable)
26
Workshop Results
- CERIF: Quite powerful representation means with respect to Assessment Requirements (-> REF)
- Best Way utilize CERIF is Best Practice
- Qualitative / Quantitative Data Requirements (across European activities)
27
Workshop Results
- Formalization and Measurement is necessary in order to do Analysis
- also the Measurement of soft qualities has to be formalized for Measurement
- CRISs the place to store even soft facts
28
Workshop Results
- CRISs the place to - document / archive soft facts (impacts) - for Reporting / Strategy / Planning …
29
Recommended