155
Draft version for public consultation – April 15, 2015 R&D Evaluation Methodology and Funding Principles Background report 10: The Small Pilot Evaluation – Feedback and Results

Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Embed Size (px)

Citation preview

Page 1: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Draft version for public consultation – April 15, 2015

R&D Evaluation Methodology and Funding Principles Background report 10: The Small Pilot Evaluation – Feedback and Results

Page 2: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

R&D Evaluation Methodology and Funding Principles

Background report 10: The Small Pilot Evaluation; Feedback and Results

April, 2015

Bea Mahieu, Xavier Potau – Technopolis Group

Page 3: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles i

Table of Contents 1. Introduction 4  

2. Feedback on the SPE processes and results 5  

2.1 Efficiency of the SPE processes 5  2.2 Effectiveness of the SPE tools and methodology 9  

3. Results of the panel calibration exercises 17  

3.1 Panel 1: Physical Sciences, Earth and related Environmental sciences 17  3.2 Panel 2: Engineering and Technology, Computer and Information Sciences 20  3.3 Panel 3: Medical and Health Sciences, Basic Medicine 23  3.4 Panel 4: Biological and Agricultural Sciences, Biological Sciences 27  3.5 Panel 5: Social Science, Sociology 31  3.6 Panel 6: Humanities, Philosophy 34  

4. Examples of SPE panel evaluation reports 37  

4.1 Introduction 37  4.2 Physical Sciences - Earth and related Environmental sciences: RU in the Institute of Analytical Chemistry of the AS CR 41  4.3 Social Science – Sociology: RU in the Masaryk University, Department of Sociology 45  

5. Bibliometric reports at the disciplinary area & field level 51  

5.1 Physical sciences 51  5.2 Engineering & Technology 55  5.3 Medical & Health sciences 62  5.4 Biological & agricultural sciences 66  5.5 Social sciences 70  5.6 Humanities 77  

Page 4: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

ii R&D Evaluation Methodology and Funding Principles

6. Examples of bibliometric reports at the RU level 80  

6.1 Physical sciences – Environmental sciences: Institute of Analytical Chemistry ASCR 80  6.2 Engineering & Technology – Computer & information sciences: Institute of Computer Science of the AS CR 87  6.3 Biological & Agricultural sciences – Biological sciences (Agrobiology): Czech University of Life Science, Faculty of Agrobiology 98  6.4 Social sciences – Sociology: Masaryk University, Faculty of Social Studies 105  

7. Examples of Comprehensive Data Reports for the SPE 113  

Table of Exhibits Exhibit 1 Efficiency of the SPE processes – the view of the participating EvUs ............. 6  Exhibit 2 Time investment by the EvUs participating in the SPE (average, 17 EvU) ..... 8  Exhibit 3 Fairness of the EM – the view of the EvU ....................................................... 15  Exhibit 4 The view of the EvUs on the evaluation results .............................................. 16  

Page 5: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results
Page 6: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 4

1. Introduction

This background report to the Third Interim report focuses on the outcomes of the Small Pilot Evaluation that was implemented in the context of this study from the month of September 2014 (launch of the preparatory activities) to end of January 2015 (final panel reports).

Information on the process of the Small Pilot Evaluation is given in the relevant chapter of the Third Interim report

In the first chapter, we report on the feedback that we received from the panel members, the panel secretariats and the participating Evaluated Units (EvUs) on the processes in the SPE and the Evaluation Methodology (EM) that was tested in the SPE in general. This was the Evaluation Methodology that was described in the Draft version of the First Interim Report. Conclusive considerations on the ‘lessons learnt’ as well as information on how the feedback was taken into account for the drafting of the Final version of the First Interim report are given in the relevant section of the Third Interim Report.

In the next chapters we reproduce the results of the panels’ work during the Small Pilot Evaluation, i.e.

• The outcomes of the calibration exercises in all panels – Section 3, and

• Two examples of SPE panel evaluation reports – Section 4

We included only those reports for which the Evaluated Units involved gave their consent to publication.

We provide examples also of the information upon which the evaluation panels based their assessment, i.e.

• Bibliometric reports at the disciplinary area and field level – Section 5

• Four examples of bibliometric reports at the RU level – Section 6

• Two examples of Comprehensive data reports at the RU level (i.e. the processed information submitted by the Research Units – Section 7

Also in this case, we included only those reports for which the Evaluated Units involved gave their consent to publication.

The readers of this report should be aware that the methodology implemented during the Small Pilot Evaluation is out-dated and does not reflect the final version of the R&D Evaluation Methodology.

During this study, the design of the R&D Evaluation Methodology was a dynamic process by excellence. It has build upon various sources of input and feedback, improving the alignment with the needs and context in the Czech Republic. It was designed from the very start as an iterative process, in which the Small Pilot Evaluation was a key step.

As a result, the feedback on the SPE by all actors involved and the main findings related to the SPE experience that are described in this report, has fed into the final design of the Evaluation Methodology, described in the Final First Interim Report. We report on it for the sake of completeness and for the readers to understand better the reasons for the changes that were implemented in the last phase of the methodology design as well as the potential outcomes of panel evaluations.

Page 7: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 5

2. Feedback on the SPE processes and results

In this chapter we summarise the feedback received from the panel members, panel secretariat and participating EvU against the main criteria of

• Efficiency, including time and cost efficiency, the user-friendliness of the processes and the clarity of the instructions and guidelines

• Effectiveness of the tools and methodology in collecting the information needed for a fair and useful assessment, taking into consideration the field specifics and the missions of the Research Organisations (are the indicators and assessment criteria sufficiently taking into account the differences between the research organisations involved, e.g. in terms of type of research conducted?)

2.1 Efficiency of the SPE processes Key questions for this assessment are whether

• The support and guidance provided to the Evaluated Units participating resulted in a reduction of the evaluation burden and allowed for a correct understanding of the information requirements

• The documentation delivered to the panels and the processes for the panel evaluation, including the criteria for the assessment, were sufficient and appropriate for the panels to forming their judgment

In the paragraphs below, we cover the following topics:

• The participating EvUs’ feedback on the efficiency of the SPE processes (Section 2.1.1)

• The time investment by the participating EvUs (Section 2.1.2)

• Clarity and usefulness of the information provided to the panels (Section Error! Reference source not found.)

2.1.1 Efficiency of the SPE processes 2.1.1 Efficiency of the SPE processes The participating EvUs’ perspective

Exhibit 1, below, summarises the feedback from the EvUs on the efficiency of the SPE processes. It shows the predominantly negative feedback on the user-friendliness of the submission forms and process and the clarity of the submission guidelines (half of the respondents giving scores of 2 or 1) and the more positive assessment of the information provided from the RD&I IS and the usefulness of the Information System as such.

In general, the feedback received highlighted the importance of a properly functioning and duly coordinated help desk in the evaluation implementation process.

In relation to the submission forms and submission process, the EvUs criticised the use of excel forms, which was dictated by the small scale of the SPE. Some institutions suggest a web-based shared system with the ability for simultaneous input and verification, as a way to speed up the process of submission and make it user-friendlier. In this feedback section, some EvUs complained on the help desk set up by the study team and stated that the process for contacting and asking questions about the forms did not really work.

Some EvUs indicated that the submission guidelines were not sufficiently clear and/or detailed and one of them suggested aligning the submission forms/tools to the criteria used in the evaluation in order to improve clarity. Other EvUs considered that, while

Page 8: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

6 R&D Evaluation Methodology and Funding Principles

the guidelines were clear, the problem was in matching internal categorisations with the categories provided in the form, especially for information related to staff and PhDs. Also the difficulty to identify FTE researchers in a university context was mentioned.

Exhibit 1 Efficiency of the SPE processes – the view of the participating EvUs

Institutions had different opinions on the user-friendliness of the information from the R&D Information System. Some EvUs considered that the information was well arranged and clear; another EvU complained on the lack of completeness (even though it was admitted that data lacked because of non-recording in the institution in the past); another EvU criticised the use of excel files and the difficulty to cut and paste the information.

Finally, generally the comments on the usefulness of the RD&I IS as a tool for information (in the context of the evaluation) were positive, although some of the institutions were a bit confused because the data did not match their internal records. One EvU also considered that the efficiency gain was relative because all data needed to be checked with the internal records anyhow.

The view of the evaluation panels

In several occasions, the panel members stressed that there is a need for clearer instructions in the submission guidelines for the EvUs, especially concerning personnel statistics and the qualitative self-assessments.

The panel secretariats also mentioned that from the discussions during the visits it became clear that the institutes need training and much more detailed and clearer instructions to be able to comply with the data submission requirements in a way that really reflects their performance.

The panel members highlighted in particular the importance for a higher level of quality and completeness in the information provided by the Evaluated Units. They considered that the EvUs seem to have misunderstood some requirements for information and reported incomplete information (possibly because of incomplete instructions to and training of the administrative personnel). FTE calculations proved to be particularly difficult, especially in Universities; even heads of operational units in the universities frequently found it difficult to approximate FTE spent on research.

0"

0.5"

1"

1.5"

2"

2.5"

3"

3.5"

4"

User-friendliness of submission forms and

process

User-friendliness of information from the RD&I

IS

Usefulness of RD&I IS as tool for information

Clarity of the submission guidelines

Feedback by the EvU on the efficiency of the SPE processes Average scores, scores 1 to 5, 17 respondents

Page 9: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 7

There were a good number of comments addressing questions on the reporting and interpretation of staff figures:

• Establishing the correct number of research FTE’s was notoriously difficult, and therefore it was hard to firmly assess productivity.

• Certain info was lacking to get a clearer sense of the ‘demographic shape’ within the RUs (i.e. true size of staff numbers, proportion of age-bands, gender distribution, information on how career development is supported, etc.)

• Information about PhD students should show only those graduated. Otherwise it includes large numbers of people who may or may not be present in the RU. It would also provide an incentive to make sure PhD students graduated.

• It would have been very useful to have data concerning research student progression, i.e. the number of doctorates awarded within a given year, the number coming in in a given year, the length of time taken to gain the doctorate and, if possible, some information on career destinations after graduation.

The panel members also considered that the qualitative information that was provided by RUs could have been more elucidating. They considered that the EvUs under-estimated the importance of the narratives, investing little time in the write-out of the qualitative information. They stated, “The narratives summarising crucial qualitative aspects of the RUs were often rather short and had a narrow scope.”

The panel members and panel secretariat made the following suggestions for improvement:

• It was often difficult to assess the RU in its context as part of a larger organizational structure. The Data Report should therefore start with a description of the RU under evaluation, i.e. its place in the overall organisation (an organisational chart could be helpful here), a full list of researchers etc.

• The organisations should be given space to comment on some of the data presented, for example explaining some of the deviations and evolution over the last 6 years.

• The list of collaborations needs some more information for the panel to judge their merit: Starting date, end date, level of funding, number of funded person-months.

• Evaluation questions / forms should be explained to the institutions. A question about strategy is a good example. Experts wanted to hear “Our Institute wants to become a European or a global or a regional leader in ABC. To achieve this we plan to strengthen this research field. Apply for Horizon 2020...” The Institution instead had a long list of research projects they are running arguing that this is how strategy is perceived by the Academy of Sciences or another funding body. Given the novelty of the criterion, examples understandable in the Czech context should be provided for societal relevance

• The eligibility of authors of the most ‘outstanding’ outputs needs to be clarified

• Several people within the same institution completed the forms that fed into the Data Report and in some cases it felt the full set of forms were not verified by one person-in-charge at the Institution. This created some confusion for the experts in understanding the data.

Page 10: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

8 R&D Evaluation Methodology and Funding Principles

In terms of the efficiency of the SPE process in providing support to the panels in the form of guidelines and reports delivered to them, in particular the ‘comprehensive’ and the bibliometrics ones, contained extensive information and data. However, the panel members and secretariat also mentioned the importance of the timely delivery of the information to the panels for the proper implementation of the remote assessments and reviews. The panel secretariats considered that 1 month prior to the country visit is an appropriate time frame; panels considered at least 2-3 weeks

Some panel members, however, considered that the structure of the panel guidelines needed to be more concise. Acronyms need to be explained in a table and information should be easy to find without reading linearly. Provided instructions should be restricted to the minimum necessary to carry out the task in a professional and useful way. Evaluation forms should include all the necessary points to be considered for the evaluation for each item (i.e., productivity, PhD management, financial support, etc.), avoiding excessive use of different instruction documents.

Finally, they considered that throughout the evaluation, process transparency on the work and data flow is crucial, for both the reviewers and the evaluated units. It is important that errors and/or misunderstandings are detected, logged and corrected. This is an important component for building trust on the procedure and finally ensuring its efficiency and impact.

2.1.2 Time investment by the participating EvU The finalisation of the SPE also provided the opportunity for the study team to collect information from the participating EvU related to the indirect costs of the EM implementation, i.e. the costs of the self-assessment process in terms of time investment by categories of EvU staff.

The panel secretariat informed that they were told that the institutions spent more time than originally envisaged for this SPE.

Exhibit 2, below, shows that the EvU invested in average 16.5 days for the completion of the submission forms, starting from the registration of the RU and identification of the researchers involved (Form 1). The administrative information requested required the highest level of effort (Form 3a); EvUs spent the least of their time in filing in information of the availability and use of research infrastructure by the RU (Form 4.2) and on the description of the value of their research for development in R&D and society at large (Form 4.4).

Exhibit 2 Time investment by the EvUs participating in the SPE (average, 17 EvU)

Time spent (mandays)

Nr of people involved (FTE)

Form 1 – Registration form 2.1 1.7

Form 2 – Submission of best research outputs of RU 2.3 1.7 Form 3a – Administrative information at the EvU level (staff, HR management, career development PhDs & postdocs, institutional funding

2.7 2.8

Form 3b – Administrative information at the RU level (staff, research funding, research outputs) 2.3 3.6

Form 4-1 – Research management, Research strategy (narrative) 1.9 1.7

Form 4-2 – Research infrastructure at/for the RU 1.5 1.7

Form 4-3 – Research presence and collaboration 2.3 2.2

Form 4-4 – Value for R&D and Societal impacts (narratives) 1.5 1.4

TOTAL 16.4

Page 11: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 9

2.2 Effectiveness of the SPE tools and methodology In this section we cover the feedback related to the tools and methodological measures used in the SPE and their adequateness for the attainment of a fair assessment.

Feedback from the evaluation panels regarded

• The methodological tools in the SPE, i.e. the usefulness of the calibration exercise and the site visits, and the completeness and quality of the comprehensive data and bibliometrics reports

• The adequateness of the assessment criteria to guarantee a fair and robust evaluation

• Both the evaluation panels and the EvUs provided their feedback on the fairness of the Evaluation Methodology.

• The participating EvUs also assessed the rightfulness and usefulness of the evaluation results

2.2.1 The usefulness of the calibration exercise and site visits At the very first stage of the evaluation process, the evaluation panels implemented a calibration exercise in order to ensure that all members of the panel had a similar understanding of the assessment criteria. During this process they discussed the field-specific interpretation of key terms used in the assessment criteria and on the importance of the main sub-criteria, in the field as such and for the different types of research organisations.

Overall, the calibration exercise was useful as it allowed experts to arrive at shared definitions and understanding of terms and criteria. The panel members considered these calibration exercises as a “must” to get a proper understanding of the grading among panel members and the significance of each sub-criteria and its contribution to the overall assessment criteria. The calibration discussion was also useful in bringing to light contextual information on Czech academic structures and politics.

Key words (such as originality, rigour, significance) could be differently interpreted according to the referees’ background and research area as well as the local institutional environment and working conditions of the evaluated units. The referees need to agree therefore on the definitions of the keywords after getting some basic information on the evaluated units, in order to:

• Avoid misunderstandings leading to biases in the referees’ assessments.

• Ideally the same definitions should be applied for all evaluated units of the same research field. Calibration is absolutely necessary to establish common ground when using somewhat arbitrary concepts (i.e., excellence)

In the context of the SPE, the site visits had the function of providing the panels with an opportunity to check the rightfulness of the evaluation results based on the submitted information and eventually ask for clarifications.

Visits were particularly important to obtain a better picture of the research infrastructure at EvU and RU. In most cases these visits were essential to realise that the RU was much stronger than what was presented and perceived from the reports.

Also the presentation of research strategies was made much clearer during the visits, compared to what was presented in the reports. In this context, however, the panel members found that those units that had, for example, a clear strategy or societal impact found their way to convey that in the self-assessment. Those that had not thought about their strategy before submitting the information ended up writing quite obscure narratives.

Page 12: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

10 R&D Evaluation Methodology and Funding Principles

It was also pointed out that organisations that are used to being evaluated wrote better self-assessment information than others who are not used to these types of exercise.

The panels’ conclusive reflections were that the site visit was useful especially for the institutions that under-presented their information in the self-assessment forms or wrongly understood the information requirements. In the latter case, site visits helped a lot in clarifying the aspects that RUs got wrong from the submission guidelines. The panels suggested that in case site visits are absolutely out of the question in the full-scale EM due to cost reasons, the evaluation agency may consider organizing a Q&A session using online tools and/or videoconferencing. Slides or even video of a presentation of the people in charge would be useful.

Significant differences in the scores before and after the site visits were mainly due to the lack of comprehensive completion of the self-assessment forms.

• For the Physical Sciences panel, the final assessments did not vary much between before and after the site visit in terms of research output and research excellence. However, for the assessment of the research environment, the site visit was extremely helpful in this respect to clarify misconceptions, and also to correct the impression that arose from the incomplete descriptions provided by the RU and EvU.

• From the Engineering panel, in several cases the assessments were somewhat different and in one of them radically different. The differences arose mainly from a lack of information in the data reports submitted by the EvUs.

• The assessments of the Medical sciences panel were radically different before and after the site visit. Information deficits in the reports were attributed to lack of training (and/or disinformation) of administrative personnel to prepare the requested documents.

• For the Biological & Agricultural Sciences the assessments were not radically different, but felt very provisional before the site visits. The site visits improved the grades in many cases. The panel makes a very strong case for the site visits arguing that they would contribute to the acceptance and support of the whole assessment process, providing units with the opportunity to explain and clarify things.

• In the Social Sciences the grading was not significantly changed due to the site visits, although panel members were generally more informed and more confident of the decisions they made.

• For the Humanities panel, the assessment was very different in one of the cases. Other differences in the assessment before and after the visit were caused by poor understanding of the information that needed to be reported in the reports provided by the RU, which was clarified during the visit.

2.2.2 The completeness and quality of the (data) reports to the panels Both the panel members and the panel secretariat stressed the relevance of providing background information on the Czech RD&I system, the existing evaluation methods and the funding system, as well as the positioning of the Research Units within their institutional environment for the panels properly to understand the context for their evaluations.

From this perspective, the panel members considered appropriate the level of background information provided in the SPE. They considered that the instructions and documentation received prior to the meeting were mostly clear and transparent of what was intended.

Also the participation of the study project manager in the panels’ meetings in the role of ‘specialist advisor on the context’, i.e. setting the background on the evaluation

Page 13: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 11

methodology and the Czech RD&I system and responding to additional questions, was considered particularly useful for the interpretation of the evaluation criteria, instructions and process of the SPE. Panels also appreciated the presence of a leading Czech scientist during some of panel meetings who could clarify specific queries.

Based on the above experience, it is suggested that either a physical or virtual meeting is set up for the panel members prior to the evaluation process. This meeting should introduce the main documents, clarify some definitions, provide instructions and set out the calibration exercise.

The panels considered that the data that was provided in the Comprehensive Data Report was relatively clear - when filled by the Evaluated Units correctly (which not always was the case). Questions were found to cover all the important areas of the assessment.

Some panel members also mentioned that additional potentially useful information would be a list of the names, qualifications and short CVs of researchers, a list of published papers, and a list of the recommendations from prior evaluations as well as the actions taken in reply to those recommendations.

In general, the panel members appreciated the information provided by the ‘bibliometric’ report. The shared impression was that the information efficiently covered the necessary parameters.

There were several suggestions for improvement of the report in order to enhance its clarity and user-friendliness for the panels and therefore increase its value and relevance for the future evaluation exercises.

• Style and clarity of information. A general reflection of the study team is that the wording of the headlines of the report and indicator names needs to be revised and simplified, with a more direct indication of what the purpose is of the indicator in the context of the evaluation.

At times they found it difficult to understand the meaning or the relevance of the data presented. It was suggested to give clearer explanations to the panel members on how the individual indicators are constructed, and especially, what information can be drawn from them and guidance for the correct interpretation of the data. This was particularly the case for some of the more elaborate metrics, i.e. Indicator C4 and Indicator D2

The panels also pointed out that some of the charts did not add any substantial information and that tables would be sufficient for the purpose.

The physical sciences panel (‘Earth and related Environmental sciences’ field) also indicated that the table in indicator C5 would be more useful if the entries were sorted alphabetically on journal name. The same panel also requested:

1. An indication of the impact factor of journals in that table

2. A measure of average impact factors in the field in order to help calibration

• There were some concerns as to the robustness of bibliometric data when small research units were evaluated.

• Structure and length of the report. Some panel members suggested that in order to improve the user-friendliness of the report, the names of the headings for the different sections could reflect the (sub)criteria for which the information is meant to be used.

Some panels suggested separating the initial “national” information from the field/RU one in the report and provide the information in two separate reports.

Page 14: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

12 R&D Evaluation Methodology and Funding Principles

Finally, the overall length of the report was discussed repeatedly. Many panel members pointed out that the report was longer and more complex than necessary, even though there was no agreement on which indicators to leave out.

Panel members found the bibliometric data was too disaggregated and as a result, a bit overwhelming to interpret. This was usually related to:

1. Very low percentages reported in some categories: Clustering or aggregating into categories was pointed out as a potential solution to this problem

2. Irrelevant figures reported in the shares of national output

3. Long lists of publications: Sorting and categorisation was offered as a potential solution

• Field-specific feedback on the Social Sciences and Humanities. In the Social Sciences, the panel members stressed the need to have the data well in advance in order to spend enough time assessing the quality and diversity of the output. This was not possible during the SPE, where final versions of the bibliometric reports were produced well into in the assessment process.

Panel members were also prudent in using it, as they were aware of the general limitations of bibliometrics for the SSH fields. Some of the panel members considered the limited usability of the international data because under-representative of Czech language publications, books and other outputs.

On the positive side, the data was still used to understand the contribution of the unit to the discipline and was found to cover all the necessary parameters for an evaluation.

The humanities panel considered that using WoS citation indexes, as a proxy for reputation in the field was counterproductive. In order to mitigate this, the members suggested:

− To use the panel members own assessment of quality in humanities journals

− To use a specialised index such as The European Reference Index for the Humanities and the Social Sciences (ERIH PLUS)1

• Full list of research outputs. Several panels asked to view also the full list of research outputs produced by the RU. The list that was provided to them was the one extracted from the RD&I IS and provided to the RU for the selection of their most outstanding outputs. Considerations were that these lists could constitute an important source of information provided the data was sorted and categorised. Suggestions were:

− Journal articles should be separate from conference proceedings

− It would be instructive to sort or cluster its entries by quality.

� Journal articles could be sorted, e.g., by the journals 5-year impact factor,

� Conference proceedings, e.g., by the acceptance rate of the conference.

− For calibration, one would need average impact factors and acceptance rates in a field. These could be established by a sample of popular journals and conferences in the field.

− Panel members in the Social sciences/Humanities panels suggested:

1 https://dbh.nsd.uib.no/publiseringskanaler/erihplus/about/index

Page 15: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 13

� Order chronologically per researcher

� Allow researchers to indicate and/or provide a sample of their most relevant work (e.g. his/her four most important publications.

• The bibliometric data did not allow to distinguish how much of the research outputs are published in in-house journals. In case of such journals, information should be provided if they are peer-reviewed and what type of peer-review is exercised (e.g. international, national including committee members outside of the institutions etc.).

2.2.3 The clarity of the assessment criteria The SPE panel members’ assessment was that in general, the assessment criteria were clear and relevant, as well as straightforward and adequate for the assessment purposes. Their impression was that the five fields of assessment made sense also to the Czech community.

There was some room for improvement: the panel members saw ambiguities with a number of sub-criteria that made the assessment more confusing, especially in ‘membership of the national and global community’ and ‘research excellence’. There is overlap between the assessment of ‘research excellence’ and ‘research performance’, and some confusion on the assessment and its relation to ‘productivity’.

Some field-specific considerations were as follows:

• The Social Sciences panel found it difficult to categorise one RU in the Criterion B “Membership of national and global research community”. The scale does not provide space to assess RUs that are not sufficiently recognised nationally and internationally. The scale seemed disproportionate as all units would score 2 or 3, 1 and 4 would be essentially out of scope. This leaves very little possibility for distinguishing among the units.

• In Humanities the criterion on research excellence is almost exclusively based on output readings as bibliometrics in the field are acknowledged as redundant. This means that humanities experts should be given the chance to read all material (including books, monographs) in all relevant languages (criterion for selecting panel members). For small units 1-2% of outputs sometimes meant 1-3 items for reading, experts have been expecting and were willing to read more.

• The Physical Sciences panel raised an issue regarding new research groups and how can trends be accounted for in the methodology. This panel also emphasised the need to have a full list of publications and further information such as journal impact factors made available for the panels

• The Engineering and Agricultural sciences panels suggested that the HR management and research strategy sub-criterion should be split into two, therefore in total there should be three sub-categories should be:

− Research environment

− Research strategy

− HR management – also includes central supporting units providing services, e.g. putting together H2020 proposals.

Some panels recommended that the scale for the quality levels should not be a scale from 0-4, but from 1-5. First of all, the latter is much more common in other countries. Secondly, interpretation becomes more adequate. For example, having the score ‘2’ out of ‘4’ sounds not really satisfactory, but having a ‘3’ out of ‘5’ is not so bad and probably understandable. Also clearer instructions for the grading in the scoring

Page 16: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

14 R&D Evaluation Methodology and Funding Principles

system were felt needed. The difference between grades 2-3 and 3-4 is particularly difficult to judge in most criteria and there should be some clarification between ‘international’ understood as ‘global’, ‘European’ or just with neighbouring countries and if/how does that impact assessment and grading.

The panels made the following recommendations:

• For the assessment of research excellence, the panel members found that the number of submitted publications for evaluation should be increased. The general suggestion was to at least include three papers for each RU or more in case of a larger RU and/or interdisciplinary research, and a complete list of publications. The assessment of research excellence based on only one paper was insufficient.

• The Societal relevance criterion was discussed at length in the panels and was one of the most difficult ones to assess. It should be emphasised that ‘Relevance for Society’ needs to be judged on more than just contributions to industry/economy. Science has an important obligation to seek solutions for problems for which there is currently insufficient economic interest, but that (may) affect many in society (e.g. antibiotics crisis largely neglected by industry). Additional questions should be considered, e.g. outreach activities including high schools, organisation of summer schools etc.

• In the Management and development potential criterion there was confusion on the level of management that should be taken into account. During the site visits it became clear that there were a couple of units with brilliant Deans at Faculty level, while the heads of RUs had poor vision. Faculty level influences what can be decided at the RU level but a question remained to what extent this should be taken into account in the assessment. Also a suggestion was to include a sort of check (e.g. after 3-5 years) on the claims units make in the section on management and development plans to ensure that they are not tempted to “build sand castles” just to improve their assessment but are thinking strategically and realistically about future options.

2.2.4 Fairness of the evaluation methodology The panels found the methodology capable of giving a fair assessment of research outcome. The methodology can be applied for the assessment of various types of research organizations because it contains adequate assessment criteria and sub-criteria.

Some panel members considered that the EM is best suited for scientific (and mainly basic) research. Teaching, national service and other institutional responsibilities are more downplayed. For teaching units however, the methodology is less straightforward because in everyday life of these organizations teaching and research are intertwined. This is bound to be a challenge when evaluating research only in such a system and the problem becomes more acute when comparing ‘pure’ research units with university teaching units. The Engineering panel also raised an issue regarding infrastructure organisations and judging them based on publications. A primary objective of these organisations is service provision, so the question was on whether taking the same criteria into account would be fair for them.

Related to the fairness from a scientific field perspective, the panels stressed the importance of qualitative information and warned against too much reliance on the metrics collected. On the one hand, they considered, “When a measure becomes a target, it ceases to be a good measure.” On the other hand, they pointed out that research is an activity that is not always well structured, uniformly conducted across different fields (or even inside the same field), producing measurable output at a steady rate, giving easily quantifiable results, and having a clearly identified impact in the short or medium term. Therefore, establishing the evaluation of research units upon data and performance metrics only may lead to a certain bias, favouring

Page 17: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 15

particular types of research and motivating researchers against working with more fundamental or “high-risk” problems.

Finally, they considered that to some extent, the scoring of an RU will depend on the level of education and efforts/time spent in the preparation of the evaluation. EvUs and RUs that are used to go through such processes may as a result have a head start and this needs to be taken into account for a fair assessment. A recommendation would be to organise online webinars explaining the assessment criteria and possibly videoconferences between the panel evaluators and the RUs.

The perception of the participating EvU was equally positive: the overall majority of the involved EvU scored the fairness of the EM against scientific fields and mission at the level of 3 or higher (15 out of 17 – see Exhibit 3, below).

Exhibit 3 Fairness of the EM – the view of the EvU

Critical comments related to the fairness from a scientific field perspective were collected mainly from the EvU in the field of humanities; the general comment from these EvU was that the EM did not allow for a sufficient collection of data in order to ensure a fair and robust evaluation. One of the institutions also complained about the composition of the panels used in the SPE not matching their research profile, i.e. their specialisation in the field of philosophy.

Comments related to the RO mission focused predominantly on the relevance of certain indicators for specific RO types rather than others, e.g. the questions on PhD students and educational programmes. Other comments – especially by the PSRO and the research infrastructure, regarded the perceived emphasis on publications (“the only kind of excellent result”); some RUs in the universities noted the lack of consideration for the teaching function in the universities.

2.2.5 Rightfulness and usefulness of the evaluation results The majority of the participating EvU scored the EM positively also in relation to the evaluation results, and more specifically the rightfulness of the panels’ judgment and the usefulness of the evaluation results (Exhibit 4, below).

Only 4 out of the 17 EvU scored the rightfulness and/or the usefulness of the evaluation results below 3. Particularly critical were the EvU active in the field of Humanities as well as some of the non-scientific research organisations.

0"

0.5"

1"

1.5"

2"

2.5"

3"

3.5"

4"

Fairness of the evaluation - field specifics Fairness of the evaluation -RO mission

Feedback by the EvU on the fairness of the EM Average scores, scores 1 to 5, 17 respondents

Page 18: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

16 R&D Evaluation Methodology and Funding Principles

Exhibit 4 The view of the EvUs on the evaluation results

Criticism revolved around the EM putting too much emphasis on scholarly outputs and bibliometrics, while not sufficiently considering applied research outputs such as prototypes, software and utility models. There were also complaints as to the number of outputs that could be submitted for review and the limited usability of the bibliometric data for the field of Humanities. Also the operational deficiencies in the implementation of the SPE caused by the limits to the SPE (see the Third Interim Report) were indicated as negatively influencing the rightfulness and usefulness of the SPE evaluation results.

An institution suggested involving evaluation experts (external to the panels) that are well acquainted with the local context. They considered, “This would enable the evaluators to understand more some of the pragmatic aspect of the evaluated units' academic behaviour. For instance, the constantly changing system of financing both teaching and research has a great impact on the strategies of the evaluated unit.”

There is a discrepancy between institutions when commenting on the usefulness of the results. While several of them regard the suggestions from the evaluation panel as beneficial and state that the evaluation results are being integrated in their institutional strategy, others considered that the panels’ recommendations were unrealistic in a Czech context.

0"

0.5"

1"

1.5"

2"

2.5"

3"

3.5"

4"

Rightfulness of the evaluation results Usefulness of the evaluation results

Feedback by the EvU on the rightfulness and usefulness of the evaluation results

Average scores, scores 1 to 5, 17 respondents

Page 19: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 17

3. Results of the panel calibration exercises

In the Evaluation Methodology, the link from evidence to assigning assessment scores is judgemental and not based on arithmetic and there is a reasonable level of freedom for the expert panels to define field-specific variations of the common generic criteria.

At the very first stage of the evaluation process, the evaluation panels implement a calibration exercise in order to ensure that all members of the panel have a similar understanding of the assessment criteria. They decide on field-specific interpretation of key terms used in the assessment criteria and on the importance of the main sub-criteria for the different types of research organisations. This will be expressed in the form of % weights for the sake of simplicity, but the overall judgment is not based on arithmetic. The panel members can also decide that all sub-criteria are equally important.

The sections below contain the sections of the SPE Panel Reports on the outcomes of these calibration exercises.

The calibration exercise in the SPE focused on

• The field-specific definition of the keywords ‘originality, significance and rigour’ for the assessment of performance against the criterion Research excellence, and the keywords ‘reach and significance’ for the criterion Societal relevance

• The identification of the importance to be attributed to the following sub-criteria, in the field and for the different types of Research Organisations that were assessed by the panels

− For the criterion Institutional management and development potential, the sub-criteria ‘Quality adequacy of the research environment’ and ‘Research strategy & (HR) management’

− For the criterion Membership of the national and global research community, the sub-criteria ‘Level of participation and recognition at the international level’ and ‘Level of participation and recognition at the national level’

− For the criterion Research performance, the sub-criteria ‘Research productivity’, ‘Ability to attract PhD students’, and ‘The overall quality of the research performance’

3.1 Panel 1: Physical Sciences, Earth and related Environmental sciences

3.1.1 Field-specific definition of keywords in the assessment criteria Research excellence

The assessment criterion on Research Excellence required the panels to assess the quality level of the RU outputs in terms of originality, significance and rigour.

The panel defined these keywords as follows:

• Originality: Describes a certain novelty in the idea or research approach that is going beyond the state of the art, and has not been described yet in the literature. The degree of originality is the extent to which the work goes beyond the current state of the art. It is not geographically restricted, originality is the same on national and international scale.

• Significance: Describes the impact a particular discovery or development has on the advancement of science and knowledge in a certain area of research. Although the time scale can vary largely until the significance of a particular finding or

Page 20: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

18 R&D Evaluation Methodology and Funding Principles

development is realized, one criterion is whether the particular work will be referenced by fellow scientists.

• Rigour: Describes the scientific thoroughness with which a particular research has been conducted. It refers to the use of sound and adequate methodologies, their consistent use to the best standards to achieve the particular goal. The choice of methods and of study objects is appropriate, and data evaluation and the interpretation of results have been done in a critical, comprehensive, informed and unbiased way.

Societal Relevance

The assessment criterion on Societal Relevance required the panels to assess the impact of the RU activities in terms of reach and significance

The panel defined these keywords as follows:

• Reach: Refers to the ability to inform the primary target groups of the results of a particular research. This includes the proper identification of the target groups, but also consideration about the best way (method, medium, level and depth of information) in which this information is conveyed. Both lay and expert audiences, as well as decision makers are important target groups.

• Significance: In contrast to scientific significance, this is the importance that the scientific result produced has for the end users or directly affected parties. It will depend on several criteria such as alignment to the end users’ need, ease of implementation, and how well/effective the result is communicated to the target group(s).

3.1.2 Importance of the sub-criteria for fields and RO typologies The panel defined the importance to be attributed to specific sub-criteria in the context of the field and for the RO typologies in the field. The panel did so in the form of %, even though the overall judgment was not to be based on arithmetic. The panel could also decide that all sub-criteria are equally important or that no distinction was to be made for the different RO typologies.

The panel took into consideration only the RO typologies that are active in the field in the Czech Republic.

Institutional management and development potential

Importance for the field overall: Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 50%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management 50%

Equal weight is given to both sub-criteria, since:

• The research strategy is of key importance – in a well-managed research organisation, there should be a clear vision for the scientific and/or societal goals of the work. To realise this vision requires a clear implementation plan, a proactive management, good communication structures, adequate human and financial resources.

• An adequate research environment (including state- of-the-art scientific equipment, adequate facilities, information services and ancillary services) is

Page 21: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 19

required to allow successful and effective R&D and to create a stimulating atmosphere.

• The same weights should be applied for all different types of research institutions.

Membership of the national and global research community

Importance for the field overall:

Sub-criteria Weight

Level of participation and recognition at the international level 60%

Level of participation and recognition at the national level 40%

• Many problems in the field of environmental sciences are at an international scale. It is thus essential to have international collaboration to arrive at solutions for the problems that cannot be addressed or resolved at national level.

• In order to be able to improve locally existing methodology and knowledge, one needs to be part of international consortia and networks. Being part of international networks, it is easier to arrive at international excellence; moreover, the participation in international networks and activities is one way to ensure the comparability of data and to provide validation of newly developed methods. It is thus considered essential that Czech research organisations be internationally integrated.

• International integration is nowadays a must in the field of physical sciences! It is hardly possible today to perform internationally recognized work when acting isolated on a local basis only.

• Weighting across the different types of institutions: Even if the objective of different research organisations may be different (e.g. it is likely that the mission of a governmental laboratory is to serve on national rather than on international level), international recognition must be an important factor as it often is the key to high-quality and high-impact research. Therefore the same weighting factors (60% vs 40%) have been proposed for all types of institutes.

Research performance

Importance for the field - HEIs and Academy institutes:

Sub-criteria Weight The research productivity 40% Ability to attract PhD students (registered/awarded or trained) 30% The overall quality of the research performance (all other indicators for evaluation) 30%

Importance for the field - government laboratories:

Sub-criteria Weight The research productivity 40% Ability to attract PhD students (registered/awarded or trained) 20% The overall quality of the research performance (all other indicators for evaluation) 40%

• Research productivity is an important indicator to assess the scientific activity of a research organisation. Since research results are only useful and will only have an

Page 22: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

20 R&D Evaluation Methodology and Funding Principles

impact (scientific and societal) if communicated, it appears appropriate to assign a weight of 40% to this sub-category which shall consider all relevant forms of scientific (scholarly) outputs.

• The ability to attract students is in academia a good indicator for the reputation of the research institution, and the quality of the infrastructure and level of resources existing there. This parameter naturally has a higher weight at the HEI and Academy Institutes (30%) in comparison to the government labs or public research institutes (20%) for whom it is not the primary task to train PhD candidates.

• The overall quality of research performance is a complex category that has both qualitative aspects (e.g. peak research excellence reached) as well as quantitative aspects (e.g. research productivity). It is assessed on the basis of a large number of indicators and has a larger weight for governmental labs (40%) versus University and Academy Institutes (30%).

3.2 Panel 2: Engineering and Technology, Computer and Information Sciences

3.2.1 Field-specific definition of keywords in the assessment criteria Research excellence

The assessment criterion on Research Excellence required the panels to assess the quality level of the RU outputs in terms of originality, significance and rigour.

The panel defined these keywords as follows:

• Originality: The originality associated with an output identifies research activity that has or is likely to have established a new field or has made new and incremental contributions to an existing field.

• Significance: This is a characteristic of research work that has had an impact on current technologies or methodologies; this can also be demonstrated through long-term impact e.g. on advancement of the scientific field.

• Rigour: This is associated with a research output that has a solid underlying foundation possibly based on theoretical computer science or associated with experimental verification. Also, with outputs that have a clear and transparent structure.

Societal Relevance

The assessment criterion on Societal Relevance required the panels to assess the impact of the RU activities in terms of reach and.

The panel defined these keywords as follows:

• Reach: This is research work that has impacted any aspect of society either directly or indirectly.

• Significance: The research output has produced contributions to society that provided significant added value for the users.

3.2.2 Importance of the sub-criteria for fields and RO typologies The panel defined the importance to be attributed to specific sub-criteria in the context of the field and for the RO typologies in the field. The panel did so in the form of %, even though the overall judgment was not to be based on arithmetic. The panel could also decide that all sub-criteria are equally important or that no distinction was to be made for the different RO typologies.

The panel took into consideration only the RO typologies that are active in the field in the Czech Republic.

Page 23: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 21

Institutional management and development potential

Importance for the field overall: Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 60%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management 40%

The research strategy is a crucial factor for any research unit and should precisely define the Unit’s research mission statement and objectives. The mission statement should provide the researchers with a clear collective objective for their research, and the implementation of such a mission statement. The Human Resources management is more focussed on facilitating research activities and providing a career development path for all researchers, for example, promotion criteria, mentoring etc. It has to be noted that this criterion is about research management, and should not be understood as encouragement to increase the administrative bodies.

Importance for the Infrastructure Research Organisation:

Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 70%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management 30%

The facilities available to a researcher should enable him/her to carry out national and/or international competitive research. In the case of equipment it should be research focussed, state of the art and well maintained and the latest software, prototypes etc. should also be considered part of equipment. Library facilities should enable easy access electronically to the latest journals and conference publications. For grant applications, guidance to appropriate calls for funding should be given as well as help in the formation of proposals and their submission etc. Support for industrial liaison and industry project development should be regarded as part of the research support infrastructure as well as guidance for other possibilities such as patent applications.

Membership of the national and global research community

Importance for the field overall:

Sub-criteria Weight

Level of participation and recognition at the international level 70%

Level of participation and recognition at the national level 30%

Research in Computer and Information Sciences in many aspects is international in nature and as a result contacts throughout the international research community are extremely important and almost essential for any researcher. The opportunities to attend international meetings, conferences, workshops etc. should be readily available, particularly for young researchers. This should result in more cooperation and collaboration with researchers throughout the world as well as invitations to join editorial boards, conference committees etc.

In addition researchers should present themselves for membership of national and international societies and also should be hoping to be recognised through the award

Page 24: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

22 R&D Evaluation Methodology and Funding Principles

of distinguished memberships, fellowships of major international societies etc. as well as winning best paper awards at conferences and meetings and being invited to give keynote conference addresses etc.

Importance for the Infrastructure Research Organisation:

Sub-criteria Weight Level of participation and recognition at the international level 70% Level of participation and recognition at the national level 30%

Infrastructure research organisations should be nationally and internationally networked and providing a service to their community which enables their researchers to compete nationally and internationally. Their facilities should be state of the art and regularly updated both in hardware and software. In addition, given the nature of their activities, they should be well linked in and cooperating with other similar bodies around the world and act as a contact point for the section of their community who would require facilities not available nationally. They should also provide education and training in the use and exploitation of their infrastructure facilities.

Research performance

Importance for the field overall:

Sub-criteria Weight The research productivity 30% Ability to attract PhD students (registered/awarded or trained) 30% The overall quality of the research performance (all other indicators for evaluation) 40%

There are now well-established criteria for evaluating research operating in many countries, which among other things, impact the allocation of research funding. The productivity of research is one of the main criteria for assessing research performance, and largely depends on the output of the researcher in major high impact journals and conference proceedings, the amount of grant funding attracted as a result of the research activity and the source of that funding, and the number of PhD students attracted to the research topic and successfully supervised.

Importance for the Infrastructure Research Organisation:

Sub-criteria Weight The research productivity 35% Ability to attract PhD students (registered/awarded or trained) 20% The overall quality of the research performance (all other indicators for evaluation) 45%

In the case of a research infrastructure organisation the nature of their research is primarily more applied in its focus and can have a multidisciplinary component in the sense that the nature of the activity is of major benefit to another discipline rather than computer science. Such applied research requires a different skill set and expertise than those of researchers working in a single discipline environment.

Page 25: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 23

3.3 Panel 3: Medical and Health Sciences, Basic Medicine

3.3.1 Field-specific definition of keywords in the assessment criteria Research excellence

The assessment criterion on Research Excellence required the panels to assess the quality level of the RU outputs in terms of originality, significance and rigour.

The panel defined these keywords as follows:

• Originality: The concept of originality goes against and beyond the existing paradigms in a given science. For example: (1) an experimental approach to falsate (i.e. to falsify) the generally accepted paradigm of neural plasticity (i.e., changes in neuronal connectivity) as the process underlying learning and memory; (2) going from reductionist approaches to holistic approaches (i.e. system biology and data integration); and (3) development of animal models (mouse/rat is different from humans), i.e. to develop a model with predictive power used in pre-clinical (Phase 2) (as animal models are not predictive for the use of specific compounds in humans).

• Significance: If a published study changes a generally accepted paradigm or introduces a new paradigm. According to present standards, one way to measure the significance of a published study would be via the number of times it has been cited over the five years of its publication. Nevertheless, changes in accepted paradigms take time and the significance of a given study is dependent on the general acceptance of involved scientific audiences.

• Rigour: Interesting but difficult to assess. The rigour of a scientific publication can only be determined/measured when the full data and protocol are easily accessible and when several laboratories succeeded in repeating the experiment. One way to assess rigour is by visiting the laboratory and observing an on-going experiment (how the process is organised, availability of the lab, etc.).

Societal Relevance

The assessment criterion on Societal Relevance required the panels to assess the impact of the RU activities in terms of reach and significance.

The panel defined these keywords as follows:

• Reach: Acquired knowledge and related cognitive abilities should reach sectors able of applying them (for example, clinicians, industrial and pharmaceutical sectors). Other important point is to disseminate knowledge to the general public (e.g. to patient organisations). Also to reach the mass media (TV interviews, written newspapers and magazines, book editors, social media).

• Significance: A research study could be significant if, for example, it contributes to decrease the cost of a given disease or represent a help to its prevention. The significance of a research study could also be determined by its impact on the pharmaceutical industry to the development of a new drug or, simply, by the acquisition of original knowledge able of reaching the cultural environment of the related, involved society.

3.3.2 Importance of the sub-criteria for fields and RO typologies The panel defined the importance to be attributed to specific sub-criteria in the context of the field and for the RO typologies in the field. The panel did so in the form of %, even though the overall judgment was not to be based on arithmetic. The panel

Page 26: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

24 R&D Evaluation Methodology and Funding Principles

could also decide that all sub-criteria are equally important or that no distinction was to be made for the different RO typologies.

The panel took into consideration only the RO typologies that are active in the field in the Czech Republic.

Institutional management and development potential

Importance for the field overall:

Sub-criteria Weight The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 30%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management 70%

The quality adequacy (i.e., including the following items: research capacity, funding stability, and availability of equipment) of the research environment is assigned a weight of 30%: One needs the equipment to implement important parts of the strategy. General and basic equipment needs to be maintained and/or renovated in accordance to the Research Unit needs.

Research strategy & (HR) management (that is, considering quality and feasibility of the research plan and quality of HR menagement9 are assigned a weight of 70%: It is important and necessary to adapt the research strategy taking into account availability of equipment. The SWOT (strengths, weaknesses, opportunities and threats) analysis together with a strategic plan would be also very convenient. With respect to the HR management: Is there is a journal club? Do they encourage PhD? How they manage career of technicians? Is there a quality checking of the research system?

Importance for the universities:

Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment

30%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management

70%

Although Universities have different types of academic duties not necessarily present in Government laboratories, the internal organisation of Research Units within academic institutions should have similar organisation than that already described for government-dependent research units. In this regard, it is very important the necessary attention to the proper equipment of the involved RUs as well as the necessary stability in the consecution of the financial support. But, on the other hand, RUs located in academic institutions need a well-defined research plan and the appropriate management of administrative, financial, and research works. In this case, it is obvious that an appropriate plan to follow alumni careers, the quality of jobs obtained by egressed PhD it is of great importance.

Importance for the Government Organisations:

Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 30%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management 70%

Page 27: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 25

As a general rule Government laboratories and research institutions should have the proposed balance (30% vs. 70%) between the research environment and the research strategy and (HR) management. On the one hand, funding stability and availability of general equipment is a necessary condition for a good-level of experimental research. But, on the other hand, it is impossible to organise a Research Unit in absence of an appropriated and consented research plan supported on the necessary and sufficient HR management. The latter include plans for a follow up of the main output of the RU, plans to follow the career of egressed PhD, etc.

Membership of the national and global research community

Importance for the field overall:

Sub-criteria Weight

Level of participation and recognition at the international level 50%

Level of participation and recognition at the national level 50%

In the present case (i.e., a research unit located in a government research centre, without important academic duties), it seems convenient to consider a 50-50 distribution of the levels of participation; because the RU (composed of different departments and research lines) included in this study could have a noticeable involvement at the national level. Indeed, the Institute of Experimental Medicine may have some clinical implications (neurological diseases, chronic mental pathologies, stem cell transplantation, auditory medicine, etc.) which obviously should be considered at the national level. It should be taken into consideration that the participation at the international level is mainly relevant for scientific outputs (i.e., research papers), while the national level is mainly relevant for applied outputs as already mentioned clinical applications, drug improvements, etc.

Importance for the universities:

Sub-criteria Weight

Level of participation and recognition at the international level 50%

Level of participation and recognition at the national level 50%

In the case of research institutions located in academic centres (i.e., government or private Universities) they should have the proposed balance (50% vs. 50%) between the international and national impact and recognitions of the involved RUs. In the case of the evaluated RU, the peculiarity of being an academic laboratory does not modify this proportion, because the international impact will be mainly determined by the impact factors of journals in which their papers are published (and the level of citation those papers will eventually reach), while the national level of recognition will be determined by the effects of the RU outputs on societal factors (clinical applications, cultural implications, effects on industrial developments, etc.). Nevertheless, in the case of RUs located in academic institutions a particular attention should be paid to egressed PhDs, alumni, etc.

Page 28: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

26 R&D Evaluation Methodology and Funding Principles

Importance for Government Organisations:

Sub-criteria Weight

Level of participation and recognition at the international level 50%

Level of participation and recognition at the national level 50%

As a general rule Government laboratories and research institutions should have the proposed balance (50% vs. 50%) between the international and national impact and recognitions of the involved RUs. In the case of the evaluated RU, the peculiarity of being a government laboratory does not modify this proportion, because the international impact will be mainly determined by the impact factors of journals in which their papers are published (and the level of citation those papers will eventually reach), while the national level of recognition will be determined by the effects of the RU outputs on societal factors (clinical applications, cultural implications, effects on industrial developments, etc.).

Research performance

The overall quality of the research performance should be taken into account when considering the international impact and repercussion of the evaluated RU. At the national level it is very important to consider the ability to attract PhD students as evaluated RUs need a link to the public universities from their neighbourhood. The research productivity should be expressed in quantitative terms: number of publications, quality of them determined by quantitative procedures, number of authors, number of research centres involved in the published study.

Importance for the field overall:

Sub-criteria Weight

The research productivity 35%

Ability to attract PhD students (registered/awarded or trained) 35%

The overall quality of the research performance (all other indicators for evaluation) 30%

Importance for the universities:

Sub-criteria Weight

The research productivity 35%

Ability to attract PhD students (registered/awarded or trained) 35%

The overall quality of the research performance (all other indicators for evaluation)

30%

Importance for the Government Organisations:

Sub-criteria Weight

The research productivity 35%

Ability to attract PhD students (registered/awarded or trained) 35%

The overall quality of the research performance (all other indicators 30%

Page 29: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 27

for evaluation)

3.4 Panel 4: Biological and Agricultural Sciences, Biological Sciences

3.4.1 Field-specific definition of keywords in the assessment criteria Research excellence

The assessment criterion on Research Excellence required the panels to assess the quality level of the RU outputs in terms of originality, significance and rigour.

The Panel defined these keywords as follows:

• Originality: This keyword was interpreted in its usual academic sense, i.e. the presentation of new knowledge or ideas, not exclusively based on a summary, review or synthesis of earlier published work, nor existing knowledge simply presented in a new way. The originality might relate to experimental results or survey findings, or to theories or methods, or to possible applications, such as in industry or policy, or to the transfer of knowledge from one field to another.

• Significance: Given the second criterion below (i.e. Societal Relevance; see 1.2.2 below), this keyword was interpreted primarily in a scientific sense, i.e. the potential (or actual, if time since publication allowed) influence of RU outputs on further scientific activity, wherever carried out. This might depend on the species or variety studied, e.g. work on the plant Arabidopsis (the “white mouse” of botany) would suggest greater (wider) significance than that done on a generally unfamiliar plant.

• Rigour: This keyword was also interpreted in the usual academic sense, i.e. evidence of appropriate scientific methods (e.g. choice of plant variety, sample selection and size, testing) and (as appropriate) adequate statistical significance or pro-application argument. Approximate synonyms are robustness, reliability, validity.

Societal Relevance

The assessment criterion on Societal Relevance required the panels to assess the impact of the RU activities in terms of reach and significance.

The Panel defined these keywords as follows:

• Reach: This keyword was interpreted in terms of the range, in terms of numbers or size, of interested groups (“stakeholders”) actually or potentially in contact with the output, directly or indirectly (e.g. through the media). This range might include various groups within the Czech Republic, e.g. fellow-scientists, farmers, food producers, policy-makers, general citizens, or their equivalents in other countries.

• Significance: This keyword was interpreted in terms of the likely influence on the thoughts or actions of stakeholders reading the output or being made aware of it (e.g. through the media), including economic, cultural or welfare effects arising from changed understanding and/or behaviour by those stakeholders.

3.4.2 Importance of the sub-criteria for fields and RO typologies The Panel defined the importance to be attributed to specific sub-criteria in the context of the field and for the RO typologies in the field. The Panel did so in the form of %s, even though the overall judgment was not to be based on arithmetic. The Panel

Page 30: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

28 R&D Evaluation Methodology and Funding Principles

could also decide that all sub-criteria are equally important, or that no distinction was to be made for the different RO typologies.

The Panel took into consideration only the RO typologies that are active in the field in the Czech Republic.

Institutional management and development potential

Importance for the field overall:

Sub-criteria Weight The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 50%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management

50% (but see below)

The Panel considered that these two sub-criteria, at least for the field under their scrutiny, might be better as three, with the second split into (i) Research Strategy, which is primarily the responsibility of leading scientists within an institutional and policy framework, and (ii) Human Resources (HR) (and perhaps other, e.g. financial) Management, which is at least partly the responsibility of university and other administrators within a wider framework for such parameters and processes such as salary scales, promotion regulations, health and safety considerations, etc. The Panel suggested that, if this were done, the weights for the second sub-criterion should be split as 10% for Research Strategy and 40% for HR Management.

Importance for Scientific Research Organisations:

Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 50%

Research strategy, i.e. quality and feasibility of the research plan 10%

(HR) management, i.e. quality of HR management 40%

The Panel agreed that the same weights for these sub-criteria as for its field as a whole should apply to scientific research organisations such as universities, and primarily conducts basic research. In line with the comment under the first table in this sub-section, the above table is split into three weights, which the Panel considered should be the same as for the field as a whole.

Membership of the national and global research community

Importance for the field overall: Sub-criteria Weight

Level of participation and recognition at the international level 70%

Level of participation and recognition at the national level 30%

The Panel noted that scientific communication and debate is increasingly international, with academic preference often in favour of greater international contacts. Such considerations suggested weights of 70% and 30% for the field overall.

It was also noted that international (and national) exchange of students was important to “Participation and Recognition”, and that the national origins of staff themselves would lead naturally to greater or lesser “community membership”, if only through linguistic skills and personal contacts.

Furthermore, the quality of “participation” should be considered, from low levels such as formal institutional memberships and occasional informal contacts, to high levels

Page 31: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 29

such as long-term and in-depth collaboration in scientific work, including in some cases leadership (rather than partnership or “followership”) in scientific networks.

A final point of consideration was that the guidance provided to the Panel as regards “starred quality” assessments was often stated in highly “international” terms, e.g. to award a score of 2* required at least some level of international recognition.

Importance for Scientific Research Organisations:

Sub-criteria Weight Level of participation and recognition at the international level 80% Level of participation and recognition at the national level 20%

For scientific research organisations such as the universities, which primarily conduct basic research, the international aspects of community membership, participation and recognition are particularly important, and so the Panel decided to award weights of 80% and 20% to international and national levels respectively.

Opportunities to attend international meetings, conferences, workshops, etc. should be readily available, particularly for young researchers. This should result in more cooperation and collaboration with researchers throughout the world as well as invitations to join editorial boards, conference committees, etc. (many of who would welcome Central European participants, especially younger ones). In addition, researchers should seek membership of national and international societies, and hope to be recognised through the award of distinguished memberships, fellowships of major international societies, etc. as well as through “best conference paper” awards and invitations to give keynote addresses.

Research performance

Importance for the field overall:

Sub-criteria Weight Research productivity 40% Ability to attract PhD students (registered/awarded or trained) 20% Overall quality of research performance (all other indicators for evaluation) 40%

The Panel noted some possible confusion within the above sub-criteria, in that the third sub-criterion deals with “overall” quality (of “research performance”) which might be taken to overlap somewhat with the first two sub-criteria. However, this did not materially affect the award of scores. The term “productivity” was interpreted in a largely quantitative sense, though with due attention to the “international” terms of the scoring guidance.

It was noted that only universities are entitled to enrol PhD students and award PhD degrees, but that such students sometimes do their research at the institutes of the Academy of Sciences or in government laboratories, with teaching and supervision of each student spread amongst both institutions. There are related issues of funding, e.g. state scholarships (stipends) and project finance (often short-term).

In many countries, there are now well-established criteria for evaluating research, which, among other things, impacts the allocation of research funding. The productivity of research is one of the main criterion for assessing research performance, and largely depends on the output of the researchers in major high-impact journals and conference proceedings, the amount of grant funding attracted as

Page 32: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

30 R&D Evaluation Methodology and Funding Principles

a result of the research activity and the source of that funding, and the number of PhD students attracted to the research topic and successfully supervised.

Page 33: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 31

Importance for Scientific Research Organisations: Sub-criteria Weight

The research productivity 40% Ability to attract PhD students (registered/awarded or trained) 10% The overall quality of the research performance (all other indicators for evaluation) 50%

Given that the university primarily conducts basic research, the ability to attract PhD students was regarded by the Panel as inherent in its nature and strategy. Thus the “overall quality” sub-criterion was given somewhat greater weight, i.e. 50%, with the “PhD attractiveness” sub-criterion lowered correspondingly to 10%.

3.5 Panel 5: Social Science, Sociology

3.5.1 Field-specific definition of keywords in the assessment criteria Research excellence

The assessment criterion on Research Excellence required the panels to assess the quality level of the RU outputs in terms of originality, significance and rigour.

The panel defined these keywords as follows:

• Originality: research that provides new theoretical and empirical perspectives in the field, asks new questions, stimulating new research ideas, goes into new directions, puts forward new thoughts and unusual combination of methods from different fields (transdisciplinary work), or provides a methodological innovation. A key issue is how interesting and novel the research is in light of the current state-of-art knowledge. Some aspect of surprise or illumination for the reader could be a factor.

• Significance: research that opens up a new field, provides a contribution to the literature or adds to the field or a set of fields. This could also be a critique that brings new insight or that changes the view of how things are done. Significant research has the potential to change the perspectives as well as the daily practices of researchers. Significant research needs to be world leading, i.e., research results can certainly apply to, but cannot be limited to the local context.

• Rigour: research that is methodologically and theoretically sound, provides transparency about how data is used, makes sense of the research tools employed in the field. The argumentation should be consistent and followed throughout the work. Essentially it is about showing mastery of the research craft.

Societal Relevance

The assessment criterion on Societal Relevance required the panels to assess the impact of the RU activities in terms of reach and significance.

The panel defined these keywords as follows:

• Reach: relevant audiences are policy makers (local, national, international), general public, industry, NGOs/civil society sector. Research that feeds into policy-making and legislative processes. Examples could include contract research for policy makers, involvement in expert groups, commissions, media presence and visibility, coverage in popular publications.

• Significance: foremost, the research has to be of high quality (see 1.2.1) as bad research is always socially irrelevant and possibly even destructive. It has to have the potential to be applied in order to provide solutions for important societal

Page 34: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

32 R&D Evaluation Methodology and Funding Principles

problems or point towards potential solutions and policy directions. It may be research providing new or critical perspectives that question taken-for-granted-constructs and approaches.

3.5.2 Importance of the sub-criteria for fields and RO typologies The panel defined the importance to be attributed to specific sub-criteria in the context of the field and for the RO typologies in the field. The panel did so in the form of %, even though the overall judgment was not to be based on arithmetic. The panel could also decide that all sub-criteria are equally important or that no distinction was to be made for the different RO typologies.

The panel took into consideration only the RO typologies that are active in the field in the Czech Republic.

Institutional management and development potential

Importance for the field overall: Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 35%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management 65%

While there are some exceptions to this rule, in general technical equipment is less important in sociology. There is a threshold that has to be met, and access to research infrastructure is the basic issue in this criterion. Every institute should have access to the necessary printed and electronic bibliographic resources, as well as access to databases and the computational tools necessary for data analysis. National archives are a cost efficient solution for providing adequate access. Stable funding that supports and encourages both curiosity driven and applied research is necessary. Strategy and human resource management is the more important aspect. This aspect is more influenced by what the research unit does and how it acts strategically, not where it is located and what the opportunities or constraints are for having proper facilities. This weighting would not change across different RO typologies.

Membership of the national and global research community

Importance for the field overall: Sub-criteria Weight

Level of participation and recognition at the international level 60%

Level of participation and recognition at the national level 40%

The international level is more important than the national one, particularly in small countries with relatively small research fields. To a large extent, research recognition at the national level depends on activity at the international level. It is difficult to think about a sociology that is not international. At the same time research contributions at the national level needs to be properly recognised because social science has a large impact on national social development, policy making, and critical discourse.

Page 35: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 33

Importance for Governmental Labs:

Sub-criteria Weight Level of participation and recognition at the international level 40% Level of participation and recognition at the national level 60%

For the RO typology Governmental Labs it is more important to have position on national issues: the weighting is reversed - 40% / 60%. The reasoning behind this is that a research organisation advising the Government needs to have very high local legitimacy, and therefore participation and recognition at the national level is extremely important. This, however, does not take away that research in Government Labs needs to be of high quality and have international academic credibility.

Research performance

Importance for the field overall:

Sub-criteria Weight The research productivity 40% Ability to attract PhD students (registered/awarded or trained) 20% The overall quality of the research performance (all other indicators for evaluation) 40%

The panel believes that productivity and quality should have an equal weight. This double pressure captures the great challenge facing any research institution. Training the new generation of researchers is obviously important, but the panel believes that research productivity and a quality is a pre-requisite for successful PhD training, therefore this criterion is weighted only at a fifth of the total weight. The criterion for the ability to attract PhD students is slightly arbitrary and subject to gaming in Czech Republic. It is important to recognise not only the attraction of PhD students, but also how they are supervised, how many PhD students actually graduate, and how long they need to finish their dissertation. This is also an indication of the quality of human resource management.

Importance for Governmental Labs:

Sub-criteria Weight The research productivity 50% Ability to attract PhD students (registered/awarded or trained) - The overall quality of the research performance (all other indicators for evaluation) 50%

For governmental labs there are no PhDs. Research productivity and overall quality of research performance is 50% / 50%.

Page 36: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

34 R&D Evaluation Methodology and Funding Principles

3.6 Panel 6: Humanities, Philosophy

3.6.1 Field-specific definition of keywords in the assessment criteria Research excellence

The assessment criterion on Research Excellence required the panels to assess the quality level of the RU outputs in terms of originality, significance and rigour.

The panel defined these keywords as follows:

• Originality: a research output that is original may develop a new theory, offer a novel perspective or one that opens up a new way of thinking or even an entire new field; and/or it may work on material that has not been researched before, or not in this way.

• Significance: a research output is significant to the extent that it enables progress in the field and is recognised as required reading in the field or a sub-field.

• Rigour: a research output that has rigour will show it in two areas: (i) argumentative rigour: the quality and clarity of the argumentation that substantiates its claims; and (ii) scholarly rigour: showing awareness of the relevant literature and supporting this with an appropriate bibliographical infrastructure.

Societal Relevance

The assessment criterion on Societal Relevance required the panels to assess the impact of the RU activities in terms of reach and significance.

The panel defined these keywords as follows:

• Reach: the reach of an activity of societal relevance refers to its degree of dissemination in relation to geographical spread and/or type of target audience; it may also refer to broadcast visibility, e.g. whether the activity is taken up by local/regional, national or global media.

• Significance: the significance of an activity of societal relevance refers to demonstrable effects it is shown to have had within the target audience; this might be significant work that does not have high reach, such as research with a local hospital, research linked to the traditional culture or identity of the region or nation; a significant activity must show impact, by demonstrable behavioural change in the target audience e.g. schools, museum-goers, local history societies etc.).

3.6.2 Importance of the sub-criteria for fields and RO typologies The panel defined the importance to be attributed to specific sub-criteria in the context of the field and for the RO typologies in the field. The panel did so in the form of %, even though the overall judgment was not to be based on arithmetic. The panel could also decide that all sub-criteria are equally important or that no distinction was to be made for the different RO typologies. The panel took into consideration only the RO typologies that are active in the field in the Czech Republic.

Page 37: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 35

Institutional management and development potential

Importance for the field overall: Sub-criteria Weight

The quality adequacy of the research environment, i.e. research capacity, funding stability, availability of equipment 35%

Research strategy & (HR) management, i.e. quality and feasibility of the research plan and quality of HR management 65%

The panel took the view that these two sub-criteria adequately covered the field of Criterion A, and weighted them in relation to overall importance. They noted three points:

i. A RU may have a positive research environment or its research environment may fail to provide a framework for optimum research activity (see ii below); in either case, as far as the material and physical environment is concerned this is unlikely to be something the RU itself is directly responsible for. Research strategy, on the other hand, is at least in part (other elements may come from the EvU or the HEI itself) the responsibility of the RU leadership, and can be assessed from the self-assessment report. A good research strategy demonstrates ambition, vision, leadership and an awareness of how the RU team works now as a research unit and should develop in the future. It will include a clear idea of the current and the planned demographic shape of the RU, including numbers, proportions and management (training, support) of PhDs and postdocs, early-career staff, staff of both sexes and a range of age-bands, as well as seniority.

ii. As noted above, issues of research infrastructure concern the adequacy (age, range) of all material elements that support research; in the case of humanities fields like philosophy, this would include libraries, access to online databases and journals, buildings, facilities, IT and office space. These aspects are assessed from the self-assessment report. It may be the case that a good research strategy is hampered by a poor level of research infrastructure, in which case the combined grade of this Criterion will need to be supplemented by this opinion, for example showing that the RU is doing very well and with improved resources it could perform even better.

iii. The panel noted that while research in philosophy is predominantly carried out at the level of the individual scholar, group interaction is nonetheless important and growing in importance internationally. Thus a RU needs to focus also on management of such elements as staff and student seminars, PhD training and support, visiting scholars and exchange visits etc.

Membership of the national and global research community

Importance for the field overall: Sub-criteria Weight

Level of participation and recognition at the international level 65%

Level of participation and recognition at the national level 35%

The panel took the view that these two sub-criteria adequately covered the field of Criterion B, and weighted them in relation to overall importance. They noted one point:

Participation and recognition at the international level will also imply that national level participation and recognition exist, as a RU would not have international partners alone. However, the difference, which is reflected in the weighting, is that

Page 38: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

36 R&D Evaluation Methodology and Funding Principles

international collaborations indicate a more important and significant positioning of the RU in the discipline, whereas national participation alone does not. If its collaborations are purely national, that would not be regarded as a positive sign, even though this would be mitigated to some degree if the research is focused on national or regional material; however, even in such cases and certainly in broader-based research, participation and recognition at the international level is essential.

Research performance

Importance for the field overall:

Sub-criteria Weight The research productivity 50% Ability to attract PhD students (registered/awarded or trained) 10% The overall quality of the research performance (all other indicators for evaluation) 40%

The panel took the view that these three sub-criteria were problematic and that the full Metodika research assessment will require some adjustment. It thus weighted them with some caution, and noted the following points:

i. The first sub-criterion - Research productivity - refers to overall numbers of research outputs of sufficient quality, as measured against FTE of research-active staff in the RU. In the light of problems with the other two sub-criteria, this was weighted as equivalent to the other two combined – but in the future, if the other sub-criteria are reformulated, that might need to change.

ii. The second sub-criterion - Ability to attract PhD students - is currently intended to reflect the popularity or prestige that an RU and its staff have in the eyes of potential PhDs, who are the future of the field. The panel feel this is a too limited criterion of excellence, particularly given the extreme clustering of PhD intake in philosophy in the Czech Republic. The question of PhD students should be assessed not only in terms of simple numbers recruited but on the basis of more information on PhD student trajectories; in particular, information is needed on: FTE (whether students are ‘internal’ or ‘external’, [co-]supervised elsewhere, full-time or part-time); outcomes (completion date, drop-out rate, length of time taken) and ultimate careers (whether, after a specified time, the PhDs are ‘inbred’, i.e. return to a post in this RU; or attain an academic post elsewhere, temporary or permanent, in the CR or abroad; or take up another career; or remain unemployed).

iii. In relation to the third sub-criterion - The overall quality of the research performance - the panel considered this particularly difficult to weight because, as termed, it appears to incorporate the other two. It was suggested that it could be re-named as ‘Other indicators of research performance’. In particular, this is the place where evidence of external (national or international) research funding obtained is to be assessed. The panel used it largely in this way.

Page 39: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 37

4. Examples of SPE panel evaluation reports

4.1 Introduction In this chapter we reproduce two examples of the SPE panel evaluation reports. These are the reports for the Research Units in Evaluated Units that gave their consent for the publication of the reports.

We include only the main reports, not the outcomes of the submitted outputs review, which were delivered as appendixes to the panel evaluation reports.

Below we give first an overview of the evaluation criteria and the relevant indicators that were used in the SPE.

4.1.1 Institutional management and development potential Starred quality level

Definition

4 Quality of the managements and the research infrastructure is outstanding and beyond international norms

3 Quality of the managements and the research infrastructure is state of the art from an international perspective.

2 Quality of the managements and the research infrastructure is adequate. 1 Quality of the managements and the research infrastructure is not adequate at all point

Unclassified Quality of the managements and the research infrastructure cannot be assessed adequately.

The assessment will be based on quantitative and qualitative data and will focus on

• The quality adequacy of the research environment

• Research strategy & (HR) management

The importance attributed to these 2 sub-criteria in the field and for the RO typologies will be defined. This will be in the form of %, even though the overall judgment should not be based on arithmetic. The panels can also decide that all sub-criteria are equally important.

Particular factors to take into account are:

• The research capacity, including the longer-term financial resources stability, the size of the institution (does it have critical mass), the age and career progression of researchers, the ratio of students involved in research to the overall number of staff members, the ratio of part-time versus full-time research staff, the ratio of teaching versus research time

• The availability and adequacy of the infrastructure for research, including

− The capability of the environment adequately to support state-of-the-art research in the field: the nature and quality of the research infrastructure, investments in its renewal, shared or collaborative use of research infrastructure

− Capacity to gain competitive access to major research infrastructure and facilities, the availability of technical staff

• The quality of the research strategy, including

− The strength of the research plan (e.g. statement on main objectives and activities, balance between long-term and short-term research, the

Page 40: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

38 R&D Evaluation Methodology and Funding Principles

development of infrastructure to facilitate research), its feasibility and sustainability

− The approach for the development and maintenance of structures and practices that foster good research

• The adequacy of human resources development, including

− Quality of the processes for career development (appraisal and monitoring systems, competency framework, frequency of performance reviews and the employees’ feedback, promotion criteria and individual targets),

− The career development of PhDs and postdocs (objectives and outcomes of the PhD programme, the support to early-career researchers, approach to PhD supervision, educational components for the training of the PhDs, the attention for gender)

− The level of inbreeding

4.1.2 Membership of the national and global research community

Starred quality level

Definition

4 The RU participates and is recognised in excellent international networks involving the international leaders in the field

3 Participates and is recognised in international and national networks 2 Participates only or mostly in national networks 1 Participates only or mostly in regional or local networks Unclassified Little or no substantive collaboration

The assessment will be based on quantitative and qualitative data and will focus on

• Level of participation and recognition at the international level, and

• Level of participation and recognition at the national level

The importance attributed to these 2 sub-criteria in the field and for the RO typologies will be defined. This will be in the form of %, even though the overall judgment should not be based on arithmetic. The panels can also decide that all sub-criteria are equally important.

Particular factors to take into account are:

• The positioning of the RU at the international level, including

− The intensity and quality of the collaborations: the profile and prestige of the partner organisations, incoming and outgoing study visits, the geographical distribution of the partner organisations, the level of co-publications

− International competitiveness: the capability successfully to participate in international competitive projects, the competitive use of major international research infrastructure

− Esteem: the interest of the international conferences organised by the RU for an international audience, membership of international editorial boards of journals

• The positioning of the RU at the national level, including

− The intensity and quality of the collaborations: the RU involvement in centres and research infrastructure in the country, the involvement of the RU in PhD training,

Page 41: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 39

− National competitiveness: the capability successfully to participate in international competitive projects, the shared or collaborative use of RI, the focus and diversity of research collaborations and partnerships, national co-publications

− Esteem: scientific awards and memberships of scientific advisory boards in academia

4.1.3 Research excellence Starred quality level

Definition

4 Quality that is world leading in terms of originality, significance and rigour. 3 Quality that is internationally excellent in terms of originality, significance and

rigour but which nonetheless falls short of the highest standards of excellence. 2 Quality that is recognised internationally in terms of originality, significance and

rigour. 1 Quality that is recognised nationally in terms of originality, significance and rigour Unclassified Quality that falls below the standard of nationally recognised work

This criterion assesses scientific research excellence. It sets high-level requirements for the submitted outputs to reach high scores against excellence: we used the REF description for this criterion, which is the PRFS setting the highest possible expectations for research excellence. The intensive competitive edge of this criterion is also defined by the limits set for the number of research publications and typologies of publication that can be submitted – see the next chapter describing the specific rulings.

We designed a two-stage evaluation process with a clear division of roles for the panels versus referees:

• The two referees assess the research outputs and assign to each publication a starred quality level, accompanied by an explanatory statement

• The panel member(s) expert(s) in the field assign(s) the final starred quality level, based on the average scores for all submitted scholarly outputs as assigned by the referees, and informed with bibliometric data (JIF)

The panel members will define the field-specific interpretation of the terms originality, significance and rigour prior to the review by the referees and the referees will not have access to the bibliometric data.

As the focus of assessment is on scientific excellence, this criterion will be equal for all research organisation typologies.

The next chapter describing the specific rulings gives information on which publications are eligible for this criterion as well as the minimum and maximum number that could be submitted.

4.1.4 Research performance

Starred quality level

Definition

4 Research performance is world-leading or internationally excellent in terms of outputs, productivity and overall quality

3 Research performance is internationally competitive in terms of outputs, productivity and overall quality

2 Research performance is nationally competitive in terms of outputs, productivity

Page 42: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

40 R&D Evaluation Methodology and Funding Principles

Starred quality level

Definition

and overall quality 1 Research performance is regionally or locally competitive in terms of outputs,

productivity and overall quality Unclassified Research performance is of little to no competitiveness

The assessment will be based on quantitative and qualitative data and will focus on

• The research productivity

• Ability to attract PhD students

• The overall quality of the research performance

The importance attributed to these 2 sub-criteria in the field and for the RO typologies will be defined. This will be in the form of %, even though the overall judgment should not be based on arithmetic. The panels can also decide that all sub-criteria are equally important.

Particular factors to take into account are:

• The research productivity, including the trends and typologies of research outputs, bibliometric analyses and the share of publications in national/international journals. It should be mentioned that research productivity will not be established arithmetically. However, the panels will obviously take into account the size of the research unit in terms of FTE researchers. The production will be considered in terms of trends and typologies of publications (set also against the field total in the CR).

• The ability to attract PhD students, including, the number and trends of PhD students enrolled and PhDs awarded, the effectiveness of the PhD education and trend (ratio PhDs awarded/enrolled), the number and trends of PhD students trained (in institutes that are not entitled for enrolment), the level of investment in PhD training (ratio PhD students/researchers, trained PhDs/researchers)

• Overall quality, including the capacity to gain competitive funding from international and national sources (competitive and contract research), international citations, the competitive use of major national and international RI, the value of the activities from a scientific or socio-economic perspective (narrative)

4.1.5 Societal relevance

Score Definition 4 In the context of its field, the work has outstanding impacts in terms of reach and

significance 3 In the context of its field, the work has very considerable impacts in terms of reach

and significance 2 In the context of its field, the work has considerable impacts in terms of reach and

significance 1 In the context of its field, the work has modest impacts in terms of reach and

significance Unclassified The work has little to no impacts in terms of reach and significance

Prior to the assessment, the panel members will define the field-specific interpretation of the terms reach and significance.

Particular factors to take into account are:

• The intensity and quality of the knowledge and technology transfer

Page 43: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 41

− For industry: the collaboration with industry, the volume of competitive & contract research with/for industry, income from the commercialisation of research outputs, IPR-related outputs (patents) and the geographical distribution of the patent offices, shared or collaborative use of RI, participation in incubators or clusters, the value of the research activities (narrative)

− For other societal actors: the volume of competitive & contract research with/for the public sector or other societal actors, the publication of non-traditional scholarly outputs, the value of the research activities (narrative)

• Esteem: membership of advisory boards

• Impacts: narrative on societal impacts

4.2 Physical Sciences - Earth and related Environmental sciences: RU in the Institute of Analytical Chemistry of the AS CR Scientific activities of the Institute of Analytical Chemistry AS CR are focused on research and development in the field of analytical chemistry, particularly in the area of biological sciences with a focus on genomics, proteomics, medicine, human health and environmental protection. The Evaluated Unit has expanded its potential and tapped into a diverse set of science and technological fields. The Institute collaborates with two Centres of Excellence: the Centre for Advanced Bioanalytical Technologies and the Centre for Studies and Toxicity of Nanoparticles. International collaborations exist with the USA, Canada, Turkey and Brazil.

The RU “Environmental Analytical Chemistry” is one of the 5 units that constitute the EvU. Its main activities include (i) automation in analytical chemistry, (ii) flow analysis, gas liquid chromatography and atomic spectroscopy methods, (iii) preconcentration and separation techniques, (iv) trace and ultra-trace analysis of environmental samples and clinical, biological and modern advanced materials, and (v) nanoparticles in the environment. The RU currently consists of 9 researchers, which represents 22.4 % of staff of the EvU. The activities of the RU complement other research units of the EvU. They effectively share common infrastructure available, for example a mechanical and electronic workshop.

4.2.1 Results of the panel evaluation - overview Criterion number Quality criteria

Starred quality level reached

A Institutional management and development potential 3 B Membership of the national and global research community 2 C Research excellence 4 D Research performance 2 E Societal relevance 1

4.2.2 Criterion A: Institutional management and development potential

Starred quality level Definition

4* Quality of the managements and the research infrastructure is outstanding and beyond international norms

3* Quality of the managements and the research infrastructure is

Page 44: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

42 R&D Evaluation Methodology and Funding Principles

Starred quality level Definition

state of the art from an international perspective. 2* Quality of the managements and the research infrastructure is adequate.

1* Quality of the managements and the research infrastructure is not adequate at all point

Unclassified Quality of the managements and the research infrastructure cannot be assessed adequately.

The funds of the RU seem to be stable with a share of EvU institutional funding ranging from 9% to 13%. The funds of the RU are obtained successfully from national competitive funding, while other funding sources are rather limited.

The number of FTE in the RU is stable over time, which seem to be comparable to the trends at the level of EvU. The number of postdoctoral fellows has dropped from 10 in 2010 to only 4 in 2014, while the number of associated scientists increased from 1 to 10 in recent years – an indication of successful career progression of the staff.

The infrastructure and conditions to perform research seem to be appropriate. This covers both instrumentation and facilities as well as technical support to maintain the laboratories including the access to mechanical and electronic workshop available at the EvU. Usage of RI outside the EvU is not described in the self-assessment report indicating self-sufficiency of the RU in terms of equipment and services.

The research strategy is limited to the development of analytical methodologies in aerosol analysis and their applications. Limited information is provided in the self-assessment report on how this strategy meets the strategic goals established outside the RU (e.g. by the EvU or the national and international strategic goals).

Regular evaluations are carried out to assess staff promotion with the criteria being highly focused on the number of research articles. The number of postdocs is declining in recent years. The PhDs are not retained at the RU level . The gender distribution is balanced. The PhDs defend their thesis at the respective universities. Their topic is related to the research projects implemented at RU, which offers the possibility to continue their research career on these topics in the future.

4.2.3 Criterion B: Membership of the national and global research community

Starred quality level Definition

4* The RU participates and is recognised in excellent international networks involving the international leaders in the field

3* Participates and is recognised in international and national networks 2* Participates only or mostly in national networks 1* Participates only or mostly in regional or local networks Unclassified Little or no substantive collaboration

The international collaboration is limited to a few good international partners that compliment their research activities. Outgoing visits are reported in the self-assessment report and they seem appropriate. Incoming visits are however lacking. Outgoing visits are reported in the self-assessment report and they seem appropriate. Incoming visits are however lacking.

The RU organises international meetings/conferences in aerosol science on a biannual basis. There is no information provided on participation in editorial boards of international journals in the self-assessment report.

Page 45: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 43

The RU is not sufficiently involved in competitive international projects, neither does it report on the use of research infrastructures abroad.

Selected articles are co-published with researchers outside the RU at both national and international level. The performance of the RU at the national level is considered good, including PhDs involvement and co-authorship in articles. There is however no information reported on the shared use of the RI facilities.

The RU successfully obtained national competitive funding and is well integrated nationally. The members of the RU obtained national and international prizes and are very active in thematic scientific organisations and societies. They report important collaboration with 4 different national research groups that compliment well their research activities.

Overall, the research group seem to be very well integrated at the national level but less so at the international level.

4.2.4 Criterion C: Research excellence

Starred quality level Definition

4* Quality that is world leading in terms of originality, significance and rigour.

3* Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence.

2* Quality that is recognised internationally in terms of originality, significance and rigour.

1* Quality that is recognised nationally in terms of originality, significance and rigour

Unclassified Quality that falls below the standard of nationally recognised work

The paper of Mikuška et al. (2012) published in Analytica Chimica Acta can be classified as a world class paper dealing with a very challenging and difficult topic related to sampling artefacts in atmospheric measurements. The experimental part combined extensive experimental work supported by appropriate theoretical background. As a result a new denuder system has been developed that efficiently retains organic and inorganic gaseous compounds from the sampled air. The denuder was also validated in the field conditions.

The paper deals with the improvements of the analytical step which reduces significantly the uncertainty in environmental analysis – the sampling artefacts. Although the numerical estimates of improved uncertainty estimates are not sufficiently elaborated, the paper will certainly contribute to better and more accurate sampling and measurement procedures.

The publication is ranked very high due to the prestigious journal in the field of analytical chemistry.

Page 46: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

44 R&D Evaluation Methodology and Funding Principles

4.2.5 Criterion D: Research performance

Starred quality level Definition

4* Research performance is world-leading or internationally excellent in terms of outputs, productivity and overall quality

3* Research performance is internationally competitive in terms of outputs, productivity and overall quality

2* Research performance is nationally competitive in terms of outputs, productivity and overall quality

1* Research performance is regionally or locally competitive in terms of outputs, productivity and overall quality

Unclassified Research performance is uncompetitive

The articles produced by the RU are published in very good international journals mostly covering environmental analytical chemistry and environmental sciences. Usually papers are published in co-authorship with collaborating institutions. A considerable number of articles in proceedings are reported. The citation rate is below the world and EU 28 averages. Some work has been published in co-authorship with international partners, but in most cases national co-authorship is practised.

Other forms of research outputs are not included in the self-assessment report. There are no patents or other forms of IP described, although the RU is accredited to perform animal experiments.

The data provided for the ability to attract PhD students show that the number and trends of PhD students enrolled and awarded have been declining since 2009. Also the number of students from other research organizations implementing research at the RU shows a significant decrease.

The external funding is minimal and the data reported on the relevance for industry are unclear. It seems that the share of industrial funding is at the level of about 1% only.

4.2.6 Criterion E: Societal relevance

Starred quality level Definition

4* In the context of its field, the work has outstanding impacts in terms of reach and significance

3* In the context of its field, the work has very considerable impacts in terms of reach and significance

2* In the context of its field, the work has considerable impacts in terms of reach and significance

1* In the context of its field, the work has modest impacts in terms of reach and significance

Unclassified The work has little to no impact in terms of reach and significance

Non-academic collaboration was demonstrated through the development of an advanced portable system for explosive agent sampling. However, there was no collaboration reported through participation in incubators and clusters nor in advisory boards outside academia. The RU’s outreach and societal impacts are described only in broad terms and not fully elaborated.

Page 47: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 45

Although in the self-assessment report the activities are described to be of high scientific and socio-economic relevance, the quantitative data disclosed do not support these statements convincingly.

4.2.7 Conclusions and recommendations The research environment and infrastructure at the EvU provide a very good environment for the RU to further progress and develop its potential. The focal area of the research follows novel trends in the field of nanoparticles.

There was insufficient information provided in the self-assessment report on the involvement with the CoE with which the EvU has established good collaboration. Strengthened international collaboration in the EU framework programmes, Horizon 2020 as well as other international funding mechanisms would advance their excellence in research. Moreover, an enhanced exchange programme of researchers would increase the attractiveness of the RU for PhD students both at national and international level.

The RU’s involvement in PhD training has been dramatically reduced in recent years. The reasons were not explained in the self-assessment report and due to the lack of site visit remained unexplained.

There obviously seem to be some problems in relation to research strategy and long-term planning in relation to institutional (EvU), national, and international level.

The nature of the RU’s work and research environment provide numerous possibilities to strengthen research collaboration with non-academic sectors both nationally and internationally, which should be exploited.

4.2.8 Additional conclusions and recommendations following the site visit Due to scheduling conflicts a site visit was not possible to undertake. Therefore the evaluation presented in this panel report is solely based on the information provided in the self-assessment report.

4.3 Social Science – Sociology: RU in the Masaryk University, Department of Sociology

The Department of Sociology at Masaryk University is one of the three ‘most important’ research organisations in the field of sociology in the Czech Republic. It belongs to the Faculty of Social Studies, which has a total of seven departments. The Department of Sociology is divided into the Division of Gender Studies (BA level only), Division of Social Anthropology (BA level only), and the Division of Sociology, which offers PhD training (in both Czech and English).

The Department prides itself of a grand place in the history of European sociology, founded in 1921, by a I. A. Bláha, a student of Émile Durkheim, and home to the so-called Brno school of sociology.

The department focuses on research topics such as globalisation, migration, changes in identity, family, demography, and societal transformations. The Office for Population Studies has been set up as a recent strategic research endeavour that emphasises theory-driven empirical analyses of issues and problems related to family and demography, broadly defined.

Page 48: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

46 R&D Evaluation Methodology and Funding Principles

4.3.1 Results of the panel evaluation - overview Criterion number Quality criteria

Starred quality level reached

A Institutional management and development potential 2* B Membership of the national and global research community 3* C Research excellence 2* D Research performance 2* E Societal relevance 1*

4.3.2 Criterion A: Institutional management and development potential

Starred quality level

Definition

4* Quality of the managements and the research infrastructure is outstanding and beyond international norms

3* Quality of the managements and the research infrastructure is state of the art from an international perspective.

2* Quality of the managements and the research infrastructure is adequate. 1* Quality of the managements and the research infrastructure is not adequate at all point

Unclassified Quality of the managements and the research infrastructure cannot be assessed adequately.

Masaryk University appears to supply the Department with access to basic research infrastructure, including library resources, databases, and software. Despite the Department being relatively small and having a small budget, it is increasingly successful in attracting competitive funding. Thus, the panel members perceive the infrastructure to be adequate. The Department has a clearly defined research strategy in population studies, where all funding appears to be invested. This is an area that is topical and enables international publications. The panel believes that this is very well chosen. From the documentation is not entirely clear whether the research strategy has grown from the department or whether it is a strategic decision at faculty or university level. The self-assessment report states that researchers are regularly assessed for bonuses, but it did not describe the basis for the assessment, nor if there is any relation to research group performance. The trajectory of improvement (attraction of national funding, publications, productivity) is very good. The panel does not perceive of the Department as international state of the art (3) but it is certainly above ‘adequate’. This is a strong 2.

4.3.3 Criterion B: Membership of the national and global research community

Starred quality level

Definition

4* The RU participates and is recognised in excellent international networks involving the international leaders in the field

3* Participates and is recognised in international and national networks 2* Participates only or mostly in national networks 1* Participates only or mostly in regional or local networks Unclassified Little or no substantive collaboration

The panel is under the impression that the Department has good collaborations, although the number of cooperation partners is limited. In 2013, 22% of Web of Science publications had national co-authors, 11% had international co-authors, and half of the best publications were co-authored internationally. The Department is also the Czech hub of the European Value Survey. However, only a few members of the

Page 49: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 47

Department are participating in academic networks. Apparently, the Department has a small group of excellent researchers that travel and publish internationally. Strangely, the individual mobility does not appear to overlap with the international collaborative partners mentioned in the self-assessment report (Tilburg and Yale). At the national level, the key and only collaboration is with institutes in the Academy of Sciences. The panel members acknowledge that the Department participates and is recognised in both international and international networks of high quality. However, the panel really expected membership in the national and global research community on a larger scale than the present.

4.3.4 Criterion C: Research excellence The assessment for this criterion was partly based on the submitted publications. The process during the panel meeting took place in 3 steps:

• First the panel members acting as referees agreed on a common score for each publication, using the ‘research excellence’ criteria

• Second, the average score of all submitted publications was calculated

• Third, the panel decided on the final score for Research Excellence, based on the average scores for the submitted publications and taking into consideration the bibliometric profile of the RU activities, i.e. the data related to citations and journal impact factors

Starred quality level

Definition

4* Quality that is world leading in terms of originality, significance and rigour. 3* Quality that is internationally excellent in terms of originality, significance and

rigour but which nonetheless falls short of the highest standards of excellence. 2* Quality that is recognised internationally in terms of originality, significance and

rigour. 1* Quality that is recognised nationally in terms of originality, significance and rigour Unclassified Quality that falls below the standard of nationally recognised work

Over the five-year period (2008-2012) 64% of the Department’s publications appear in peer-reviewed journals. However, in that period only 8 articles are published in international journals indexed by Web of Science and not all of them are in highly cited journals. According to the bibliometric report, the citation impact skyrocketed in 2012, which is a really good sign of progress. Overall, the Department does not publish internationally and in the most cited journals, but some publications have made it to that level. The four publications submitted for review are signs of potential rather than a representation of the Department at large. It should be stressed that the four submitted publications were consistently of a very high quality. Had the evaluation of this criterion been based only on the best output, the panel would have given research excellence a score of 3. However, the panel concluded that the best output is not representative of the bulk of research carried out in the Department and one of the paper appeared to have been written by a visiting researcher, who was not at the Department at the time of publication. The Department has an international profile and is clearly aware of the international research field. It should be possible for this Department to move up towards international excellence in the near future, but as things stand now it is rated as internationally recognised.

Page 50: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

48 R&D Evaluation Methodology and Funding Principles

4.3.5 Criterion D: Research performance Starred quality level

Definition

4* Research performance is world-leading or internationally excellent in terms of outputs, productivity and overall quality

3* Research performance is internationally competitive in terms of outputs, productivity and overall quality

2* Research performance is nationally competitive in terms of outputs, productivity and overall quality

1* Research performance is regionally or locally competitive in terms of outputs, productivity and overall quality

Unclassified Research performance is uncompetitive

Without firm indicators of research FTEs it is difficult to conclusively assess productivity. However, the Department has reasonable publication productivity, consisting of 28 papers in indexed journals and 5 monographs in 2013. Over the period 2008-2013, the Department produced 17% of all Czech sociology papers indexed in Web of Science, and close to 30% of Czech sociology papers indexed in SCOPUS.

The self-assessment report raises many questions regarding PhD education. There are many PhD students but very few seem to graduate. This suggests that the Department does manage to attract PhD students, although there is no information on how they are selected or funded. At the same time, this also suggests that the Department does not do a very good job in training and supervising its PhD students. There is not much indication about the way in which PhD research is supervised, and there is only superficial evidence of any programme available for the PhD students for their training. With 15 new PhD students a year (1.5 PhD per head count researcher), one might doubt whether there is supervisory capacity for these students at the right level.

The Department has been able to attract competitive funding, also internationally, which indicates that they have an international role. The panel finds that research performance is clearly nationally competitive, but not quite internationally competitive.

4.3.6 Criterion E: Societal relevance

Starred quality level

Definition

4* In the context of its field, the work has outstanding impacts in terms of reach and significance

3* In the context of its field, the work has very considerable impacts in terms of reach and significance

2* In the context of its field, the work has considerable impacts in terms of reach and significance

1* In the context of its field, the work has modest impacts in terms of reach and significance

Unclassified The work has little to no impact in terms of reach and significance

The panel members found the department’s societal impact to be modest. The self-reported narrative revolves around potentiality, that is, it gives descriptions of societal relevance and impact that ‘can be’, mostly on topics to which any sociologist would agree. However, the report does not explain what exactly happened, if anything. Nor does it explain what the Department would like to achieve in terms of societal relevance. The only project with a real impact seems to be project on Roma inclusion, which is a combination of research and public communication that appears to be both important and significant. A project for the Ministry of Culture also seems to have been positive in terms of focus and data collection. There are no ‘non-traditional

Page 51: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 49

scholarly outputs’. Overall, societal relevance is good, but there is simply not much of it. The panel believes that the Department has very good potential to increase its societal relevance.

4.3.7 Conclusions and recommendations The Department of Sociology at Masaryk University is well-organised as a research unit. The panel is particularly impressed by the explicit and timely research focus on population studies and the very high quality of its best publications send for review. In the panel’s view these are clear signs that, in terms of the criteria and scoring system applied in this pilot evaluation report, the Department has strong potential for significant improvement in the mid-term (i.e., 2-5 years). In short, the panel thinks that the Department is on the right track, that it should continue with what it is doing, but step up the ambition on research strategy, PhD training, and publication. Below is a short list of general recommendations based on the collective research experience of the panel reviewers, including recommendations for future panel evaluations. The panel recommends that the department:

• Define an explicit research strategy for the Office of Population Studies, detailing what it is supposed to achieve, what is its added value and how it leads to research excellence

• Continue collaborations with Vienna Institute of Demography as a top ranking institution in the field

• Continue to train faculty how to publish for visibility, recognition, and impact and thereby move out of the current focus on publishing many articles in Czech, since more international publications are a prerequisite for further improvement.

• Structure the PhD programme around a commitment to train the next generation of sociologists, and to prepare them for research careers within and outside of academia

• Reflect on the societal relevance of the department’s research. More concrete examples of impact should be included in the future self-assessments. It is important to systematically collect, corroborate, and present information on societal relevance and national and international collaborations.

4.3.8 Additional conclusions and recommendations following the site visit The first thing to note in relation to the site visit is that the panel was very impressed by the enthusiasm and the strong commitment to the institution of both students and faculty. The visit revealed that the Department has critical mass, a young and enthusiastic staff, high commitment to the institution, and really good management. Thus, while the report communicated that this is a good Department the visit further reinforced its strong potential for development.

Second, the question marks raised in relation to the PhD programme remain in place after the visit. However, during the site visit the panel became acutely aware that the problem of low graduation rates arises as consequence of a sub-optimal national educational system, and is beyond the control of the Department. Still, the panel argues that with a strong and devoted management team, the Department will be able to improve the PhD programme despite the limitations of the system. The visit revealed some ad-hoc individual initiatives focused on seminar presentations and discussions for PhD students that could easily be institutionalised.

Third, the visit revealed that a) societal impact is something that has not been valued at all in the current system, and b) individual researchers feel strongly that the research should have societal relevance, and they would like to contribute to it.

Page 52: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

50 R&D Evaluation Methodology and Funding Principles

Currently there is no incentive to do it and researchers can even get punished to some extent for devoting time to these activities.

Fourth, the site visit revealed that the Department had substantial informal international collaborations that were not accounted for in the report. One reason might be the somewhat unclear instructions for the report, but another reason is that many such collaborations are in fact built around teaching collaboration that spill over into research collaborations. Moreover, it became clear during the visit that the Department has attracted foreign post-doctoral researchers, who appear to contribute very productively to the Department. Thus, the Department seemed to be more internationally oriented than came across from the documentation.

In sum, the site visit underscored the importance of having very clear instructions and definitions for the self-assessment reporting; a) some of the statistics simply do not make sense when analysed at the unit of evaluation, b) instructions for self-assessments must encourage a detailed description of actual activities and discourage commonplace remarks about the field in general.

Page 53: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 51

5. Bibliometric reports at the disciplinary area & field level

In this chapter we reproduce Section A: Characteristics of the field of the bibliometric reports for all the SPE panels. This chapter therefore gives an overview of the bibliometric profile in the Czech Republic at the level of disciplinary areas and fields. Sources are the Web of Science (WoS) and the data in the RD&I Information System (IS).

5.1 Physical sciences This group of indicators are all meant to give the panel an overview of the characteristics of the field at country level, thereby providing a context for interpreting the indicators for a specific Research Unit’s in a specific field.

The same indicators will appear at the level of Research Units in categories B-F below. More details about the indicators and their interpretation will be given there.

Indicator A1. Relative size of the fields’ major area in the publication data from WoS and R&D IS.

Indicator A2. Relative size of the field within its major area in the publication data from WoS and R&D IS.

0%

5%

10%

15%

20%

25%

30%

35%

Physical Sciences

Engineering and technology

Medical and Health sciences

Biological and Agricultural

Sciences

Social sciences Humanities

Distribution of the scholarly publications in the Czech Republic among major disciplinary Areas

(2008-2013)

WoS R&D IS

Page 54: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

52 R&D Evaluation Methodology and Funding Principles

Indicator A3. Number of publications in scholarly R&D IS categories. Physical sciences.

Category Math Phys Chem Earth

Articles in peer-reviewed journals (J) 5386   11297   8503   6952  

Monographs (B) 135   62   41   339  

Book chapters (C) 225   265   339   767  

Proceedings papers (D) 3351   5380   2988   4331  

Indicator A4. Number of and percentage Web of Science publications among all peer-reviewed R&DIS publications. Physical sciences.

Data Source Math Phys Chem Earth Other

Web of Science (N) 3,361   8,361   9,394   3,892   829  

R&D IS (N) 9,097   17,004   11,871   12,389   0  

Web of Science (%) 37%   49%   79%   31%   -­‐  

Indicator A5. Mean and median number of authors and addresses per WoS publication. Mean number of authors and addresses in the Physical Sciences.

Data Source Math Phys Chem Earth

Authors 2.3 134.1 5.0 4.9

Addresses 1.89 16.88 2.63 3.18

Indicator A6. Percentage Web of Science publications in the most cited 10 per cent , 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports. Physical sciences.

0% 5%

10% 15% 20% 25% 30% 35% 40%

Mathematics (Math)

Physical sciences (Phys)

Chemical sciences (Chem)

Earth and related environmental

sciences (Earth)

Other

Distribution of the scholarly publications in the Czech Republic among fields within its major area.

Example: Physical sciences (2008-2013)

Web of Science R&D IS

Page 55: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 53

Indicator A6 is supplied by a listing of WoS journals in each field, ranked by the number of articles from the Czech Republic, specifying each journal taking more than 5 per cent of the total volume of articles in the field.

Indicator A7. Mean Field Normalized Citation Impact, compared to the world average (=1,00) and the EU28 average (=1,00). Physical sciences, comparison with the world average. 2008 2009 2010 2011 2012

Chemical  sciences   0.81   0.92   0.98   0.99   1.24  

Earth  and  related  environmental  sciences   0.98   1.00   1.08   1.10   1.22  

Mathematics   0.91   0.84   1.16   1.09   1.55  

Physical  sciences  and  astronomy   1.20   1.08   1.33   1.45   2.40  

Indicator A8. Number and percentage publications among the top 10%, and 25% most cited publications (world).

Percentile of publications Chemical  

sciences  

Earth  and  related  environmental  

sciences   Mathematics  

Physical  sciences  and  astronomy  

Most cited 10 % 7%   7%   9%   13%  

Most cited 25 % 13%   15%   17%   19%  

Most cited 50 % 11%   13%   10%   6%  

Least cited 50 % 69%   65%   65%   64%  

Indicator A9. Percentage Web of Science publications exhibiting international collaboration in the addresses. Physical Sciences.

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100%  

Math   Phys   Chem   Earth  

Publishing  Pro8ile.  %  of  WoS  publications  in  most  cited  journals  in  the  8ield,  based  on  2014  JIF.  Example:  Physical  Sciences  

Most  cited  10  %   Most  cited  25  %   Most  cited  50  %   Least  cited  50  %  

Page 56: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

54 R&D Evaluation Methodology and Funding Principles

Indicator A10. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared globally in the field.

Environmental Sciences Articles Citation impact

(compared to world)

USA   334   1.91  

France   256   2.43  

United  Kingdom   230   2.72  

Slovakia   216   1.13  

Austria   170   2.75  

0.0  %  

10.0  %  

20.0  %  

30.0  %  

40.0  %  

50.0  %  

60.0  %  

70.0  %  

Chemical  sciences   Earth  and  related  environmental  sciences  

Mathematics   Physical  sciences  and  astronomy  

Percentage  WoS  publications  exhibiting  international  collaboration.    Example:  Physical  Sciences.  (2008-­‐2013)  

Page 57: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 55

5.2 Engineering & Technology Indicator A1. Relative size of the fields’ major area in the publication data from WoS and R&D IS.

0% 5%

10% 15% 20% 25% 30% 35%

Physical Sciences

Engineering and technology

Medical and Health sciences

Biological and Agricultural

Sciences

Social sciences Humanities

Distribution of the scholarly publications in the Czech Republic among major disciplinary Areas

(2008-2013)

WoS R&D IS

Page 58: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

56 R&D Evaluation Methodology and Funding Principles

Indicator A2. Relative size of the field within its major area in the publication data from WoS and R&D IS.

0%  

5%  

10%  

15%  

20%  

25%  

30%  

Civil  engineering  (Civil)  

Electrical,  electronic,  information  

engineering  (EEI)  

Computer  and  information  sciences  (CIS)  

Mechanical  engineering  (Mech)  

Chemical  engineering  (ChemEng)  

Materials  engineering  (Mat)  

Medical  engineering  (MedEng)  

Environmental  engineering  (EnvEng)  

Environmental  biotechnology  (EnvBio)  

Industrial  Biotechnology  (IndBio)  

Nano-­‐technology  (Nano)  

Other  engineering  and  technologies    

Distribution  of  the  scholarly  publications  in  the  Czech  Republic  among  8ields  within  its  major  area.    Example:  Engineering  and  technology  (2008-­‐2013)  

Web  of  Science   R&D  IS  

Page 59: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 57

Indicator A3. Number of publications in scholarly R&DIS categories. Engineering and technology.

Category Civil EEI CIS Mech ChemEng Mat

Articles in peer-reviewed journals (J) 3613   4633   2771   1654   716   4848  

Monographs (B) 278   104   155   54   7   130  

Book chapters (C) 351   303   388   117   44   279  

Proceedings papers (D) 8499   11623   6861   2628   1107   6025  

Category MedEng EnvEng EnvBio IndBio Nano Other

Articles in peer-reviewed journals (J) 118   1244  

-­‐  416  

-­‐  2789  

Monographs (B) 2   66   -­‐   4   -­‐   71  

Book chapters (C) 3   128   -­‐   59   -­‐   229  

Proceedings papers (D) 141   2103   -­‐   290   -­‐   2986  

Indicator A4. Number of and percentage Web of Science publications among all peer-reviewed R&DIS publications. Engineering and technology.

Data Source Civil EEI CIS Mech ChemEng Mat

Web of Science (N) 248   1,218   1,476   1,344   770   3,517  

R&D IS (N) 12,741   16,663   10,175   4,453   1,874   11,282  

Web of Science (%) 2%   7%   15%   30%   41%   31%  

Data Source MedEng EnvEng EnvBio IndBio Nano Other

Web of Science (N) 632   864   856   88   370   2,610  

R&D IS (N) 264   3,541   0   769   0   6,075  

Web of Science (%) 240%   24%   -­‐   11%   -­‐-­‐   43%  

Indicator A5. Mean and median number of authors and addresses per WoS publication. Mean number of authors and addresses in engineering and technology.

Data Source Civil EEI CIS Mech ChemEng Mat

Authors 3.03   2.96   2.69   6.07   4.15   4.65  

Addresses 1.44   1.22   1.32   2.35   1.99   2.17  

Data Source MedEng EnvEng EnvBio IndBio Nano Other

Authors 5.25   3.51   6.02   6.53   4.98   17.15  

Addresses 2.75   1.66   3.07   3.18   2.15   4.09  

Indicator A6. Percentage Web of Science publications in the most cited 10 per cent , 25 per cent, and 50 per cent of the journals in the field (counted from the top by the

Page 60: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

58 R&D Evaluation Methodology and Funding Principles

number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports. Engineering and technology.

Indicator A6 is supplied by a listing of WoS journals in each field, ranked by the number of articles from the Czech Republic, specifying each journal taking more than 5 per cent of the total volume of articles in the field.

Indicator A7. Mean Field Normalized Citation Impact, compared to the world average (=1,00) and the EU28 average (=1,00). Engineering and technology, comparison with the world average.

2008 2009 2010 2011 2012

Civil 0.33 0.48 0.35 0.72 0.71

EEI 0.29 0.27 0.37 0.46 0.65

CIS 0.56 0.37 0.64 1.03 0.85

Mech 0.91 1.21 1.58 0.99 2.09

ChemEng 0.95 1.13 0.86 0.99 1.26

Mat 0.79 0.77 0.84 1.30 0.88

MedEng 0.55 2.68 0.42 1.80 0.93

EnvEng 0.37 0.37 0.38 1.01 0.97

EnvBio 0.91 0.88 0.88 1.08 1.30

IndBio - - - - -

Nano 0.03 0.03 0.02 0.10 0.04

Other 1.17 0.93 0.96 1.38 1.36

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100%  

Civil   EEI   CIS   Mech  Chem  Eng   Mat   Med  Eng  EnvEng   EnvBio   Nano   Other  

Publishing  Pro-ile.  %  of  WoS  publications  in  most  cited  journals  in  the  -ield,  based  on  2014  JIF.  

Example:  Engineering  and  technology  

Most  cited  10  %   Most  cited  25  %   Most  cited  50  %   Least  cited  50  %  

Page 61: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 59

59

Indicator A8. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

Percentile of publications

Civil Electronic Computer Mechanical Chemical

Most cited 10 % 5% 2% 3% 11% 7%

Most cited 25 % 8% 4% 7% 14% 16%

Most cited 50 % 4% 4% 10% 10% 10%

Least cited 50 % 85% 89% 80% 65% 67%

Percentile of publications MedEng EnvEng EnvBio IndBio Nano

Most cited 10 % 5% 1% 8% - 0%

Most cited 25 % 9% 1% 17% - 0%

Most cited 50 % 8% 15% 6% - 0%

Least cited 50 % 77% 83% 69% - 100%

Indicator A9. Percentage Web of Science publications exhibiting international and collaboration in the addresses. Engineering and technology.

Indicator A10. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared to EU28 in the field.

0.0  %  5.0  %  10.0  %  15.0  %  20.0  %  25.0  %  30.0  %  35.0  %  40.0  %  45.0  %  

Civil   EEI   CIS   Mech   Chem  Eng  

Mat   Med  Eng  EnvEng   EnvBio   Nano   Other  

Percentage  WoS  publications  exhibiting  international  and  national  collaboration.    

Example:  Physical  Sciences.  (2008-­‐2013)  

International  

Page 62: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

60 R&D Evaluation Methodology and Funding Principles

Chemical engineering Articles Citation impact

(compared to EU 28)

Italy 29 1.84

USA 28 1.36

United Kingdom 20 1.35

Poland 17 1.11

France 16 2.29

Civil engineering Articles Citation impact

(compared to EU 28)

USA 31 0.86

France 23 0.75

Italy 21 0.71

United Kingdom 19 0.64

Netherlands 18 0.88

Computer and information sciences Articles Citation impact

(compared to EU 28)

USA 101 3.45

Slovakia 88 1.90

Spain 50 1.43

France 47 2.02

United Kingdom 37 2.47

Electrical engineering, electronic engineering, information engineering Articles Citation impact

(compared to EU 28)

USA 54 1.81

France 50 1.74

United Kingdom 49 3.01

Spain 43 1.67

Slovakia 40 1.37

Page 63: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 61

61

Environmental biotechnology Articles Citation impact

(compared to EU 28)

USA 65 2.54

Italy 37 1.53

Slovakia 37 0.58

United Kingdom 36 3.30

France 22 3.50

Environmental engineering Articles Citation impact

(compared to EU 28)

Slovakia 27 2.41

USA 18 0.27

Poland 16 4.44

United Kingdom 14 4.35

Italy 13 6.64

Materials engineering Articles Citation impact

(compared to EU 28)

Slovakia 206 0.69

France 147 1.62

USA 130 2.36

Poland 100 1.26

United Kingdom 91 1.87

Mechanical engineering Articles Citation impact

(compared to EU 28)

USA 158 2.69

Italy 155 1.99

France 154 2.54

United Kingdom 113 3.04

Russia 99 2.39

Page 64: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

62 R&D Evaluation Methodology and Funding Principles

Medical engineering Articles Citation impact

(compared to EU 28)

USA 26 13.06

United Kingdom 21 11.31

Italy 17 3.10

Belgium 16 13.78

Slovakia 16 1.27

Nano-technology Articles Citation impact

(compared to EU 28)

USA 17 0.00

France 13 0.00

Japan 11 0.00

Italy 10 0.00

Russia 7 0.00

Other engineering and technologies Articles Citation impact

(compared to EU 28)

United Kingdom 146 3.31

USA 130 3.32

France 129 2.82

Italy 110 3.30

Slovakia 104 2.70

5.3 Medical & Health sciences

This group of indicators are all meant to give the panel an overview of the characteristics of the field at country level, thereby providing a context for interpreting the indicators for a specific Research Unit’s in a specific field.

The same indicators will appear at the level of Research Units in categories B-F below. More details about the indicators and their interpretation will be given there.

Page 65: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 63

63

Indicator A1. Relative size of the fields’ major area in the publication data from WoS and R&D IS.

Indicator A2. Relative size of the field within its major area in the publication data from WoS and R&D IS.

Indicator A3. Number of publications in scholarly R&D IS categories. Medical sciences.

Category Basic Clinical Health Other

Articles in peer-reviewed journals (J) 2956   19490   2654   2575  

Monographs (B) 20   377   112   91  

0%

5%

10%

15%

20%

25%

30%

35%

Physical Sciences

Engineering and technology

Medical and Health sciences

Biological and Agricultural

Sciences

Social sciences Humanities

Distribution of the scholarly publications in the Czech Republic among major disciplinary Areas

(2008-2013)

WoS R&D IS

0% 10% 20% 30% 40% 50% 60% 70% 80%

Basic medical research (Basic)

Clinical medicine (Clinical)

Health sciences (Health)

Other medical sciences

Distribution of the scholarly publications in the Czech Republic among fields within its major area.

Example: Medical sciences (2008-2013)

Web of Science R&D IS

Page 66: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

64 R&D Evaluation Methodology and Funding Principles

Book chapters (C) 133   1509   242   199  

Proceedings papers (D) 151   843   788   289  

Indicator A4. Number and percentage of Web of Science publications among all peer-reviewed R&DIS publications. Medical sciences.

Data Source Basic Clinical Health Other

Web of Science (N) 4,747   6,909   1,565   0  

R&D IS (N) 3,260   22,219   3,796   3,154  

Web of Science (%) 146%   31%   41%   0%  

Indicator A5. Mean and median number of authors and addresses per WoS publication. Mean number of authors and addresses in the medical sciences.

Data Source Basic Clinical Health Other

Authors 6.47   8.22   -­‐   6.81

Addresses 3.78   5.51   -­‐   4.57

Indicator A6. Percentage Web of Science publications in the most cited 10 per cent , 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports. Medical sciences.

Indicator A6 is supplied by a listing of WoS journals in each field, ranked by the number of articles from the Czech Republic, specifying each journal taking more than 5 per cent of the total volume of articles in the field.

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100%  

Basic   Clinical     Health  

Publishing  Pro-ile.  %  of  WoS  publications  in  most  cited  journals  in  the  -ield,  based  on  2014  JIF.  Example:  Medical  sciences  

Most  cited  10  %   Most  cited  25  %   Most  cited  50  %   Least  cited  50  %  

Page 67: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 65

65

Indicator A7. Mean Field Normalized Citation Impact, compared to the world average (=1,00) and the EU28 average (=1,00). Medical sciences, comparison with the world average.

2008 2009 2010 2011 2012

Basic 0.82   0.92   0.83   1.25   1.23  

Clinical 1.22   1.26   1.71   1.97   2.07  

Health 1.05   1.06   1.18   1.13   1.52  

Indicator A8. Number and percentage publications among the top 10%, and 25% most cited publications (world).

Percentile of publications

Basic medical research

Clinical Health sciences

Most cited 10 % 7%   13%   12%  

Most cited 25 % 13%   14%   21%  

Most cited 50 % 12%   9%   6%  

Least cited 50 % 68%   65%   62%  

Indicator A9. Percentage Web of Science publications exhibiting international and collaboration in the addresses. Physical Sciences.

Indicator A10. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared globally in the field.

0.0  %  

10.0  %  

20.0  %  

30.0  %  

40.0  %  

50.0  %  

60.0  %  

70.0  %  

Basic     Clinical   Health  

Percentage  WoS  publications  exhibiting  international  and  national  collaboration.    

Example:  Medical  and  health  sciences.  (2008-­‐2013)  

International  

Page 68: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

66 R&D Evaluation Methodology and Funding Principles

Basic Medical Research Articles Citation impact

(compared to world)

USA 390 2.15  

United Kingdom 245 2.69  

Slovakia 169 0.74  

France 156 2.27  

Italy 147 3.06  

5.4 Biological & agricultural sciences This group of indicators is meant to give the panel an overview of the characteristics of the field at country level, thereby providing a context for interpreting the indicators for a specific Research Unit in a specific field.

The same indicators will appear at the level of Research Units in categories B-F below. More details about the indicators and their interpretation will be given there.

Indicator A1. Relative size of the fields’ major area in the publication data from WoS and R&D IS.

0% 5%

10% 15% 20% 25% 30% 35%

Physical Sciences

Engineering and technology

Medical and Health sciences

Biological and Agricultural

Sciences

Social sciences Humanities

Distribution of the scholarly publications in the Czech Republic among major disciplinary Areas

(2008-2013)

WoS R&D IS

Page 69: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 67

67

Indicator A2. Relative size of the field within its major area in the publication data from WoS and R&D IS.

Indicator A3. Number of publications in scholarly R&DIS categories. Biological and agricultural sciences.

Category Bio Agri Ani Vet Other

Articles in peer-reviewed journals (J) 13895   4904   1332   1150   438  

Monographs (B) 181   153   16   8   13  

Book chapters (C) 996   348   37   19   6  

Proceedings papers (D) 1680   1497   522   324   112  

Indicator A4. Number of and percentage Web of Science publications among all peer-reviewed R&DIS publications. Biological and agricultural sciences.

Data Source Biological  

sciences  

Agriculture,  forestry,  and  

fisheries  

Animal  and  dairy  science  

Veterinary  science  

Other  agricultural  

sciences  

Web of Science (N) 10,481   1,309   371   736   274  

R&D IS (N) 16,752   6,902   1,907   1,501   569  

Web of Science (%) 63%   19%   19%   49%   48%  

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Biological sciences (Bio)

Agriculture, forestry and

fisheries (Agri)

Animal and dairy science (Ani)

Veterinary science (Vet)

Other agricultural science

Distribution of the scholarly publications in the Czech Republic among fields within its major area.

Example: Bio- and agri-sciences. (2008-2013)

Web of Science R&D IS

Page 70: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

68 R&D Evaluation Methodology and Funding Principles

Indicator A5. Mean and median number of authors and addresses per WoS publication. Mean number of authors and addresses in the biological and agricultural sciences.

Data Source Bio Agri Ani Vet Other

Authors 6.07   4.53   4.53   5.29   2.85  

Addresses 3.78   2.40   2.04   2.48   1.38  

Indicator A6. Percentage Web of Science publications in the most cited 10 per cent , 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports. Biological and agricultural sciences

Indicator A6 is supplied by a listing of WoS journals in each field, ranked by the number of articles from the Czech Republic, specifying each journal taking more than 5 per cent of the total volume of articles in the field.

Indicator A7. Mean Field Normalized Citation Impact, compared to the world average (=1,00) and the EU28 average (=1,00). Biological and agricultural sciences, comparison with the world average.

2008 2009 2010 2011 2012

Agriculture,  forestry,  and  fisheries   1.01   1.03   1.25   1.62   1.72  

Animal  and  dairy  science   0.79   0.84   0.97   0.87   1.18  

Biological  sciences   1.06   1.01   1.09   1.29   1.70  

Other  agricultural  sciences   0.67   0.63   0.51   0.79   1.16  

Veterinary  science   1.20   1.05   1.08   1.34   1.67  

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100%  

Bio   Agri   Ani   Vet   Other  

Publishing  Pro-ile.  %  of  WoS  publications  in  most  cited  journals  in  the  -ield,  based  on  2014  JIF.  Example:  Bio-­‐  and  agri-­‐sciences  

Most  cited  10  %   Most  cited  25  %   Most  cited  50  %   Least  cited  50  %  

Page 71: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 69

69

Indicator A8. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

Percentile of publications

Agriculture,  forestry,  

and  fisheries  

Animal  and  

dairy  science  

Biological  sciences  

Other  agricultural  

sciences  Veterinary  

science  

Most cited 10 % 14%   5%   9%   4%   11%  

Most cited 25 % 17%   17%   13%   8%   17%  

Most cited 50 % 9%   10%   16%   11%   9%  

Least cited 50 % 59%   67%   63%   77%   62%  

Indicator A9. Percentage Web of Science publications exhibiting international collaboration in the addresses. Biological and agricultural sciences.

Indicator A10. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared to EU28 in the field.

Agriculture, forestry and Fisheries Articles Citation impact

(compared to EU 28)

Slovakia   90   2.33  

France   89   3.49  

United  Kingdom   74   3.08  

USA   63   3.47  

Austria   60   2.37  

0.0  %  

10.0  %  

20.0  %  

30.0  %  

40.0  %  

50.0  %  

60.0  %  

Bio   Agri   Ani   Vet   Other  

Percentage  WoS  publications  exhibiting  international  and  national  collaboration.    

Example:  Bio-­‐  and  agri-­‐sciences.  (2008-­‐2013)  

Page 72: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

70 R&D Evaluation Methodology and Funding Principles

5.5 Social sciences

Indicator A1. Relative size of the fields’ major area in the publication data from WoS and R&D IS.

0% 5%

10% 15% 20% 25% 30% 35%

Physical Sciences

Engineering and technology

Medical and Health sciences

Biological and Agricultural

Sciences

Social sciences Humanities

Distribution of the scholarly publications in the Czech Republic among major disciplinary Areas

(2008-2013)

WoS R&D IS

Page 73: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results
Page 74: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 72

Indicator A2. Relative size of the field within its major area in the publication data from WoS and R&D IS. Social sciences.

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Psychology Economics and business

Educational sciences

Sociology Law Political science Social and economic geography

Media and communication

Other social sciences

Distribution of the scholarly publications in the Czech Republic among fields within its major area. Example: Social sciences (2008-2013)

Web of Science R&D IS

Page 75: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results
Page 76: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 74

Indicator A3. Number of publications in scholarly R&DIS categories. Social sciences. Category Psych Econ Educ Socio

Articles in peer-reviewed journals (J) 1174   8644   3495   1317  

Monographs (B) 198   1198   740   255  

Book chapters (C) 482   1492   2484   773  

Proceedings papers (D) 836   10287   5081   629  

Category Law Politics Geo Media Other

Articles in peer-reviewed journals (J) 3724   1515   228   228   690  

Monographs (B) 572   390   27   29   58  

Book chapters (C) 1262   911   68   61   186  

Proceedings papers (D) 2612   294   282   136   451  

Indicator A4. Number and percentage of Web of Science publications among all peer-reviewed R&DIS publications. Social sciences.

Data Source Psych Econ Educ Socio

Web of Science (N) 604   1,204   139   490  

R&D IS (N) 2,690   21,621   11,800   2,974  

Web of Science (%) 22%   6%   1%   16%  

Data Source Law Politics Geography Media Other

Web of Science (N) 13   222   336   48   67  

R&D IS (N) 8,170   3,110   605   454   1,385  

Web of Science (%) 0%   7%   55%   11%   5%  

Indicator A5. Mean and median number of authors and addresses per WoS publication. Mean number of authors and addresses in the social sciences.

Data Source Psych Econ Educ Socio

Authors 3.63   1.78   2.05   1.92  

Addresses 2.46   1.26   1.15   1.49  

Data Source Law Politics Geography Media Other

Authors 1.32   1.71   2.27   2.05   1.76  

Addresses 1.29   1.48   1.54   1.24   1.27  

Page 77: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 75

Indicator A6. Percentage of Web of Science publications in the most cited 10 percent , 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports. Social sciences.

Indicator A6 is supplied by a listing of WoS journals in each field, ranked by the number of articles from the Czech Republic, specifying each journal taking more than 5 per cent of the total volume of articles in the field.

Indicator A7. Mean Field Normalized Citation Impact, compared to the world average (=1,00) and the EU28 average (=1,00). Social sciences, comparison with the world average. 2008 2009 2010 2011 2012

Economics  and  business   0.29   0.28   0.22   0.36   0.62  

Educational  sciences   0.21   0.07   0.20   2.09   0.62  

Media  and  communication   0.47   0.12   0.05   0.07   0.98  

Political  Science   0.74   0.63   0.78   1.72   1.81  

Psychology   0.49   0.18   0.27   0.43   0.66  

Social  and  economic  geography   0.68   0.35   1.23   0.92   1.11  

Sociology   0.25   0.20   0.50   1.05   0.48  

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100%  

Psych   Econ   Educ   Socio   Law   Politics   Geography   Media  

Publishing  Pro8ile.  %  of  WoS  publications  in  most  cited  journals  in  the  8ield,  based  on  2014  JIF.  Example:  Social  sciences  

Most  cited  10  %   Most  cited  25  %   Most  cited  50  %   Least  cited  50  %  

Page 78: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 76

Indicator A8. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

Percentile of publications

Economics  and  business  

Educational  sciences  

Media  and  communication  

Political  Science  

Most cited 10 % 3%   4%   4%   13%  

Most cited 25 % 6%   9%   2%   43%  

Most cited 50 % 1%   0%   0%   0%  

Least cited 50 % 90%   87%   94%   44%  

Percentile of publications Psychology   Social  and  economic  geography   Sociology  

Most cited 10 % 3%   6%   6%  

Most cited 25 % 6%   14%   4%  

Most cited 50 % 3%   3%   2%  

Least cited 50 % 88%   77%   88%  

Indicator A9. Percentage Web of Science publications exhibiting international and collaboration in the addresses. Social sciences.

0.0  %  

2.0  %  

4.0  %  

6.0  %  

8.0  %  

10.0  %  

12.0  %  

14.0  %  

16.0  %  

18.0  %  

20.0  %  

Econ   Educ   Media   Politics   Psych   Geography   Socio  

Percentage  WoS  publications  exhibiting  international  collaboration.    Example:  Social  sciences.  (2008-­‐2013)  

Page 79: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 77

Indicator A10. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared globally in the field.

Sociology Articles Citation impact

(compared to world)

USA   29   2.37  

United  Kingdom   24   3.32  

France   19   3.48  

Italy   9   4.36  

Slovakia   9   0.80  

5.6 Humanities This group of indicators are all meant to give the panel an overview of the characteristics of the field at country level, thereby providing a context for interpreting the indicators for a specific Research Unit’s in a specific field.

The same indicators will appear at the level of Research Units in categories B-F below. More details about the indicators and their interpretation will be given there.

Indicator A1. Relative size of the fields’ major area in the publication data from WoS and R&D IS.

0% 5%

10% 15% 20% 25% 30% 35%

Physical Sciences

Engineering and technology

Medical and Health sciences

Biological and Agricultural

Sciences

Social sciences Humanities

Distribution of the scholarly publications in the Czech Republic among major disciplinary areas

(2008-2013)

WoS R&D IS

Page 80: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 78

Indicator A2. Relative size of the field within its major area in the publication data from WoS and R&D IS. Humanities.

Indicator A3. Number of publications in scholarly R&DIS categories. Humanities. Category H&A L&L PER Arts

Articles in peer-reviewed journals (J) 6540   4980   2552   2809  

Monographs (B) 1587   997   724   924  

Book chapters (C) 4565   4118   2090   2569  

Proceedings papers (D) 2318   3204   691   1468  

Indicator A4. Number of and percentage Web of Science publications among all peer-reviewed R&DIS publications. Humanities.

Data Source H&A L&L PER Arts Other

Web of Science (N) 191   317   662   200   52  

R&D IS (N) 15,010   13,299   6,057   7,770   0  

Web of Science (%) 1%   2%   11%   3%   -­‐  

Indicator A5. Mean and median number of authors and addresses per WoS publication. Mean number of authors and addresses in the humanities.

Data Source H&A L&L PER Arts Other

Authors 2.17   1.46   1.05   1.55   1.17  

Addresses 1.71   1.17   1.16   1.19   1.18  

INDICATORS A6-A10: INSUFFICIENT WoS COVERAGE OF HUMANITIES

• Indicator A6. Percentage Web of Science publications in the most cited 10 percent , 25 per cent, and 50 per cent of the journals in the field (counted from the top by

0% 5%

10% 15% 20% 25% 30% 35% 40%

History and archaeology (H&A)

Languages and literature (L&L)

Philosophy, ethics and religion (PER)

Art (arts, history of arts, performing

arts, music)

Other humanities

Distribution of the scholarly publications in the Czech Republic among fields within its major area.

Example: Humanities (2008-2013)

Web of Science R&D IS

Page 81: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 79

the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports.

Indicator A6 is supplied by a listing of WoS journals in each field, ranked by the number of articles from the Czech Republic, specifying each journal taking more than 5 per cent of the total volume of articles in the field.

Indicator A7. Mean Field Normalized Citation Impact, compared to the world average (=1,00) and the EU28 average (=1,00).

• Indicator A8. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

• Indicator A9. Percentage of Web of Science publications exhibiting international and national collaboration in the addresses.

• Indicator A10. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared to EU28 in the field.

Page 82: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 80

6. Examples of bibliometric reports at the RU level

In this chapter we reproduce the bibliometric reports for the Research Units (RUs) of the Evaluated Units that gave their consent for publication.

This regards RUs in:

• The Institute of Analytical chemistry ASCR (Section 6.1)

• The Institute of Computer Science of the AS CR (Section 6.2)

• The Czech University of Life Science, Faculty of Agrobiology (Section 6.3)

• The Masaryk University, Faculty of Social Studies (Section 6.4)

6.1 Physical sciences – Environmental sciences: Institute of Analytical Chemistry ASCR

6.1.1 B: Characteristics of the Research Unit (RU) Indicator B1 and B2 are given for all included Research Units in all fields in the evaluation methodology, both within fields and within Evaluated Units. This will give an overview of the relative contributions of each Evaluated Unit to each field in the Czech Republic. In the Small Pilot Evaluation, the two indicators are given for each selected RU with the field characteristics (Indicator group A above) as context.

Interpretation: These indicators identify the relative size – measured in publication output (research activity) of the Research Unit within its field in the country and within the Evaluated Unit it belongs to. Coverage in Web of Science is field dependent. The bibliometric report will advise how to use, or not to use, indicators based on Web of Science in each specific field.

Indicator B1. Percentage shares of all Czech publications (WoS and R&D IS)

Source   Area   Field   Count   %  CZ  

WoS  

Physical  Sciences  

Mathematics   1   0.02%  

Physical  sciences  and  astronomy   1   0.01%  

Chemical  sciences   20   0.19%  

Earth  and  related  environmental  sciences   8   0.18%  

Engineering  and  technology  

Materials  engineering   4   0.07%  

Nano-­‐technology   3   0.24%  

Other  engineering  and  technologies   4   0.11%  

Medical  and  Health  sciences   Basic  medical  research   1   0.02%  

 Biological  and  Agricultural  Sciences   Biological  sciences   2   0.02%  

R&D  IS  Physical  Sciences  

Chemical  sciences   75   0.63%  

Earth  and  related  environmental  sciences   5   0.04%  

Biological  and   Biological  sciences   1   0.01%  

Page 83: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 81

Source   Area   Field   Count   %  CZ  

Agricultural  Sciences  

The table above shows the field structure of the Research Unit based on the publications registered in the WoS and the R&D IS. While the researchers composing the Research Unit conduct their research primarily in the field of evaluation, their publications may appear in journals registered against other fields.

Indicator B2. Percentage shares of all publications from the Evaluated Unit, all fields taken together

6.1.2 C: Publishing Profile (RU) Interpretation: The indicators in this group give an overview of the publishing profile of the Research Unit. This profile should be contextualized by other material provided for the evaluation panel, such as:

• The information on the level of the field given in indicator group A

• Field and type of research

• Mission, size and resources of the Evaluated Unit, as given in the self-evaluation and in other statistical information provided for the evaluation

WoS  10.8%  

R&D  IS  20.3%  

Percentage  shares  of  all  publications  from  the  EvU,  all  8ields  

Page 84: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 82

Indicator C1. Number and shares of publications in scholarly R&D IS categories

Indicator C2. Number and percentage of Web of Science publications among all scholarly R&DIS publications

Data Source Indicator

Web of Science (N) 33

R&D IS (N) 128

Web of Science (%) 25.8%

Indicator C3. Mean and median number of authors and institutional affiliations (published addresses) per WoS publication

Data Source Mean Median

Authors 4.1 4

Addresses 2.0 2

Indicator C4. Percentage Web of Science publications in the most cited 10 per cent, 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports.

Articles  in  peer-­‐reviewed  

journals  (J);  30;  24%  

Monographs  (B);  0;  0%  

Book  chapters  (C);  4;  3%  

Proceedings  papers  (D);  94;  

73%  

Number  and  shares  of  publications  in  scholarly  R&D  IS  categories  

Page 85: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 83

Indicator C5. Number and percentage of articles in the field per journal in Web of Science

Journal Articles Percentage

CECE  2012:  9TH  INTERNATIONAL  INTERDISCIPLINARY  MEETING  ON  BIOANALYSIS   6   19.4  %  

CHEMICKE  LISTY   4   12.9  %  

ANALYTICA  CHIMICA  ACTA   3   9.7  %  

JOURNAL  OF  ANALYTICAL  ATOMIC  SPECTROMETRY   2   6.5  %  

NANOCON  2011   2   6.5  %  

ATMOSPHERIC  ENVIRONMENT   2   6.5  %  

SPECTROCHIMICA  ACTA  PART  B-­‐ATOMIC  SPECTROSCOPY   2   6.5  %  

TALANTA   2   6.5  %  

INTERNATIONAL   JOURNAL   OF   ENVIRONMENTAL   ANALYTICAL  CHEMISTRY   1   3.2  %  

NANOCON  2012,  4TH  INTERNATIONAL  CONFERENCE   1   3.2  %  

CHEMICAL  SPECIATION  AND  BIOAVAILABILITY   1   3.2  %  

BIOGEOSCIENCES   1   3.2  %  

ANALYST   1   3.2  %  

11TH   INTERNATIONAL   CONFERENCE   OF   NUMERICAL   ANALYSIS  AND  APPLIED  MATHEMATICS  2013,  PTS  1  AND  2  (ICNAAM  2013)   1   3.2  %  

FRESENIUS  ENVIRONMENTAL  BULLETIN   1   3.2  %  

ENVIRONMENTAL  POLLUTION   1   3.2  %  

28.6  %  

28.6  %  

42.9  %  

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

Publishing  Pro8ile.  Share  of  WoS  publications  in  most  cited  journals  in  the  8ield,  based  on  2014  JIF.  Example:  Physical  Sciences  

Least  cited  50  %  

Most  cited  50  %  

Most  cited  25  %  

Most  cited  10  %  

Page 86: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 84

6.1.3 D: Citation Impact (RU) Interpretation: Citation indicators may inform about the international influence, impact, or usefulness of the research, as seen in the frequency of received citations per publication. Note that citation distributions are highly skewed. One or a few highly cited articles may influence the averages considerably and cause large variations from year to year as well. The two indicators presented here should therefore be seen in relation to each other. One of them identifies the average citation rate, while the other records the presence of articles among the most highly cited articles in the world and EU.

The relevance and validity of the citation indicators have to be related to the specific field and type of research. Especially relevant is the Web of Science coverage of the field. Citation indicators will not be presented in fields with low Web of Science coverage.

Indicator D1. Mean Field Normalized Citation Impact for the Research Unit, compared to the world average (=1,00) and the EU28 average (=1,00).

2009 2010 2011 2012

Compared to world average 0.25   0.73   1.07   0.29  

Compared to EU 28 average 0.21   0.63   0.90   0.24  

Indicator D2. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

6.1.4 E: Collaboration (RU) Interpretation: These indicators provide information about the relations to main partners in international and national collaboration in the field, as seen in the addresses of co-authored publications. Caution has to be taken with regard to publishing practices in the field, see Indicator B3: Mean and median number of authors and addresses per WoS publication. Generally, the relevance and validity of

0   0  2   2  3   2  

20   21  

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

World   EU  

Publishing  Pro8ile.  Publications  among  the  most  cited  (world,  EU28)  

Least  cited  50  %  

Most  cited  50  %  

Most  cited  25  %  

Most  cited  10  %  

Page 87: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 85

the collaboration indicators will have to be related to the publication practices in the specific field of research and their Web of Science coverage. The collaboration indicators will not be presented in fields with low Web of Science coverage. (The R&D IS needs further development to make the basis for similar indicators.)

Indicator E1. Percentage Web of Science publications exhibiting international and national collaboration in the addresses 2009 2010 2011 2012 2013

International 25.0%   0.0%   28.6%   6.7%   0.0%  

National 75.0%   100.0%   28.6%   40.0%   50.0%  

Indicator E2. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared globally in the field.

Share of international co-publications

Citation impact (compared to world)

Belgium   20%   1.71  

France   20%   1.71  

Italy   20%   1.71  

Netherlands   20%   1.71  

United  Kingdom   20%   0.60  

6.1.5 F: Scholarly outputs and non-traditional scholarly outputs – statistical data (RU) Interpretation: the statistical data presented below give the panels a view on the trends in scholarly and non-traditional scholarly outputs during the evaluation period and set the publication profile of the Research Units in the context of the field in the CR. It also provides information on the potential reach of the journals in which the articles are published (national versus international), based upon the databases in which the journals are registered.

Indicator F1: Publication of scholarly outputs and non-traditional scholarly outputs by the RU and their distribution over the years (raw data)

2008 2009 2010 2011 2012 2013 Scholarly outputs Article in a periodical (J) 6   4   4   4   7   5  Monographs and books (B)  

         Book chapter (C.)   4          Conference proceedings / Article in

proceedings (D)   14   14   22   26   18  Non-traditional research outputs

           Results used by the funding provider, i.e. into legislation or norm, into non-legislative or strategic documents (H)

           Research report containing classified information (V)

           Certified methodologies, art conservation methodologies, specialized map works (N)          2              

Page 88: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 86

Indicator F2: Shares of scholarly outputs and non-traditional scholarly outputs in the R&D IS categories – Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR)

RU share of Field total

Scholarly outputs Article in a periodical (J) 5,755   0.52%  Monographs and books (B) 368   0.00%  Book chapter (C) 1,112   0.36%  Conference proceedings / Article in proceedings (D) 6,380   1.47%  Non-traditional scholarly outputs    Results used by the funding provider, i.e. projected into legislation or norm, projected into non-legislative or strategic documents (H) 60   0.00%  Research report containing classified information (V) 4   0.00%  Certified methodologies, art conservation methodologies, specialized map works (N) 3,624   0.06%  

Indicator F3: Number and shares of articles published in typologies of journals (national versus international reach) – RU (2008-2013)

Publication channels Number Share of the total

Article in a periodical registered in the Web of Science (Jimp) 24   88.9% Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 1   3.7% Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 0   0.0% Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 2   7.4% TOTAL 27 100%

Indicator F4: Shares of articles published in typologies of journals (national versus international reach) - Research Unit versus Field total in the CR (2008-2013)

Publication channels Number in the Field (CR)

RU share of Field total

Article in a periodical registered in the Web of Science (Jimp) 3,488   0.69%  Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 421   0.24%  Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 3   0.00%  Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 1,253   0.16%  

6.1.6 G: IPR-related outputs (RU) Interpretation: also in this case, the panels are given a view on the trends in patents awarded during the evaluation period. The data set the information for the Research Units in the context of the field in the CR and give a view on the importance of the patents in terms of geographical reach.

Page 89: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 87

Indicator G1: Patents and other forms of IP awarded to the RU and their distribution over the years (raw data)

2008 2009 2010 2011 2012 2013 Patent (P) 0 0 0 0 0 0 Plant variety (Zodru) 0 0 0 0 0 0 Animal breed (Zplem) 0 0 0 0 0 0

Indicator G2: Shares of IPR-related outputs in the R&D IS categories - Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR) RU share of Field total

Patent (P) 43 0.0% Plant variety (Zodru) 0 - Animal breed (Zplem) 0 - TOTAL 43 0.0%

Indicator G3: Number of patents in the patent offices in different countries (2008-2013)

Number for the RU

Number in the Field (CR)

Patents in the Czech Industrial Property Office 0 40  

Patents in the European Patent Office (EPO) 0 1  

Patents in the US/Japan Patents Offices 0 1  

Patents in other international patent offices 0 1  

Total Patents 0 43

6.2 Engineering & Technology – Computer & information sciences: Institute of Computer Science of the AS CR

6.2.1 B: Characteristics of the Research Unit (RU) Indicator B1 and B2 are given for all included Research Units in all fields in the evaluation methodology, both within fields and within Evaluated Units. This will give an overview of the relative contributions of each Evaluated Unit to each field in the Czech Republic. In the Small Pilot Evaluation, the two indicators are given for each selected RU with the field characteristics (Indicator group A above) as context.

Interpretation: These indicators identify the relative size – measured in publication output (research activity) of the Research Unit within its field in the country and within the Evaluated Unit it belongs to. Coverage in Web of Science is field dependent. The bibliometric report will advise how to use, or not to use, indicators based on Web of Science in each specific field.

Page 90: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 88

Indicator B1. Percentage shares of all Czech publications (WoS and R&D IS)

Source   Area   Field   Count   %  CZ  

WoS  

Physical  Sciences   Mathematics   21   0.45%  

Physical  Sciences   Physical  sciences  and  astronomy   5   0.04%  

Physical  Sciences   Chemical  sciences   3   0.03%  

Physical  Sciences  Earth  and  related  environmental  sciences   8   0.18%  

Engineering  and  technology  Electrical  engineering,  electronic  engineering,  information  engineering   26   0.41%  

Engineering  and  technology   Computer  and  information  sciences   105   2.11%  

Engineering  and  technology   Mechanical  engineering   1   0.03%  

Engineering  and  technology   Chemical  engineering   2   0.19%  

Engineering  and  technology   Medical  engineering   2   0.18%  

Engineering  and  technology   Environmental  engineering   1   0.04%  

Engineering  and  technology   Nano-­‐technology   1   0.08%  

Engineering  and  technology   Other  engineering  and  technologies   1   0.03%  

Medical  and  Health  sciences   Basic  medical  research   9   0.16%  

Medical  and  Health  sciences   Clinical  medicine   6   0.07%  

Medical  and  Health  sciences   Health  sciences   9   0.48%  

 Biological  and  Agricultural  Sciences   Biological  sciences   10   0.09%  

Social  sciences   Psychology   1   0.13%  

Social  sciences   Economics  and  business   4   0.11%  

Social  sciences   Educational  sciences   1   0.10%  

Social  sciences   Sociology   2   0.20%  

Social  sciences   Social  and  economic  geography   1   0.14%  

R&D  IS  

Physical  Sciences   Mathematics   132   1.45%  

Engineering  and  Technology  Electrical  engineering,  electronic  engineering,  information  engineering   23   0.14%  

Engineering  and  Technology   Computer  and  information  sciences   270   2.65%  

The tables above show the field structure of the Research Unit based on the publications registered in the WoS and the R&D IS. While the researchers composing the Research Unit conduct their research primarily in the field of evaluation, their publications may appear in journals registered against other fields.

Indicator B2. Percentage shares of all publications from the Evaluated Unit, all fields taken together

Page 91: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 89

6.2.2 C: Publishing Profile (RU) Interpretation: The indicators in this group give an overview of the publishing profile of the Research Unit. This profile should be contextualized by other material provided for the evaluation panel, such as:

• The information on the level of the field given in indicator group A

• Field and type of research

• Mission, size and resources of the Evaluated Unit, as given in the self-evaluation and in other statistical information provided for the evaluation

Indicator C1. Number and shares of publications in scholarly R&D IS categories

Page 92: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 90

Indicator C2. Number of and percentage Web of Science publications among all scholarly R&DIS publications

Data Source Indicator

Web of Science (N) 156

R&D IS (N) 425

Web of Science (%) 36.7%

Indicator C3. Mean and median number of authors and institutional affiliations (published addresses) per WoS publication

Data Source Mean Median

Authors 2.9 3

Addresses 1.8 1

Indicator C4. Percentage Web of Science publications in the most cited 10 per cent, 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports.

Indicator C5. Number and percentage of articles in the field per journal in Web of Science Journal Articles Percentage

NEURAL NETWORK WORLD 12 7.7 %

NEURAL NETWORKS 4 2.6 %

CHAOS 3 1.9 %

NEURAL COMPUTATION 2 1.3 %

32ND ANNUAL IEEE SOFTWARE ENGINEERING WORKSHOP, PROCEEDINGS 2 1.3 %

NONLINEAR PROCESSES IN GEOPHYSICS 2 1.3 %

Page 93: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 91

Journal Articles Percentage

PROCEEDING OF THE THIRD INTERNATIONAL CONFERENCE ON INTELLIGENT HUMAN COMPUTER INTERACTION (IHCI 2011) 2 1.3 %

PROCEEDINGS OF THE 2009 FIFTH INTERNATIONAL CONFERENCE ON NEXT GENERATION WEB SERVICES PRACTICES, NWESP 2009 2 1.3 %

ADVANCES IN INTELLIGENT WEB MASTERING 3 2 1.3 %

SOFTWARE ENGINEERING, ARTIFICIAL INTELLIGENCE, NETWORKING AND PARALLEL/DISTRIBUTED COMPUTING 2 1.3 %

ADAPTIVE AND NATURAL COMPUTING ALGORITHMS, PT I 2 1.3 %

METHODS OF INFORMATION IN MEDICINE 2 1.3 %

NEUROCOMPUTING 2 1.3 %

2012 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC) 2 1.3 %

2009 IEEE/WIC/ACM INTERNATIONAL JOINT CONFERENCES ON WEB INTELLIGENCE (WI) AND INTELLIGENT AGENT TECHNOLOGIES (IAT), VOL 3 2 1.3 %

SOFSEM 2012: THEORY AND PRACTICE OF COMPUTER SCIENCE 2 1.3 %

11TH INTERNATIONAL CONFERENCE OF NUMERICAL ANALYSIS AND APPLIED MATHEMATICS 2013, PTS 1 AND 2 (ICNAAM 2013) 2 1.3 %

JOURNAL OF COMPLEXITY 1 0.6 %

NANO-NET 1 0.6 %

MODELING DECISIONS FOR ARTIFICIAL INTELLIGENCE, PROCEEDINGS 1 0.6 %

TIM 2012 PHYSICS CONFERENCE 1 0.6 %

METEOROLOGY AND ATMOSPHERIC PHYSICS 1 0.6 %

MEDICAL INFORMATICS IN A UNITED AND HEALTHY EUROPE 1 0.6 %

MATHEMATICAL FOUNDATIONS OF COMPUTER SCIENCE 2011 1 0.6 %

LANDSLIDES 1 0.6 %

LANDSCAPE AND URBAN PLANNING 1 0.6 %

JOURNAL OF STATISTICAL PLANNING AND INFERENCE 1 0.6 %

JOURNAL OF MATHEMATICAL IMAGING AND VISION 1 0.6 %

JOURNAL OF COMPUTER AND SYSTEM SCIENCES 1 0.6 %

NEURAL INFORMATION PROCESSING, PT 2, PROCEEDINGS 1 0.6 %

JOURNAL OF CLINICAL EPIDEMIOLOGY 1 0.6 %

JOURNAL OF CLASSIFICATION 1 0.6 %

INTERNATIONAL JOURNAL OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE 1 0.6 %

INTERNATIONAL JOURNAL OF APPROXIMATE REASONING 1 0.6 %

INTERNATIONAL DAYS OF STATISTICS AND ECONOMICS 1 0.6 %

INTELLIGENT DATA ENGINEERING AND AUTOMATED LEARNING - IDEAL 2010 1 0.6 %

INTELLIGENT COMPUTING THEORIES AND APPLICATIONS, ICIC 2012 1 0.6 %

INFORMATION TECHNOLOGY IN BIO- AND MEDICAL INFORMATICS 1 0.6 %

INFORMATION AND SOFTWARE TECHNOLOGY 1 0.6 %

JOURNAL OF FORENSIC SCIENCES 1 0.6 %

PHYSIOLOGICAL MEASUREMENT 1 0.6 %

SALUD I CIENCIA 1 0.6 %

RESEARCH INTO PRACTICE-REALITY AND GAPS, PROCEEDINGS 1 0.6 %

RECENT ADVANCES IN INTELLIGENT ENGINEERING SYSTEMS 1 0.6 %

SIMULATED EVOLUTION AND LEARNING 1 0.6 %

Page 94: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 92

Journal Articles Percentage

SOFSEM 2010: THEORY AND PRACTICE OF COMPUTER SCIENCE, PROCEEDINGS 1 0.6 %

QUASSOSS 09: 1ST INTERNATIONAL WORKSHOP ON THE QUALITY OF SERVICE-ORIENTED SOFTWARE SYSTEM 1 0.6 %

IET SOFTWARE 1 0.6 %

PROCEEDINGS OF THE 8TH IEEE INTERNATIONAL CONFERENCE ON COGNITIVE INFORMATICS 1 0.6 %

PROCEEDINGS OF THE 2013 IEEE SYMPOSIUM ON COMPUTATIONAL INTELLIGENCE FOR HUMAN-LIKE INTELLIGENCE (CIHLI) 1 0.6 %

PROCEEDINGS OF THE FOURTEENTH INTERNATIONAL CONFERENCE ON GENETIC AND EVOLUTIONARY COMPUTATION CONFERENCE 1 0.6 %

NATURAL COMPUTING 1 0.6 %

PRAGUE ECONOMIC PAPERS 1 0.6 %

SOFTWARE ENGINEERING RESEARCH, MANAGEMENT AND APPLICATIONS 2009 1 0.6 %

PHOTOMEDICINE AND LASER SURGERY 1 0.6 %

PERSPECTIVES IN PLANT ECOLOGY EVOLUTION AND SYSTEMATICS 1 0.6 %

NUKLEONIKA 1 0.6 %

SOFTWARE-PRACTICE & EXPERIENCE 1 0.6 %

NEW EDUCATIONAL REVIEW 1 0.6 %

NEURORADIOLOGY 1 0.6 %

TEACHING FORMAL METHODS, PROCEEDINGS 1 0.6 %

THEORETICAL COMPUTER SCIENCE 1 0.6 %

SEAMLESS CARE - SAFE CARE: THE CHALLENGES OF INTEROPERABILITY AND PATIENT SAFETY IN HEALTH CARE 1 0.6 %

THEORY OF COMPUTING SYSTEMS 1 0.6 %

SOFTWARE ENGINEERING, ARTIFICIAL INTELLIGENCE, NETWORKING AND PARALLEL-DISTRIBUTED COMPUTING 2010 1 0.6 %

ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE 1 0.6 %

ARTIFICIAL NEURAL NETWORKS - ICANN 2009, PT I 1 0.6 %

ARTIFICIAL INTELLIGENCE AND SOFT COMPUTING, PT II 1 0.6 %

APPLIED MATHEMATICS AND COMPUTATION 1 0.6 %

AMPHIBIA-REPTILIA 1 0.6 %

AMERICAN JOURNAL OF HUMAN BIOLOGY 1 0.6 %

ADVANCES IN NEURO-INFORMATION PROCESSING, PT II 1 0.6 %

ADVANCES IN NEURAL NETWORKS - ISNN 2011, PT III 1 0.6 %

ADVANCES IN NEURAL NETWORKS - ISNN 2011, PT I 1 0.6 %

ADVANCES IN NEURAL NETWORKS - ISNN 2010, PT 1, PROCEEDINGS 1 0.6 %

ADVANCES IN INTELLIGENT WEB MASTERING-2, PROCEEDINGS 1 0.6 %

ADVANCES IN INFORMATION TECHNOLOGY 1 0.6 %

ADVANCED INTELLIGENT COMPUTING THEORIES AND APPLICATIONS: WITH ASPECTS OF ARTIFICIAL INTELLIGENCE 1 0.6 %

BELIEF FUNCTIONS: THEORY AND APPLICATIONS 1 0.6 %

ADAPTIVE AND NATURAL COMPUTING ALGORITHMS 1 0.6 %

2012 FOURTH INTERNATIONAL CONFERENCE ON COMPUTATIONAL ASPECTS OF SOCIAL NETWORKS (CASON) 1 0.6 %

6TH INTERNATIONAL DAYS OF STATISTICS AND ECONOMICS 1 0.6 %

2013 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC) 1 0.6 %

2012 IEEE INTERNATIONAL CONFERENCE ON FUZZY SYSTEMS (FUZZ-IEEE) 1 0.6 %

Page 95: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 93

Journal Articles Percentage

2012 IEEE 24TH INTERNATIONAL CONFERENCE ON TOOLS WITH ARTIFICIAL INTELLIGENCE (ICTAI 2012), VOL 1 1 0.6 %

2012 38TH EUROMICRO CONFERENCE ON SOFTWARE ENGINEERING AND ADVANCED APPLICATIONS (SEAA) 1 0.6 %

RADIATION PROTECTION DOSIMETRY 1 0.6 %

2011 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC) 1 0.6 %

2010 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC) 1 0.6 %

2009 INTERNATIONAL CONFERENCE ON INTELLIGENT ENGINEERING SYSTEMS 1 0.6 %

2009 INTERNATIONAL CONFERENCE ON COMPUTATIONAL ASPECTS OF SOCIAL NETWORKS, PROCEEDINGS 1 0.6 %

2009 IEEE INTERNATIONAL CONFERENCE ON WEB SERVICES, VOLS 1 AND 2 1 0.6 %

2009 35TH EUROMICRO CONFERENCE ON SOFTWARE ENGINEERING AND ADVANCED APPLICATIONS, PROCEEDINGS 1 0.6 %

IEEE TRANSACTIONS ON NEURAL NETWORKS 1 0.6 %

ADVANCED INTELLIGENT COMPUTING 1 0.6 %

ENTROPY 1 0.6 %

2011 IEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN, AND CYBERNETICS (SMC) 1 0.6 %

BERNOULLI 1 0.6 %

ICTAI: 2009 21ST INTERNATIONAL CONFERENCE ON TOOLS WITH ARTIFICIAL INTELLIGENCE 1 0.6 %

ICMS2010: PROCEEDINGS OF THE THIRD INTERNATIONAL CONFERENCE ON MODELLING AND SIMULATION ICMS2010, VOL 5 1 0.6 %

ICAART 2009: PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON AGENTS AND ARTIFICIAL INTELLIGENCE 1 0.6 %

HOMO-JOURNAL OF COMPARATIVE HUMAN BIOLOGY 1 0.6 %

FORMAL MODELING AND ANALYSIS OF TIMED SYSTEMS 1 0.6 %

FLEXIBLE QUERY ANSWERING SYSTEMS: 8TH INTERNATIONAL CONFERENCE, FQAS 2009 1 0.6 %

E-TECHNOLOGIES AND NETWORKS FOR DEVELOPMENT 1 0.6 %

IEEE TRANSACTIONS ON INFORMATION THEORY 1 0.6 %

ECONOMICS & HUMAN BIOLOGY 1 0.6 %

DIGITAL INFORMATION AND COMMUNICATION TECHNOLOGY AND ITS APPLICATIONS, PT II 1 0.6 %

COMPUTERS & CHEMICAL ENGINEERING 1 0.6 %

COMPUTATIONAL STATISTICS 1 0.6 %

CATALYSIS TODAY 1 0.6 %

BIOCYBERNETICS AND BIOMEDICAL ENGINEERING 1 0.6 %

BMC CANCER 1 0.6 %

ENVIRONMETRICS 1 0.6 %

CANCER INVESTIGATION 1 0.6 %

COMPUTATIONAL SCIENCE AND ITS APPLICATIONS - ICCSA 2010, PT 4, PROCEEDINGS 1 0.6 %

CHEMCATCHEM 1 0.6 %

CHEMOSPHERE 1 0.6 %

COMMUNICATIONS IN STATISTICS-SIMULATION AND COMPUTATION 1 0.6 %

COMMUNICATIONS IN STATISTICS-THEORY AND METHODS 1 0.6 %

Page 96: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 94

Journal Articles Percentage

COMPONENT-BASED SOFTWARE ENGINEERING, PROCEEDINGS 1 0.6 %

COMPUTATIONAL INTELLIGENCE 1 0.6 %

COGNITIVE COMPUTATION 1 0.6 %

BMC MEDICAL INFORMATICS AND DECISION MAKING 1 0.6 %

COMPUTATIONAL FORENSICS 1 0.6 %

6.2.3 D: Citation Impact (RU) Interpretation: Citation indicators may inform about the international influence, impact, or usefulness of the research, as seen in the frequency of received citations per publication. Note that citation distributions are highly skewed. One or a few highly cited articles may influence the averages considerably and cause large variations from year to year as well. The two indicators presented here should therefore be seen in relation to each other. One of them identifies the average citation rate, while the other records the presence of articles among the most highly cited articles in the world and EU.

The relevance and validity of the citation indicators have to be related to the specific field and type of research. Especially relevant is the Web of Science coverage of the field. Citation indicators will not be presented in fields with low Web of Science coverage.

Indicator D1. Mean Field Normalized Citation Impact for the Research Unit, compared to the world average (=1,00) and the EU28 average (=1,00).

2009 2010 2011 2012

Compared to world average 0.36 0.55 1.15 1.13

Compared to EU 28 average 0.25 0.47 1.04 0.99

Indicator D2. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

Page 97: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 95

6.2.4 E: Collaboration (RU) Interpretation: These indicators provide information about the relations to main partners in international and national collaboration in the field, as seen in the addresses of co-authored publications. Caution has to be taken with regard to publishing practices in the field, see Indicator B3: Mean and median number of authors and addresses per WoS publication. Generally, the relevance and validity of the collaboration indicators will have to be related to the publication practices in the specific field of research and their Web of Science coverage. The collaboration indicators will not be presented in fields with low Web of Science coverage. (The R&D IS needs further development to make the basis for similar indicators.)

Indicator E1. Percentage Web of Science publications exhibiting international and collaboration in the addresses

2008 2009 2010 2011 2012 2013

International - 13.6% 8.8% 26.7% 21.2% 20.0%

National - 45.5% 61.8% 50.0% 33.3% 53.3%

Indicator E2. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared to EU28 in the field.

Share of international co-publications

Citation impact (compared to EU 28)

Italy 7 27%

Russia 5 19%

USA 4 15%

United Kingdom 2 8%

Austria 2 8%

6.2.5 F: Scholarly outputs and non-traditional scholarly outputs – statistical data (RU) Interpretation: the statistical data presented below give the panels a view on the trends in scholarly and non-traditional scholarly outputs during the evaluation period and set the publication profile of the Research Units in the context of the field in the CR. It also provides information on the potential reach of the journals in which the articles are published (national versus international), based upon the databases in which the journals are registered.

Page 98: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 96

Indicator F1: Publication of scholarly outputs and non-traditional scholarly outputs by the RU and their distribution over the years (raw data)

2009 2010 2011 2012 2013 Scholarly outputs Article in a periodical (J) 33 26 36 24 17 Monographs and books (B) 4 1 1 2 1 Book chapter (C.) 8 2 11 7 8 Conference proceedings / Article in proceedings (D) 60 53 50 45 36 Non-traditional research outputs Results used by the funding provider, i.e. into legislation or norm, into non-legislative or strategic documents (H)

1 1 1

Research report containing classified information (V)

Certified methodologies, art conservation methodologies, specialized map works (N) 1

Indicator F2: Shares of scholarly outputs and non-traditional scholarly outputs in the R&D IS categories – Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR)

RU share of Field total

Scholarly outputs Article in a periodical (J) 4,925 2.76% Monographs and books (B) 251 3.59% Book chapter (C) 626 5.75% Conference proceedings / Article in proceedings (D) 11,942 2.04% Non-traditional scholarly outputs Results used by the funding provider, i.e. projected into legislation or norm, projected into non-legislative or strategic documents (H) 7 42.86% Research report containing classified information (V) 2 0.00% Certified methodologies, art conservation methodologies, specialized map works (N) 25 4.00%

Indicator F3: Number and shares of articles published in typologies of journals (national versus international reach) – RU (2008-2013)

Publication channels Number Share of the total

Article in a periodical registered in the Web of Science (Jimp) 74 54.4% Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 17 12.5% Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 1 0.7% Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 28 20.6%

Page 99: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 97

Indicator F4: Shares of articles published in typologies of journals (national versus international reach) - Research Unit versus Field total in the CR (2008-2013)

Publication channels Number in the Field (CR)

RU share of Field total

Article in a periodical registered in the Web of Science (Jimp) 1,495 4.95% Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 1,081 1.57% Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 1 100.00% Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 983 2.85%

6.2.6 G: IPR-related outputs (RU) Interpretation: also in this case, the panels are given a view on the trends in patents awarded during the evaluation period. The data set the information for the Research Units in the context of the field in the CR and give a view on the importance of the patents in terms of geographical reach.

Indicator G1: Patents and other forms of IP awarded to the RU and their distribution over the years (raw data)

2008 2009 2010 2011 2012 2013 Patent (P) Plant variety (Zodru) Animal breed (Zplem)

Indicator G2: Shares of IPR-related outputs in the R&D IS categories - Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR) RU share of Field total

Patent (P) 32 0.00% Plant variety (Zodru)

-

Animal breed (Zplem)

-

Indicator G3: Number of patents in the patent offices in different countries (2008-2013)

Number for the RU

Number in the Field (CR)

Patents in the Czech Industrial Property Office 21

Patents in the European Patent Office (EPO) 2

Patents in the US/Japan Patents Offices 8

Patents in other international patent offices 1

Total Patents 32

Page 100: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 98

6.3 Biological & Agricultural sciences – Biological sciences (Agrobiology): Czech University of Life Science, Faculty of Agrobiology

6.3.1 B: Characteristics of the Research Unit (RU) Indicator B1 and B2 are given for all included Research Units in all fields in the evaluation methodology, both within fields and within Evaluated Units. This will give an overview of the relative contributions of each Evaluated Unit to each field in the Czech Republic. In the Small Pilot Evaluation, the two indicators are given for each selected RU with the field characteristics (Indicator group A above) as context.

Interpretation: These indicators identify the relative size – measured in publication output (research activity) of the Research Unit within its field in the country and within the Evaluated Unit it belongs to. Coverage in Web of Science is field dependent. The bibliometric report will advise how to use, or not to use, indicators based on Web of Science in each specific field.

Indicator B1. Percentage shares of all Czech publications in RU (WoS and R&D IS)

  Area   Field   Count   %  CZ  

WoS  

Physical  Sciences   Chemical  sciences   11   0.10%  

Engineering  and  technology   Other  engineering  and  technologies   32   0.87%  

Medical  and  Health  sciences  Basic  medical  research   1   0.02%  

Health  sciences   9   0.48%  

Biological  and  Agricultural  Sciences  

Biological  sciences   21   0.18%  

Agriculture,  forestry,  and  fisheries   48   2.33%  

Other  agricultural  sciences   1   0.22%  

R&D  IS  

Engineering  and  Technology   Other  engineering  and  technologies   14   0.23%  

Biological  and  Agricultural  Sciences  

Biological  sciences   1   0.01%  

Agriculture,  forestry,  and  fisheries   207   3.00%  

Other  agricultural  sciences   1   0.18%  

Social  Sciences   Economics  and  business   14   0.07%  

The tables above show the field structure of the Research Unit based on the publications registered in the WoS and the R&D IS. While the researchers composing the Research Unit conduct their research primarily in the field of evaluation, their publications may appear in journals registered against other fields.

Page 101: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 99

Indicator B2. Percentage shares of all publications from the Evaluated Unit, all fields taken together

6.3.2 C: Publishing Profile (RU) Interpretation: The indicators in this group give an overview of the publishing profile of the Research Unit. This profile should be contextualized by other material provided for the evaluation panel, such as:

• The information on the level of the field given in indicator group A

• Field and type of research

• Mission, size and resources of the Evaluated Unit, as given in the self-evaluation and in other statistical information provided for the evaluation

Indicator C1. Number and shares of publications in scholarly R&D IS categories

WoS  16.3%  

R&D  IS  15.6%  

Percentage  shares  of  all  publications  from  the  EvU,  all  8ields  

Article  in  a  periodical  (J);  193;  82%  

Monographs  and  books  (B);  1;  0%  

Book  chapter  (C);  9;  4%  

Conference  proceedings  /  Article  in  

proceedings  (D);  34;  14%  

Number  and  shares  of  publications  in  scholarly  R&D  IS  categories  

Page 102: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 100

Indicator C2. Number of and percentage Web of Science publications among all scholarly R&DIS publications

Data Source Indicator

Web of Science (N) 80

R&D IS (N) 237

Web of Science (%) 33.8%

Indicator C3. Mean and median number of authors and institutional affiliations (published addresses) per WoS publication

Data Source Mean Median

Authors 4.8 5

Addresses 1.8 1

Indicator C4. Percentage Web of Science publications in the most cited 10 per cent, 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports.

Indicator C5. Number and percentage of articles in the field per journal in Web of Science

Journal Articles Percentage

SEED  AND  SEEDLINGS   32   39.5  %  

LISTY  CUKROVARNICKE  A  REPARSKE   10   12.3  %  

PLANT  SOIL  AND  ENVIRONMENT   9   11.1  %  

FOOD  CHEMISTRY   9   11.1  %  

JOURNAL  OF  FOOD  AGRICULTURE  &  ENVIRONMENT   5   6.2  %  

ROMANIAN  AGRICULTURAL  RESEARCH   3   3.7  %  

1  

12  

35  

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

Publishing  Pro8ile.  Share  of  WoS  publications  in  most  cited  journals  in  the  8ield,  based  on  2014  JIF.  Example:  Physical  Sciences  

Least  cited  50  %  

Most  cited  50  %  

Most  cited  25  %  

Most  cited  10  %  

Page 103: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 101

CZECH  JOURNAL  OF  FOOD  SCIENCES   2   2.5  %  

CZECH  JOURNAL  OF  GENETICS  AND  PLANT  BREEDING   2   2.5  %  

NATURAL  PRODUCT  COMMUNICATIONS   1   1.2  %  

PROSPEROUS  OIL  CROPS  2009   1   1.2  %  

LWT-­‐FOOD  SCIENCE  AND  TECHNOLOGY   1   1.2  %  

JOURNAL  OF  AGRICULTURAL  AND  FOOD  CHEMISTRY   1   1.2  %  

INTERNATIONAL  JOURNAL  OF  AGRICULTURE  AND  BIOLOGY   1   1.2  %  

FOOD  RESEARCH  INTERNATIONAL   1   1.2  %  

FOOD  CONTROL   1   1.2  %  

JOURNAL  OF  FOOD  COMPOSITION  AND  ANALYSIS   1   1.2  %  

JOURNAL  OF  CEREAL  SCIENCE   1   1.2  %  

6.3.3 D: Citation Impact (RU) Interpretation: Citation indicators may inform about the international influence, impact, or usefulness of the research, as seen in the frequency of received citations per publication. Note that citation distributions are highly skewed. One or a few highly cited articles may influence the averages considerably and cause large variations from year to year as well. The two indicators presented here should therefore be seen in relation to each other. One of them identifies the average citation rate, while the other records the presence of articles among the most highly cited articles in the world and EU.

The relevance and validity of the citation indicators have to be related to the specific field and type of research. Especially relevant is the Web of Science coverage of the field. Citation indicators will not be presented in fields with low Web of Science coverage.

Indicator D1. Mean Field Normalized Citation Impact for the Research Unit, compared to the world average (=1,00) and the EU28 average (=1,00).

2009 2010 2011 2012

Compared to world average 1.43   0.96   0.59   1.84  

Compared to EU 28 average 1.25   0.82   0.49   1.58  

Page 104: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 102

Indicator D2. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

6.3.4 E: Collaboration (RU) Interpretation: These indicators provide information about the relations to main partners in international and national collaboration in the field, as seen in the addresses of co-authored publications. Caution has to be taken with regard to publishing practices in the field, see Indicator B3: Mean and median number of authors and addresses per WoS publication. Generally, the relevance and validity of the collaboration indicators will have to be related to the publication practices in the specific field of research and their Web of Science coverage. The collaboration indicators will not be presented in fields with low Web of Science coverage. (The R&D IS needs further development to make the basis for similar indicators.)

Indicator E1. Percentage Web of Science publications exhibiting international and national collaboration in the addresses

2009 2010 2011 2012 2013

International 0.0%   0.0%   0.0%   18.2%   15.6%  

National 25.0%   75.0%   42.3%   81.8%   21.9%  

Indicator E2. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared to EU28 in the field. Share of international

co-publications Citation impact

(compared to EU 28)

Slovakia   57%   0.65  

Poland   43%   4.37  

7   6  

4   3  0   2  

35   35  

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

World   EU  

Publishing  Pro8ile.  Publications  among  the  most  cited  (world,  EU28)  

Least  cited  50  %  

Most  cited  50  %  

Most  cited  25  %  

Most  cited  10  %  

Page 105: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 103

6.3.5 F: Scholarly outputs and non-traditional scholarly outputs – statistical data (RU) Interpretation: the statistical data presented below give the panels a view on the trends in scholarly and non-traditional scholarly outputs during the evaluation period and set the publication profile of the Research Units in the context of the field in the CR. It also provides information on the potential reach of the journals in which the articles are published (national versus international), based upon the databases in which the journals are registered.

Indicator F1: Publication of scholarly outputs and non-traditional scholarly outputs by the RU and their distribution over the years (raw data)

2009 2010 2011 2012 2013 Scholarly outputs Article in a periodical (J) 57   33   33   40   30  Monographs and books (B)

 1  

     Book chapter (C.)  

1   3   2   3  Conference proceedings / Article in proceedings (D)

 2   17  

 15  

Non-traditional research outputs Results used by the funding provider, i.e. into legislation or norm, into non-legislative or strategic documents (H) Research report containing classified information (V) Certified methodologies, art conservation methodologies, specialized map works (N)

 2  

 5   5  

Indicator F2: Shares of scholarly outputs and non-traditional scholarly outputs in the R&D IS categories – Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR)

RU share of Field total

Scholarly outputs Article in a periodical (J) 9,254   2.09%  Monographs and books (B) 242   0.41%  Book chapter (C) 642   1.40%  Conference proceedings / Article in proceedings (D) 2,949   1.15%  Non-traditional scholarly outputs    Results used by the funding provider, i.e. projected into legislation or norm, projected into non-legislative or strategic documents (H) 44   0.00%  Research report containing classified information (V) 1   0.00%  Certified methodologies, art conservation methodologies, specialized map works (N) 1,631   0.74%  

Page 106: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 104

Indicator F3: Number and shares of articles published in typologies of journals (national versus international reach) – RU (2008-2013)

Publication channels Number Share of the total

Article in a periodical registered in the Web of Science (Jimp) 50   26%  Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 24   12%  

Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 0   -­‐  

Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 109   56%  

Indicator F4: Shares of articles published in typologies of journals (national versus international reach) - Research Unit versus Field total in the CR (2008-2013)

Publication channels Number in the Field (CR)

RU share of Field total

Article in a periodical registered in the Web of Science (Jimp) 2,901   1.72%  

Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 1,517   1.58%  

Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 2   0.00%  

Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 3,587   3.04%  

6.3.6 G: IPR-related outputs (RU) Interpretation: also in this case, the panels are given a view on the trends in patents awarded during the evaluation period. The data set the information for the Research Units in the context of the field in the CR and give a view on the importance of the patents in terms of geographical reach.

Indicator G1: Patents and other forms of IP awarded to the RU and their distribution over the years (raw data)

2008 2009 2010 2011 2012 2013 Patent (P) - - - - - - Plant variety (Zodru) - - - - - - Animal breed (Zplem) - - - - - -

Indicator G2: Shares of IPR-related outputs in the R&D IS categories - Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR) RU share of Field total

Patent (P) 55   0.0% Plant variety (Zodru) 220   0.0% Animal breed (Zplem) 0 - TOTAL 275 0.0%

Page 107: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 105

Indicator G3: Number of patents in the patent offices in different countries (2008-2013)

Number for the RU

Number in the Field (CR)

Patents in the Czech Industrial Property Office 0 46  

Patents in the European Patent Office (EPO) 0 0  

Patents in the US/Japan Patents Offices 0 7  

Patents in other international patent offices 0 2  

Total Patents 0 55

6.4 Social sciences – Sociology: Masaryk University, Faculty of Social Studies

6.4.1 B: Characteristics of the Research Unit (RU) Indicator B1 and B2 are given for all included Research Units in all fields in the evaluation methodology, both within fields and within Evaluated Units. This will give an overview of the relative contributions of each Evaluated Unit to each field in the Czech Republic. In the Small Pilot Evaluation, the two indicators are given for each selected RU with the field characteristics (Indicator group A above) as context.

Interpretation: These indicators identify the relative size – measured in publication output (research activity) of the Research Unit within its field in the country and within the Evaluated Unit it belongs to. Coverage in Web of Science is field dependent. The bibliometric report will advise how to use, or not to use, indicators based on Web of Science in each specific field.

Indicator B1. Percentage shares of all Czech publications in RU (WoS and R&D IS)

  Area   Field   Count   %  CZ  

WoS  

Social  sciences  

Economics  and  business   1   0.03%  

Sociology   36   3.62%  

Social  and  economic  geography   4   0.57%  

Other  social  sciences   4   1.74%  

Humanities  Languages  and  literature   1   0.18%  

Art  (arts,  history  of  arts,  performing  arts,  music)   3   0.99%  

R&D  IS   Social  Sciences   Sociology   187   6.29%  

The tables above show the field structure of the Research Unit based on the publications registered in the WoS and the R&D IS. While the researchers composing the Research Unit conduct their research primarily in the field of evaluation, their publications may appear in journals registered against other fields.

Page 108: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 106

Indicator B2. Percentage shares of all publications from the Evaluated Unit, all fields taken together

6.4.2 C: Publishing Profile (RU) Interpretation: The indicators in this group give an overview of the publishing profile of the Research Unit. This profile should be contextualized by other material provided for the evaluation panel, such as:

• The information on the level of the field given in indicator group A

• Field and type of research

• Mission, size and resources of the Evaluated Unit, as given in the self-evaluation and in other statistical information provided for the evaluation

WoS  20.3%  

R&D  IS  12.7%  

Percentage  shares  of  all  publications  from  the  EvU,  all  8ields  

Page 109: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 107

Indicator C1. Number and shares of publications in scholarly R&D IS categories

Indicator C2. Number of and percentage Web of Science publications among all scholarly R&DIS publications

Data Source Indicator

Web of Science (N) 41

R&D IS (N) 187

Web of Science (%) 21.9%

Indicator C3. Mean and median number of authors and institutional affiliations (published addresses) per WoS publication

Data Source Mean Median

Authors 1.7 2

Addresses 1.4 1

Indicator C4. Percentage Web of Science publications in the most cited 10 per cent, 25 per cent, and 50 per cent of the journals in the field (counted from the top by the number of articles in the field), based on Journal Impact Factor in the latest (2014) edition of Journal Citation Reports.

Articles  in  peer-­‐reviewed  

journals  (J);  119;  64%  

Monographs  (B);  14;  7%  

Book  chapters  (C);  44;  24%  

Proceedings  papers  (D);  10;  

5%  

Number  and  shares  of  publications  in  scholarly  R&D  IS  categories  

Page 110: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 108

Indicator C5. Number and percentage of articles in the field per journal in Web of Science

Journal Articles Percentage

SOCIOLOGICKY  CASOPIS-­‐CZECH  SOCIOLOGICAL  REVIEW   23   56.1  %  

SOCIOLOGIA   7   17.1  %  

MESTO:  PROMENLIVA  NE/SAMOZREJMOST   3   7.3  %  

INTERNATIONAL  MIGRATION   2   4.9  %  

SOCIOLOGICAL  THEORY  AND  METHODS   1   2.4  %  

POETICS   1   2.4  %  

JOURNAL  OF  YOUTH  STUDIES   1   2.4  %  

EUROPEAN  SOCIOLOGICAL  REVIEW   1   2.4  %  

TIJDSCHRIFT  VOOR  ECONOMISCHE  EN  SOCIALE  GEOGRAFIE   1   2.4  %  

INTERNATIONAL  SOCIOLOGY   1   2.4  %  

6.4.3 D: Citation Impact (RU) Interpretation: Citation indicators may inform about the international influence, impact, or usefulness of the research, as seen in the frequency of received citations per publication. Note that citation distributions are highly skewed. One or a few highly cited articles may influence the averages considerably and cause large variations from year to year as well. The two indicators presented here should therefore be seen in relation to each other. One of them identifies the average citation rate, while the other records the presence of articles among the most highly cited articles in the world and EU.

The relevance and validity of the citation indicators have to be related to the specific field and type of research. Especially relevant is the Web of Science coverage of the field. Citation indicators will not be presented in fields with low Web of Science coverage.

1  1  

35  

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

Publishing  Pro8ile.  Share  of  WoS  publications  in  most  cited  journals  in  the  8ield,  based  on  2014  JIF  

Least  cited  50  %  

Most  cited  50  %  

Most  cited  25  %  

Most  cited  10  %  

Page 111: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 109

Indicator D1. Mean Field Normalized Citation Impact for the Research Unit, compared to the world average (=1,00) and the EU28 average (=1,00). 2009 2010 2011 2012

Compared to world average 0.33   0.21   0.74   4.77  

Compared to EU 28 average 0.27   0.18   0.60   3.88  

Indicator D2. Number and percentage publications among the top 10%, and 25% most cited publications (world, EU28).

6.4.4 E: Collaboration (RU) Interpretation: These indicators provide information about the relations to main partners in international and national collaboration in the field, as seen in the addresses of co-authored publications. Caution has to be taken with regard to publishing practices in the field, see Indicator B3: Mean and median number of authors and addresses per WoS publication. Generally, the relevance and validity of the collaboration indicators will have to be related to the publication practices in the specific field of research and their Web of Science coverage. The collaboration indicators will not be presented in fields with low Web of Science coverage. (The R&D IS needs further development to make the basis for similar indicators.)

Indicator E1. Percentage Web of Science publications exhibiting international and national collaboration in the addresses 2009 2010 2011 2012 2013

International 0.0%   20.0%   0.0%   11.1%   11.1%  

National 8.3%   0.0%   50.0%   11.1%   22.2%  

2   0  2   4  

3  0  

23  26  

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

World   EU  

Publishing  Pro8ile.  Publications  among  the  most  cited  (world,  EU28)  

Least  cited  50  %  

Most  cited  50  %  

Most  cited  25  %  

Most  cited  10  %  

Page 112: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 110

Indicator E2. Five most frequent collaborating countries, their shares of the publications in indicator D1, and the Mean Field Normalized Citation Impact of articles in each country relation, compared globally in the field.

Share of international co-publications

Citation impact (compared to world)

Country name Insufficient number of WoS articles with international collaboration

6.4.5 F: Scholarly outputs and non-traditional scholarly outputs – statistical data (RU) Interpretation: the statistical data presented below give the panels a view on the trends in scholarly and non-traditional scholarly outputs during the evaluation period and set the publication profile of the Research Units in the context of the field in the CR. It also provides information on the potential reach of the journals in which the articles are published (national versus international), based upon the databases in which the journals are registered.

Indicator F1: Publication of scholarly outputs and non-traditional scholarly outputs by the RU and their distribution over the years (raw data)

2009 2010 2011 2012 2013 Scholarly outputs Article in a periodical (J) 25   18   16   32   28  Monographs and books (B) 2   4   3     5  Book chapter (C.) 12   2   11   5   14  Conference proceedings / Article in proceedings (D) 4   5   1    

 Non-traditional research outputs Results used by the funding provider, i.e.

into legislation or norm, into non-legislative or strategic documents (H)

Research report containing classified information (V)

Certified methodologies, art conservation methodologies, specialized map works (N)

Indicator F2: Shares of scholarly outputs and non-traditional scholarly outputs in the R&D IS categories – Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR)

RU   share   of  Field  total  

Scholarly outputs    Article in a periodical (J) 1317   9.04%  Monographs and books (B) 255   5.49%  Book chapter (C) 773   5.69%  Conference proceedings / Article in proceedings (D) 629   1.59%  Non-traditional scholarly outputs  Results used by the funding provider, i.e. projected into legislation or norm, projected into non-legislative or strategic documents (H) 3   0.00%  

Page 113: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 111

Research report containing classified information (V)  

-­‐  Certified methodologies, art conservation methodologies, specialized map works (N) 53   0.00%  

Indicator F3: Number and shares of articles published in typologies of journals (national versus international reach) – RU (2008-2013)

Publication channels Number Share of the total

Article in a periodical registered in the Web of Science (Jimp) 43   36.1%  Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 36   30.3%  Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 4   3.4%  Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 23   19.3%  

Indicator F4: Shares of articles published in typologies of journals (national versus international reach) - Research Unit versus Field total in the CR (2008-2013)

Publication channels Number in the Field (CR)

RU share of Field total

Article in a periodical registered in the Web of Science (Jimp) 301   16.94%  Article in a source registered in SCOPUS, which is not registered in WoS (JSC) 131   28.24%  Article in a reviewed periodical in the ERIH database, which is not registered in WoS or SCOPUS (Jneimp) 44   9.09%  Article in a reviewed Czech periodical, which is not registered in WoS, SCOPUS or ERIH (Jrec) 544   7.17%  

6.4.6 G: IPR-related outputs (RU) Interpretation: also in this case, the panels are given a view on the trends in patents awarded during the evaluation period. The data set the information for the Research Units in the context of the field in the CR and give a view on the importance of the patents in terms of geographical reach.

Indicator G1: Patents and other forms of IP awarded to the RU and their distribution over the years (raw data)

2008 2009 2010 2011 2012 2013 Patent (P) - - - - - - Plant variety (Zodru) - - - - - - Animal breed (Zplem) - - - - - -

Indicator G2: Shares of IPR-related outputs in the R&D IS categories - Research Unit versus Field total in the CR (2008-2013)

Number in the Field (CR) RU share of Field total

Patent (P) - - Plant variety (Zodru) - -

Page 114: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 10, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 112

Animal breed (Zplem) - - TOTAL - -

Indicator G3: Number of patents in the patent offices in different countries (2008-2013)

Number for the RU

Number in the Field (CR)

Patents in the Czech Industrial Property Office - -

Patents in the European Patent Office (EPO) - -

Patents in the US/Japan Patents Offices - -

Patents in other international patent offices - -

Total Patents - -

Page 115: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Background report 5, The Small Pilot Evaluation: Feedback and results - Summary report – draft version for public consultation

R&D Evaluation Methodology and Funding Principles 113

7. Examples of Comprehensive Data Reports for the SPE

In this chapter we reproduce two examples of Comprehensive Data Reports that were produced in the context of this study. These reports provided the (processed) data based on the self-assessment reports submitted by the participating EvUs.

EvUs that gave the study team consent for the publication of these reports were

• The Institute of Analytical Chemistry of the AS CR

• The Masaryk University, Faculty of Social Studies

Page 116: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results
Page 117: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

E"mail'address [email protected]

Research'Unit'Field

1.,Physical,Sciences,3,1.4,Chemical,sciences

Contact'person'for'the'Research'Unit

Scientific,Research,Organisation

Please,provide,the,name,and,contact,details,of,a,person,at,the,Research,Unit.

Name'&'Surname Zbyněk,VečeřaTelephone'number 532290168

Bohumil,BočekalPosition senior,researcherTelephone'number 532290246E"mail'address [email protected]

Type'of'the'Research'Organisation

Research,Unit,1

Panel'Data'Report

Name'&'Surname Zbyněk,Večeřa

Name'of'the'EvU: Institute'of'Analytical'Chemistry'of'the'AS'CR,'v.v.i.ID'of'the'EvU: ico:68081715

Head'of'the'internal'evaluation'committee

Position Head,of,the,Depatment,Environmental,Analytical,Chemistry,Telephone'number 532290168E"mail'address [email protected]

Contact'person'for'the'Evaluated'Unit

Name'&'Surname

Page 118: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

2009 2010 2011 2012 2013 201435.9 32.0 29.7 30.4 31.4 28.711.4 9.1 2.5 10.5 3.4 5.547.3 41.1 32.2 40.9 34.8 34.2

2009 2010 2011 2012 2013 2014

4.7 4.5 2.9 3.7 4.5 4.1

10% 11% 9% 9% 13% 12%

(in,million,CZK)

2009 2010 2011 2012 2013 20140.0 0.3 1.5 4.2 3.7 4.00.0 0.0 0.0 0.0 0.0 0.01.2 1.5 0.0 0.0 0.0 0.00.0 0.0 2.5 1.3 2.0 2.00.0 0.0 0.0 0.0 0.0 0.01.2 1.9 4.0 5.5 5.7 6.0

2009 2010 2011 2012 2013 20140.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.0

2009 2010 2011 2012 2013 20140 0 0 0 0 00 0.085 0.01 0 0 00 0.085 0.01 0 0 0

2009 2010 2011 2012 2013 20140.15 0.06 0.10 0.04 0.08 0.100.00 0.00 0.00 0.00 0.00 0.000.00 0.28 0.65 0.52 0.00 0.130.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.000.15 0.34 0.74 0.56 0.08 0.23

Contract,research,from,industryContract,research,from,ministriesContract,research,from,other,public,sector,institutions,(in,the,Czech,Republic)Contract,research,from,international,public,bodiesOther,contract,researchTotal

N5.'Competitive'international'funding

N6.'Funding'from'contract'research

A1.'Institutional'funding

N1.'Institutional'funding'"'EvU

Total,amount,of,institutional,funding,for,research,received,by,the,Evaluated,Unit,(in,million,CZK).

Institutional,funding,for,research

Background,information

A2.'Ability'to'attract'external'funding'(RU)

N3.'National'Competitive'Funding

Science,FoundationTechnology,AgencyOther,state,resources,–,Ministry,of,EnvironmentOther,state,resources,–,Ministry,of,CultureOther,state,resources,–,Ministry,….Total

N4.'National'funding'from'Structural'Funds

Other,institutional,funding,Total'institutional'funding'EvU

Share,of,EvU,institutional,funding,for,research

N2.'Institutional'funding'for'research'"'RU

Total,amount,of,institutional,funding,for,research,received,by,the,Evaluated,Unit,(in,million,CZK).

Institutional,funding,for,research,(estimate)

OP,RDI,3,European,Centres,of,ExcellenceOP,RDI,3,Regional,R&D,centresOP,Prague,Competitiveness,(OPPC),Other,–,OP,1Total

European,FundingFunding,from,other,foreign,public,sourcesTotal

Page 119: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Headcounts'

2009 2010 2011 2012 2013 201436 31 38 43 43 4318 8 12 14 16 1554 39 50 57 59 58

2009 2010 2011 2012 2013 201466.7% 79.5% 76.0% 75.4% 72.9% 74.1%33.3% 20.5% 24.0% 24.6% 27.1% 25.9%100% 100% 100% 100% 100% 100%

FTE

2009 2010 2011 2012 2013 201434.9 31.0 35.7 39.9 41.0 39.59.9 4.1 9.4 9.2 9.6 10.4

44.8 35.1 45.1 49.1 50.5 49.9

2009 2010 2011 2012 2013 201477.9% 88.3% 79.1% 81.2% 81.1% 79.2%22.1% 11.7% 20.9% 18.8% 18.9% 20.8%100% 100% 100% 100% 100% 100%

All,other,research,organisations,

2009 2010 2011 2012 2013 20140.0 0.0 0.0 0.0 0.0 0.012.5 12.0 13.0 13.0 13.3 13.27.0 6.0 5.0 6.0 9.0 9.11.0 2.0 6.6 8.2 8.7 10.410.0 10.0 10.1 9.8 6.0 4.04.4 1.0 1.0 2.9 4.0 2.80.0 1.0 1.0 3.9 3.1 2.99.9 3.1 8.4 5.3 6.4 7.5

44.8 35.1 45.1 49.1 50.5 49.9

2009 2010 2011 2012 2013 20140% 0% 0% 0% 0% 0%

28% 34% 29% 26% 26% 26%16% 17% 11% 12% 18% 18%2% 6% 15% 17% 17% 21%

22% 28% 22% 20% 12% 8%10% 3% 2% 6% 8% 6%0% 3% 2% 8% 6% 6%

22% 9% 19% 11% 13% 15%100% 100% 100% 100% 100% 100%

Total,research,staff

Total'overall

Shares,of,the,totalTotal,research,staff

Total,support,staff,(i.e.,scientific/technical/admin)Total,research,staff

Raw,data

Institutional,management,and,development,potential

A.'Quality'adequacy'of'the'research'environment

Total,support,staff,(i.e.,scientific/technical/admin)

N8.'R&D'personnel'in'the'EvU'"'Detailed'(FTE)

Shares,of,the,totalTotal,research,staffTotal,support,staff,(i.e.,scientific/technical/admin)Total'overall

Total,support,staff,(i.e.,scientific/technical/admin)Total'overall

Total'overall

A3.'Research'capacity

Research,capacity,in,the,EvU

N7.'R&D'personnel'in'the'EvU'"'Overview

Raw,data

Senior,scientist,/,scholarScientist,/,scholar,Associated,scientist,/,scholarPostdoctoral,fellow,

Raw,dataScientific,staff,

Shares,of,the,totalScientific,staff,Senior,scientist,/,scholarScientist,/,scholar,Associated,scientist,/,scholar

Research,assistant,(PhDs,included)Sup.,staff,for,research,(Technicians)Other,staff,Total

Postdoctoral,fellow,Research,assistant,(PhDs,included)Sup.,staff,for,research,(Technicians)Other,staff,Total

Page 120: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Headcounts

2009 2010 2011 2012 2013 20147 6 6 7 10 931 30 37 39 39 40

22.6% 20.0% 16.2% 17.9% 25.6% 22.5%

FTE

2009 2010 2011 2012 2013 20146 6 6 6 6 731 30 35 37 37 37

18% 20% 17% 17% 17% 19%

2009 2010 2011 2012 2013 201418.0% 20.0% 17.3% 17.1% 17.5% 19.3%82.0% 80.0% 82.7% 82.9% 82.5% 80.7%18.0% 20.0% 17.3% 17.1% 17.5% 19.3%

Ratio,FTE,/,HC

2009 2010 2011 2012 2013 201478.6% 100.0% 100.0% 90.0% 64.5% 78.9%98.4% 100.0% 93.8% 94.7% 94.7% 91.8%

Key,pieces,of,experimental,equipment,,scholarly,resources,such,as,libraries,or,databases.,Maximum,of,10,different,cases.

N10.'Researchers'in'the'RU

Raw,dataResearchers,in,the,RUResearchers,in,the,EvURU,share,of,total,researchers,in,the,EvU

Researchers,in,the,RUResearchers,in,the,EvURU,share,of,total,researchers,in,the,EvU

N11.'Level'of'researchers''full"time'employment'

Raw,dataResearchers,in,the,RUResearchers,in,the,EvURU,share,of,total,researchers,in,the,EvU

Shares,of,the,total

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'Aerosol,Spectrometer yesDescription'of'equipment'or'resource'(50'words)'

In,the,RUIn,the,EvU

A4.'Infrastructure'for'research'at'the'RU

Research,infrastructure,(description,&,assessment)

N12.'Key'experimental'equipment

Description'of'equipment'or'resource'(50'words)'Ion,Chromatograph,ICS32100,(Dionex,,USA),combines,the,Reagent3Free,technology,,electrolytic,sample,preparation,,eluent,generation,capabilities,and,auto3suppression,device,to,perform,all,types,of,electrolytically,generated,isocratic,and,gradient,IC,separations,using,conductivity,detection.,In,dependence,on,actual,configuration,,ICS32100,allows,analysis,of,anions,or,cations,after,exchange,of,separation,modules.

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'GC3MS no

Optical,spectrometer,Welas,3000,(Palas,,Germany),allows,measurement,of,size,and,concentration,of,aerosol,particles,in,size,range,of,0.25,–,30,µm,(sensor,Welas,2200),,combined,with,CNC,module,(sensor,Welas,2100),detects,particles,down,to,10,nm,in,diameter.,Spectrometer,is,based,on,scattered3light,analysis,on,single,particles.

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'Ion,Chromatograph no

Description'of'equipment'or'resource'(50'words)'Scanning,Mobility,Particle,Sizer,Spectrometer,(SMPS),Model,3936L72,(TSI,,USA),allows,measurement,of,size,and,concentration,of,aerosol,particles,in,size,range,of,8–1000,nm.,Aerosols,are,classified,via,electrical,mobility,in,Differential,Mobility,Analyzer,(DMA,3080L),and,obtained,monodisperse,particles,are,then,detected,in,Condensation,Particle,Counter,(CPC,3772).

Description'of'equipment'or'resource'(50'words)'Gas,Chromatography,–,Mass,Spectrometry,system,(Agilent,,USA),combines,separation,of,organic,compounds,in,Gas,Chromatograph,(GC,7890A),and,detection,in,Mass,Spectrometer,(2x,MSD,5975C).,GC,is,equipped,with,Deans,Switch,and,two,capillary,columns,(polar,,nonpolar).,Both,MSD,are,equipped,with,electron/chemical,ionization,and,simple,quadrupole.

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'Scanning,Mobility,Particle,Sizer,Spectrometer, no

Page 121: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Details,of,the,type,of,infrastructure,and,purpose,3,no,more,than,10,different,cases

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'DryVap,Concentrator,System noDescription'of'equipment'or'resource'(50'words)'

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'High3volume,aerosol,samplers noDescription'of'equipment'or'resource'(50'words)'High3volume,aerosol,samplers,DHA380,and,DH377,(Digitel,,Switzerland),are,used,for,autonomous,,continuous,sampling,of,aerosols,(size,fractions,PM1,,PM2.5,and,PM10),on,filters,for,subsequent,analysis,of,chemical,composition.,Sampler,DHA380,is,equipped,with,automatic,filter,changer,(up,to,15,filters),while,DH377,uses,manual,changer,of,single,filter.

Description'of'equipment'or'resource'(50'words)'Microbalance,M5P,(Sartorius,Germany),provides,highly,precise,and,very,sensitive,weighing,filters,with,diameters,ranging,up,to,160,mm,(with,upgraded,version).,Microbalance,weighs,up,to,5,000,mg,with,precision,of,1,µg.

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'Aerosol,generator yes

DryVap,Concentrator,System,(Horizon,Technology,,USA),automatically,removes,water,and,concentrates,samples,through,evaporation,of,the,extract,solvent,for,subsequent,GC/MS,or,HPLC/MS,analysis.,The,system,combines,vacuum,,heat,and,nitrogen,sparge,to,provide,gentle,evaporation,of,solvent,with,automatic,endpoint,detection,and,shut3off,in,six,parallel,samples.

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'Microbalance yes

Description'of'equipment'or'resource'(50'words)'Instrumentation,for,atomic,absorption,spectrometry,with,electrothermal,atomization,(AAnalyst,600,Perkin3Elmer,,USA),is,equipped,with,a,transversally,heated,graphite,furnace,and,an,automatic,sampler,for,introduction,of,samples.,It,enables,determination,of,trace,and,ultratrace,elements,in,various,types,of,environmental,samples,down,to,ppb3level.

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'Microwave,digestion,unit yes

Description'of'equipment'or'resource'(50'words)'Aerosol,generator,MAG32010,(Palas,,Germany),is,based,on,the,principle,of,a,Sinclair3LaMer,generator,when,vaporized,particle,material,(e.g.,DEHs),condenses,on,the,condensation,nuclei,(e.g.,NaCl).,MAG32010,is,a,portable,generator,that,generates,monodisperse,and,highly,concentrated,test,aerosols,for,calibration,and,evaluation,of,aerosol,instruments.

Name'of'the'equipment Is'the'equipment'or'resource'older'than'10'years?'Atomic,absorption,spectrometer yes

Description'of'equipment'or'resource'(50'words)'Microwave,digestion,system,Uniclever,(Plazmatronika,,Wroclaw,,Poland),is,designed,for,high,pressure,microwave,assisted,sample,decomposition,in,concentrated,high,purity,acids.,High,pressure,autoclave,is,lined,with,PTFE,mantle,and,is,equipped,with,high,density,PTFE,reaction,vessel.,This,system,enables,decomposition,of,organic,and,inorganic,environmental,samples.

N13.'Adequacy'of'the'research'infrastructure'(self"assessment)

The,laboratories,of,the,RU,are,well,equipped,with,the,instrumentation,,specified,above,,for,investigation,of,environmental,systems,and,for,development,of,new,analytical,methods,,i.e.,chromatographic,,spectroscopic,and,sampling,instruments,,as,well,as,devices,for,characterization,of,aerosols.,RU,also,possesses,an,accredited,laboratory,with,special,chamber,and,devices,for,long,lasting,inhalation,experiments,with,small,animals.,RU,is,also,equipped,with,a,clean,room,facility,and,cabinets/clean,benches,to,significantly,reduce,contamination,risk,in,determination,of,trace,elements,in,various,environmental,samples.,Due,to,background,within,EvU,(mechanical,and,electronic,workshops),the,RU,can,manufacture,and,modify,for,research,purposes,various,prototypes,of,experimental,devices.,According,to,specific,demands,of,actual,and,forthcoming,proposed,projects,a,purchase,of,new,up,to,date,complementary,instrumentation,is,planned,in,future.

N14.'Usage'of'major'national'or'international'RI'(competitive'access)

Usage,of,national/international,RI,awarded,(based,on,competition)

NONE'PROVIDED

Page 122: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Shared,or,collaborative,use,of,scholarly,resources,such,as,libraries,or,databases,3,maximum,of,10,different,cases.

Usage,of,research,infrastructures,of,other,Czech,research,organisations,3,max,10,cases

NONE'PROVIDED

N16.'Use'of'RI'in'other'Czech'research'organisations'(non"competitive)

NONE'PROVIDED

Shared,or,collaborative,use,of,RI

N15.'Shared'or'collaborative'use'of'RU'equipment'with'other'research'organisations

Page 123: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Frequency,of,performance,review,/,Frequency,employees',feedback,on,potential,concerns,,issues,,challenges,related,to,their,jobs

B1.'Research'strategy

N17.'Description'of'the'research'strategy'for'the'RU

B.'Research'strategy'and'(HR)'management'

B2.'Human'resource'development

We,expect,none,or,minor,changes,in,research,strategy,for,years,201532019,that,arises,from,our,interest,in,flow,analysis,,preconcentration,and,separation,techniques,,trace,analysis,problems,with,particular,respect,to,the,methods,of,continuous,preconcentration,of,pollutants,from,gaseous,media,to,liquid,system,(the,mass,transfer,of,pollutants,to,aerosol,or,thin,film,of,the,liquids),,gas,and,liquid,chromatography,,automation,in,analytical,chemistry,,water,and,air,pollution,,and,food,analysis,Also,participation,in,research,related,to,the,aerosol,(nano3particles),issue,is,the,most,favorable,opportunity,for,our,scientific,research,in,near,future.,There,is,also,an,open,field,in,monitoring,of,pollutants,in,air,,biogenous,emissions,of,plants,and,pilot,health,risk,studies,for,ambient,particles,and,nano3particles.

Areas,of,interest,in,particular:,•,characterization,of,nano3particles,,i.e.,determination,of,main,,minor,and,trace,elements,,which,can,significantly,influence,basic,properties,of,nano3particles,•,inhalation,studies,of,nano3particles,with,respect,to,their,potential,toxicity,,i.e.,determination,of,toxic,components,in,small,animals,and,their,organs,of,interest,,which,are,in,a,direct,contact,with,nano3particles,and,in,target,tissues,with,secondary,physiologically,trans3located,components,of,nano3particles,•,studies,of,interaction,of,nano3particles,with,plants,•,investigation,of,diffusive,gradients,in,thin,film,technique,for,characterization,of,urban,aerosols,and,also,aerosols,containing,nano3particles,•,studies,of,formation,of,nano3particles,in,electrothermal,devices,for,analytical,atomic,spectrometry,methods,•,application,of,continuous,aerosol,collector,for,the,determination,of,selected,compounds,bound,to,aerosol,particles,•,development,of,methods,for,the,determination,of,particulate,organic,compounds,both,soluble,and,insoluble,in,water,•,studies,of,chemical,composition,of,atmospheric,aerosols,sampled,in,different,locations,•,identification,of,emission,sources,of,aerosols,and,toxicity,tests,of,aerosols,•,studies,of,bioaccessible,fraction,of,metals,in,urban,aerosols,as,well,as,in,exhausts,emitted,from,different,sources,(diesel,×,gasoline,engines,,biomass,burning,,coal,combustion),•,studies,of,the,distribution,of,compounds,of,interest,between,gas,and,aerosol,phases,•,development,of,new,methods,and,instrumentation,for,the,chemical,analysis,and/or,characterization,of,environmental,systems,(gas,,water,,soils).

N18.'Processes'for'career'development'in'the'EvU

Appraisal,and,monitoring,system,/,Competency,framework,for,different,job,roles

Depending,on,duration,of,working,contract,,minimum,period,is,1,year,,maximum,5,years.,Attestation,criteria,include,following,parameters:,number,of,regularly,published,scientific,papers,in,the,national,and,international,impacted,journals,according,to,ISI,,the,number,of,received,patent,applications,and,implemented,prototype,instrumentations.,Value,of,contributions,for,quality,production,of,the,institute,in,the,form,of,Impact,points,,converted,and,adjusted,to,co3authorship.,Citation,metrics,specifies,positions:,Research,assistant,3,a,graduate,who,should,start,Ph.D.,study,within,3,years,after,employment,at,the,institute;,Doctoral,(Ph.D.),student,3,regularly,publishes,at,least,one,scientific,paper,in,an,impacted,journal,per,year;,Postdoctoral,fellow,3,has,published,at,least,three,scientific,articles,in,impacted,journals,,and,continues,in,producing,at,least,one,scientific,paper,in,impacted,journals,per,year.,After,5,years,should,fulfill,criteria,to,promote,as,associated,scientist;,Associated,scientist,3,regularly,publishes,at,least,one,scientific,paper,in,impacted,journals,per,year,,leads,diploma,and,Ph.D.,students,and,participates,in,their,education;,Scientist,3,publishes,regularly,,leads,diploma,

Yes,,we,perform,regularly,evaluation,of,outputs,of,all,research,workers,as,well,as,research,groups.,Research,workers,are,attested,before,promoting,or,prolongation,of,contract,of,employment.,Scientific,groups,are,evaluated,in,a,five3year,period,within,evaluation,process,performed,in,all,institutions,of,Academy,of,Sciences,of,the,Czech,Republic.

Career,development,policy,in,the,EvU

Page 124: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Promotion,criteria,/,Individual,targets,/,Other,ways,to,tackle,the,career,development,of,employees

Approaches,for,EvU,to,develop,and,maintain,structures,and,practices,that,foster,good,research

Approach,by,EvU,to,help,early3career,researchers,making,their,way,into,the,profession

Supervision:,rights,and,obligations,of,both,supervisors,and,PhD,candidates

Educational,components,for,the,training,of,PhDs

Headcounts

2009 2010 2011 2012 2013 20147 6 6 7 10 96 5 6 2 1 0

86% 83% 100% 29% 10% 0%

FTE

2009 2010 2011 2012 2013 20146 6 6 6 6 72 3 2 1 0 0

27% 50% 36% 10% 5% 0%

Total,researchers,in,the,RU,Total,former,PhD,students

They,are,given,by,the,study,regulations,of,the,university,the,students,belong,to.

Supervisors,are,usually,senior,scientists,approved,by,the,Scientific,Board,of,the,university,where,PhD,students,study.

Share,of,total,researchers

Policy,and,practice,concerning,PhD,education,and,the,role,of,post3docs

N19.'Career'development'of'PhDs'and'post"docs

Total,researchers,in,the,RU,Total,former,PhD,studentsShare,of,total,researchers

N20.'Employment'of'former'PhD'students'in'the'RU

They,are,set,in,career,code,based,on,the,internal,standards,of,the,Academy,of,Sciences,of,the,Czech,Republic,for,career,development,of,the,university3educated,employees.,There,are,following,qualification,levels,for,the,university3educated,employees,established:1.,Research,assistant,3,,graduates,before,enrolling,in,the,Ph.D.,degree,program.2.Ph.D.,student,3,enrolled,in,the,doctoral,(Ph.D.),degree,program3a.,Postdoctoral,fellow,3,the,employee,who,earned,Ph.D.,degree,working,under,supervision,of,senior,scientists,and,publishes,results,independently,as,well,as,with,research,team,members3b.,Associated,scientist,3,there,is,a,five,years,period,for,a,research,to,prepare,for,promotion,to,a,scientist.4.,Scientist,3,regularly,publishes,scientific,papers,in,the,national,and,international,journals.5.,Senior,scientist,3,,the,leading,scientific,personality,,usually,,"Doctor,of,Science",or,,“Professor”,,usually,leads,the,research,team,and,is,a,member,of,scientific,councils,,editorial,boards,etc.

The,unit,is,developing,in,accordance,with,goals,set,at,the,last,evaluation,process,in,2010.,In,dependence,on,scientific,outputs,individual,research,departments,are,supported,financially,,in,instrumental,equipment,and,in,human,resources.,Promising,young,researchers,are,stimulated,to,study,stays,at,renowned,scientific,institutions.,International,scientific,collaboration,is,supported,as,well.,

Level,of,inbreeding,in,the,RU

Objectives,and,outcomes,of,the,PhD,programme,(in,particular,mission,of,the,programme,and,career,destination,of,PhD3graduates)

Objectives,are,set,in,cooperation,with,the,university,to,which,the,student,belongs.,The,aim,is,to,fulfill,goals,and,to,show,the,possibility,to,continue,in,the,research,of,the,topic.

Young,researchers,are,involved,in,research,activity,of,the,team,,usually,they,take,part,at,solution,of,goals,of,grant,projects.

Page 125: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Membership,of,the,(world),research,community

International,collaborations,and,partnerships

Department,of,Immunology,and,Immunotoxicology,,Slovak,Medical,University,,Limbová,14,,SK,833,01,Bratislava,,Slovakia

Description'of'collaboration'(150'words)In,EUPHORE,(Valencia,,Spain),one,of,the,major,outdoor,smog,chambers,existing,for,the,study,of,photochemical,processes,that,take,place,in,the,atmosphere,were,proceed,long3lasting,experiments,oriented,to:,1),Definition,of,HONO,sources,(nitrophenols,,vehicle,emissions,,etc)2),Study,of,interferences,(aerosols,,nitrates,,NO2,,etc),within,HONO,measurements3),Intercomparison,of,data,of,techniques,and,19,instruments,for,determination,of,HONO.The,scenario,was,a,set,of,experiments,accounting,for,typical,urban,and,semi3rural,conditions.

C.'International'research'presence'and'collaboration''

C1.'International'mobility

N21.'Close'collaborations'with'institutions'in'other'countries

Institute,for,Nuclear,Sciences,,Ghent,University,,Proeftuinstraat,86,,B39000,,Ghent,,Belgium Prof.,Willy,MaenhautDescription'of'collaboration'(150'words)A,specially,designed,annular,diffusion,denuder,was,developed,for,simultaneous,removal,of,organic,gaseous,compounds,and,atmospheric,oxidants,in,carbonaceous,aerosol,sampling.,The,annular,diffusion,denuder,is,compatible,with,the,collection,of,aerosols,on,47,mm,diameter,quartz,fiber,filter,at,a,flow,rate,of,16.6,L,min31.,The,use,of,this,denuder,enables,one,to,sample,carbonaceous,aerosol,on,filters,without,positive,sampling,artifact,from,volatile,organic,compounds,and,interferences,from,atmospheric,oxidants.,The,annular,diffusion,denuder,has,been,applied,successfully,for,sampling,of,carbonaceous,oxidants,during,field,campaigns,of,typically,1,month,each,at,urban,and,forested,sites,in,Europe.

Most,important,research,organisations,/,close,collaboration,(max,10)

Name'of'partner'organisation Contact'person'in'the'partner'org.

Dr.,Jana,Tulinská,,PhD.Description'of'collaboration'(150'words)Institute,of,Analytical,Chemistry,(IAC),co3operates,with,Slovak,Medical,University,(SMU),in,the,frame,of,QualityNano,projects.,The,first,project,is,focused,on,the,study,of,effect,of,commercially,produced,TiO2,nanoparticles,(NPs),on,human,blood,cells.,IAC,contributed,with,nanoparticle,characterization,and,SMU,performed,in,vitro,immunotoxicity,assays.,Other,four,projects,deal,with,co3operation,at,characterization,of,organs/tissues,of,mice,that,were,exposed,to,different,nanoparticles,continuously,(24,h/day),over,long3term,period.,Inhalation,experiments,carried,out,at,IAC,with,nanoparticles,of,CdO,,PbO,and,Mn3O4,in,special,inhalation,chambers.,IAC,performs,chemical,analyses,of,Cd,or,Pb,content,in,individual,organs/tissues,of,exposed,mice,and,basic,microscope,measurements,of,exposed,tissues,and,SMU,performs,detailed,immunotoxicological,studies.Mutual,database,of,the,results,of,cytotoxicity,,immune,and,hematological,parameters,is,expected.

Name'of'partner'organisation Contact'person'in'the'partner'org.

EUPHORE,,,UMH3Fundación,CEAM,,Valencia,,Spain Dr.,Amalia,Muñoz

Name'of'partner'organisation Contact'person'in'the'partner'org.

Incoming,and,outgoing,study,visits

N22.'Incoming'study'visits

Incoming,study3visits,(max,10,cases)

NONE'PROVIDED

Page 126: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Spain

Spain

Slovakia

Slovakia

Slovakia

Slovakia

Slovakia

Večeřa,Zbyněk 1

Mikuška,Pavel 1

N23.'Outgoing'study'visits

Outgoing,study3visits,(max,10,cases)

Name'of'visiting'res. Total'months'of'visitsFundación,Centro,de,Estudios,Ambientales,del,Mediterráneo,3,CEAM,,Fundación,Centro,de,Estudios,Ambientales,del,Mediterráneo,3,CEAM,,

Institution'of'affiliation Country

Křůmal,Kamil 1

Coufalík,Pavel 1

Mikuška,Pavel 1

Večeřa,Zbyněk 1

Department,of,Immunology,and,Immunotoxicology,,Slovak,Medical,

Department,of,Immunology,and,Immunotoxicology,,Slovak,Medical,

Department,of,Immunology,and,Immunotoxicology,,Slovak,Medical,

Department,of,Immunology,and,Immunotoxicology,,Slovak,Medical,

Čapka,Lukáš 1Department,of,Immunology,and,Immunotoxicology,,Slovak,Medical,

N24.'International'conferences'organised'by'the'RU

Most,important,international,conferences,with,RU,as,the,main,organiser,(maximum,of,5,cases)

C2.'International'co"publications

See,bibliometric,report.

C3.'Reputation'and'esteem

The,usage,of,major,national,or,international,RI,(competitive,access),(N14)The,international,positioning,based,on,funding,data,(N4)Relevant'information'in'other'sections:

Title'of'the'conference Dates' Number'of'participants12th,Annual,Conference,of,the,Czech,Aerosol,Society,,Čejkovice“,,Czech,Republic November,12313,,2009 26

10th,Annual,Conference,of,the,Czech,Aerosol,Society,,Čejkovice,,Czech,Republic, November,334,,2011 31Short'description'(150'words)Twenty,oral,contributions,were,presented,on,the,conference.,The,presentations,dealt,with,various,topics,from,different,aerosol,fields,such,as,identification,of,aerosol,sources,in,urban,flat,,deposition,of,aerosols,on,filter,,concentration,of,particles,emitted,from,traffic,,concentration,and,chemical,composition,of,aerosol,particles,from,different,areas,in,the,Czech,Republic,,chemical,reactions,in,nanoscale,particles,,kinetic,of,nuclei,crystallization,,genotoxicity,of,exhaust,from,different,vehicle,fuels,,generation,of,nanoparticles,etc.

Short'description'(150'words)The,Czech,Aerosol,Society,(CAS),formed,in,1999,associates,researchers,from,various,Czech,Institutions,(such,as,institutes,of,Academy,of,Sciences,etc.),and,Universities,in,the,Czech,Republic,to,promote,collaboration,in,all,areas,of,aerosol,research.,CAS,also,supports,education,in,aerosol,related,fields,at,all,levels,and,supports,international,co3operation.,CAS,organizes,regular,annual,conferences,under,participation,of,native,and,foreign,participants.,The,conferences,are,sponsored,by,local,representatives,of,foreign,companies,supplying,aerosol,instrumentation,(TSI,,Palas).,The,oral,presentations,(19,in,total),were,focused,on,the,concentration,and,chemical,composition,of,aerosol,particles,from,various,locations,in,the,Czech,Republic,,characterization,of,particles,emitted,from,vehicles,,transport,and,deposition,of,aerosols,in,human,respiratory,system,,supersaturation,and,nucleation,of,aerosols,,size,effects,in,nanoscale,particles,etc.

Title'of'the'conference Dates' Number'of'participants

Page 127: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

NONE'PROVIDED

Title'of'the'conference Dates' Number'of'participants

Short'description'(150'words)The,oral,lectures,(25,in,total),presented,on,the,conference,concentrated,on,various,aerosol,themes,,for,example,on,variability,of,aerosols,and,chemical,composition,of,PM,in,various,environment,(metro,,…),,identification,of,emission,sources,of,aerosols,in,Ostrava,,Mladá,Boleslav,and,other,parts,of,the,Czech,Republic,,characterization,of,aerosols,above,motorway,or,in,tunnel,,seasonal,composition,of,aerosols,studied,using,AMS,,SMPS,spectra,in,dependence,on,air,masses,origin,,formation,of,nanoparticles,from,supersaturated,vapor,,production,of,titania,nanoparticles,,resuspension,of,PM,,quality,of,air,and,characterization,of,aerosols,inside,archives,and,museums,,dissociation,of,ammonium,nitrate,aerosol,etc.

N25.'Editorship'in'international'journals

Membership,of,editorial,boards,of,journals,indexed,by,Thomson3ISI,,Scopus,,ERIH,or,Engineering,Village,(max,10)

15th,Annual,Conference,of,the,Czech,Aerosol,Society,,Valtice,,Czech,Republic October,30331,,2014 42

Page 128: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Name'of'centre Focus'of'research Main'project'partners

D.'National'research'presence'and'collaboration'''

D1.''National'collaborations'and'partnerships

N26.'Involvement'in'Centres'or'RI

Participation,in,a,Research,Centre,/,CoE,/,Competence,Centre,or,Research,Infrastructure,(maximum,10,cases)

Relevant'information'in'other'sections:Shared,or,collaborative,use,of,RI,(N15,and,N16)PhDs,trained,in,the,RU,(N33)

Description'of'collaboration'(150'words)Collaboration,in,characterization,of,environmental,systems,,particularly,contaminated,soils,and,sediments,by,diffisuve,gradients,in,thin,film,technique,,in,characterization,of,metals,bioaccessibility,to,crop,plants,employing,pot,experimental,design.

Institute,of,Animal,Physiology,and,Genetics,(IAPG),ASCR,v.v.i. Dr.,Omar,ŠerýDescription'of'collaboration'(150'words)Participation,in,in,vivo,nanoparticles,(NPs),inhalation,studies,(mouse,exposition,studies).,Isolation,of,selected,organs,after,exposure,to,NPs,for,TEM,and,STEM,microscopy,and,an,element,chemical,analysis,of,inhaled,NPs,in,organs).

Name'of'partner'organisation Contact'person'in'the'partner'org.

University,of,Defence,in,Brno Dr.,Josef,KellnerDescription'of'collaboration'(150'words)The,novel,portable,device,for,analysis,of,energetic,materials,has,been,developed.,Proper,function,of,the,device,was,demonstrated,by,analyses,of,standards,,spiked,sample,of,soil,,and,finally,real,samples,of,soil,from,ammunition,disposal,place.,Developed,device,is,suitable,to,provide,fast,and,very,selective,determination,of,inorganic,nitrates,and,organic,nitramines,and,nitroesters,at,trace,concentrations,without,the,need,of,complicated,sample,pre3treatment.,Resistance,to,contamination,and,to,long,term,decommissioning,is,another,benefit,of,the,designed,novel,system.

Name'of'partner'organisation Contact'person'in'the'partner'org.

Mendel,University,in,Brno Prof.Dr.,Hana,Dočekalová,,CSc.

After,preparation,of,defined,engineered,nanoparticles,(NPs),and,characterization,of,NPs,within,inhalation,in,vivo,studies,(inhalation,chamber,,controlled,exposure,to,NPs,,analysis,of,inhaled,chemical,elements,of,NPs,in,target,organs),by,Institute,of,Analytical,Chemistry,,VRI,is,going,to,participate,on,characterization,of,in,vivo,effects,(gene,expression,changes,,membrane,proteins,and,intracellular,signaling,pathways,3lipidomics,,selected,protein,kinases,,immunotoxicity).,

Name'of'partner'organisation Contact'person'in'the'partner'org.

Centre,for,studies,on,toxicity,of,nanoparticles

Nanoparticles,,airborne,nanoparticles,,inhalation,toxicity,,in,vitro,mechanistic,studies,,global,,gene,expression

Veterinary,Research,Institute,,Hudcova,70,,CZ362100,Brno,,Czech,Republic

The,preparation,of,an,instrumentation,for,controlled,nanoparticles,long3term,inhalation,experiments,with,small,animals.,Study,of,deposition,of,inhaled,nanoparticles,in,organs,(chemical,analysis).,The,volume,sampling,in,heavily,polluted,regions,of,ČR,,followed,by,chemical,analysis,of,fine,and,nanoparticles,(metals,,organic,compounds).

Describe'your'role'in'the'Centre'(50'words)'

N27.'National'collaborations'and'partnerships

Most,important,research,organisations,with,which,you,have,established,close,collaboration,(max,10).

Name'of'partner'organisation Contact'person'in'the'partner'org.Veterinary,Research,Institute,(VRI),,Hudcova,62,,621,00,Brno Dr.,Miroslav,MachalaDescription'of'collaboration'(150'words)

Page 129: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Vicepresident G3929432014Ioannes,Marcus,Marci,Spectroscopic,Society Dočekal,Bohumil

Member,of,Examination,Committee,for,Defence,of, H3798532014Member,of,Examination,Committee,for,Defence,of, H3798532014Member,of,Committee,for,Environment,at,AS,CR H3798532014

Academy,of,Sciences,of,the,Czech,Republic

University,of,Defense,in,Brno

Faculty,of,Chemistry,,Technical,University,in,Brno

Večeřa,Zbyněk

Večeřa,Zbyněk

Večeřa,Zbyněk

Most,important,memberships,of,scientific,advisory,boards,in,academia,(max,10,cases)

Type'of'membership ResearcherID'(from'IS)'

Vicepresident H3798532014Czech,Aerosol,Society

Name'of'organisation

Večeřa,Zbyněk

Name'of'the'researcher

N29.'Panel/board'memberships

N28.'Scientific'awards

Scientific,Prizes,Scientific,Prizes,,honours,and,scientific,positions,of,trust,awarded,to,researchers,(max,10,cases)

Name'of'prize'/'position Org.'awarding'prize Name'of'the'researcher ResearcherID'(from'IS)'

Invited,speakerInternational,Conference,on,Chemistry:,Frontiers, Kořínková,Alena 0

Roland,W.,Frei,Award,2012

International,Association,of,Environmental, Gregušová,Michaela 0

Ioannes,Marcus,Marci,a,Kronland,Medal

Ioannes,Marcus,Marci,Spectroscopic,Society Dočekal,Bohumil G3929432014

D2.'National'co"publications

See,bibliometric,report.

D3.'Reputation'and'esteem

Page 130: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

2009 2010 2011 2012 2013 2014

5 4 5 7 5 80 0 0 0 0 02 0 0 0 0 27 7 11 13 9 0

0 0 0 0 0 0

0 0 0 0 0 0

0 1 0 0 0 0

0 0 0 0 0 00 0 0 0 0 00 0 0 0 0 0

Plant,variety,(Zodru)Animal,breed,(Zplem)

See,bibliometric,report,for,more,data.

Book,chapter,(C.)Conference,proceedings,/,Article,in,proceedings,(D)Non"traditional'research'outputsResults,used,by,the,funding,provider,,i.e.,,into,legislation,or,norm,,,into,non3legislative,or,strategic,

Research,report,containing,classified,information,(V)

Patents'and'other'forms'of'IPPatent,(P)

Certified,methodologies,,art,conservation,methodologies,,specialized,map,works,(N)

Research,performance

Scholarly'outputsArticle,in,a,periodical,(J)Monographs,and,books,(B)

E.'Research'productivity''''

E1.'Research'outputs

Page 131: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

2009 2010 2011 2012 2013 20143 3 3 0 0 03 2 3 2 1 06 5 6 2 1 0

2009 2010 2011 2012 2013 201450.0% 60.0% 50.0% 0.0% 0.0% 350.0% 40.0% 50.0% 100.0% 100.0% 3100.0% 100.0% 100.0% 100.0% 100.0% 3

2009 2010 2011 2012 2013 20147 6 6 7 10 96 5 6 2 1 0

0.86 0.83 1.00 0.29 0.10 0.00

2009 2010 2011 2012 2013 20146 5 6 2 1 01 1 3 1 0 0

0.17 0.20 0.50 0.50 0.00 "

PhD,students,who,did,a,substantial,part,of,their,thesis,work,(>50%),at,the,RU,but,enrolled,in,another,Research,Organisation.

2009 2010 2011 2012 2013 20143 3 3 0 0 03 2 3 2 1 06 5 6 2 1 0

2009 2010 2011 2012 2013 201450.0% 60.0% 50.0% 0.0% 0.0% 350.0% 40.0% 50.0% 100.0% 100.0% 3

100.0% 100.0% 100.0% 100.0% 100.0% "

PhD,students,who,did,a,substantial,part,of,their,thesis,work,(>50%),at,the,RU,but,enrolled,in,another,Research,Organisation.

2009 2010 2011 2012 2013 20147 6 6 7 10 96 5 6 2 1 0

0.86 0.83 1.00 0.29 0.10 0.00

Total

Total,researchers,in,the,RU,(HC)Total,PhD,students,enrolled

F.'Ability'to'attract'PhD'students

RU,ability,to,attract,PhD,students

N30.'PhD'student'enrolment

Raw,dataMaleFemaleTotal

Shares,of,the,totalMaleFemale

Raw,dataMale,Female

N31.'PhD'students'versus'total'researchers

Ratio'PhD'students'/'total'researchers

RU,ability,to,train,PhD,students

N32.'PhDs'awarded'in'the'RU

Total,PhD,student,enrolledAwarded,PhDsShare'of'PhDs'enrolled

N33.'PhDs'trained'in'the'RU

Ratio'trained'PhDs'/'researchers

Trained'PhDs

N34.'Level'of'investment'in'PhD'training

Raw,dataNumber,of,researchers,in,the,RU,(FTE)Trained,PhDs

FemaleTrained'PhDs

Shares,of,the,totalMale,

Page 132: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

See,bibliometric,report.

2009 2010 2011 2012 2013 20144.7 4.5 2.9 3.7 4.5 4.11.2 1.9 4.0 5.5 5.7 6.00.0 0.0 0.0 0.0 0.0 0.00.0 0.1 0.0 0.0 0.0 0.00.1 0.3 0.7 0.6 0.1 0.26.1 6.8 7.6 9.8 10.3 10.3

2009 2010 2011 2012 2013 201477.8% 66.2% 37.9% 37.7% 44.1% 39.8%0.0% 0.0% 0.0% 0.0% 0.0% 0.0%0.0% 1.2% 0.1% 0.0% 0.0% 0.0%2.4% 5.0% 9.7% 5.7% 0.7% 2.2%

100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

G.'Overall'quality

G1.'International'citations

Relevant'information'in'other'sections:The,international,positioning,based,on,funding,data,(N4)The,usage,of,major,national,or,international,RI,(competitive,access),(N14)

G2.'Research'outputs'quality'assessment

Total'funding'for'research

Shares,of,the,total

N3.'Overview'funding'sources'"'categories

Raw,Data,(million,CZK)Institutional,funding,for,researchNational,competitive,funding,(‘targeted’,funding)National,funding,from,Structural,Funds

N35.'Value'of'the'research'activities

However,the,main,activity,of,the,RU,was,in,general,a,basic,research,in,past,five,years,(200932014),,the,department,was,also,scientifically,oriented,to,the,development,of,new,methods,and,instrumentation,for,the,chemical,analysis,and/or,characterization,of,environmental,systems,,minor,and,trace,gaseous,components,,nano3particles,and,aerosols,in,the,air,,composition,of,nano3particles,,aquatic,systems,,soils,and,sediments,with,respect,to,organic,and,inorganic,pollutants,and,their,accessibility,to,plants,,food,chains,,health,risk,for,human,beings,etc.,The,used,methodology,was,based,on,separation,and,preconcentration,techniques,in,tandem,combination,with,novel,specific,detectors,,gas,and,liquid,chromatography,,mass,and,atomic,spectrometry,methods.

The,most,important,stones,of,the,former,research:•,the,preparation,of,an,instrumentation,for,controlled,nano3particles,long3term,inhalation,experiments,with,small,animals•,a,study,of,deposition,of,inhaled,nano3particles,in,organs,of,small,animals,(chemical,analysis),•,a,study,of,formation,,transport,and,chemical,composition,of,fine,and,nano3particles,(metals,,organic,compounds).,in,heavily,polluted,regions,in,the,Czech,Republic•,the,development,of,the,novel,portable,device,for,analysis,of,energetic,materials,,at,trace,concentrations,in,environment•,a,study,of,photochemical,processes,of,nitrous,acid,(HONO),that,take,place,in,the,atmosphere.,Sources,,interference,within,HONO,measurements,,intercomparison,of,data,of,techniques,and,instruments,for,determination,of,HONO

Results,were,predominantly,published,in,international,journals.,Some,of,further,results,stemmed,from,needs,of,the,Czech,society,the,industry,,a,ministry,of,education,for,example.

Major,area,of,value,in,research,and,development,of,the,activities,in,RU,from,a,scientific,or,socio3economic,perspective.,

A2.'Ability'to'attract'external'funding'(RU)

Overview

Total'funding'for'research

Competitive,international,funding

Institutional,funding,for,researchNational,funding,from,Structural,FundsCompetitive,international,funding

Funding,from,contract,research

Funding,from,contract,research

Page 133: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

International'positioning

2009 2010 2011 2012 2013 20140.00 0.00 0.00 0.00 0.00 0.000.00 0.09 0.01 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.00

0.00 0.09 0.01 0.00 0.00 0.00

2009 2010 2011 2012 2013 20143 0.0% 0.0% 3 3 33 100.0% 100.0% 3 3 33 0.0% 0.0% 3 3 3

" 100.0% 100.0% " " "0.0% 1.2% 0.1% 0.0% 0.0% 0.0%

Relevance'for'the'public'sector

2009 2010 2011 2012 2013 2014

1.2 1.5 0.0 0.0 0.0 0.0

0.0 0.0 2.5 1.3 2.0 2.00.0 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.6 0.5 0.0 0.1

1.2 1.8 3.2 1.8 2.0 2.1

2009 2010 2011 2012 2013 2014

100% 85% 0% 0% 0% 0%

0% 0% 80% 72% 100% 94%0% 0% 0% 0% 0% 0%

0% 15% 20% 28% 0% 6%

100% 100% 100% 100% 100% 100%20% 27% 42% 19% 19% 21%

Relevance'for'industry

2009 2010 2011 2012 2013 20140.00 0.00 0.00 0.00 0.00 0.000.15 0.06 0.10 0.04 0.08 0.100.15 0.06 0.10 0.04 0.08 0.10

2009 2010 2011 2012 2013 20140.0% 0.0% 0.0% 0.0% 0.0% 0.0%

100.0% 100.0% 100.0% 100.0% 100.0% 100.0%100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

2.4% 0.9% 1.2% 0.4% 0.7% 1.0%

OP,RDI,3,Regional,R&D,centres

Shares,of,the,totalCompetitive,funding,3Other,state,resources,–,Ministry,of,EnvironmentCompetitive,funding,3Other,state,resources,–,Ministry,of,CultureContract,research,from,ministriesContract,research,from,other,public,sector,institutions,(in,the,Czech,Republic)Total'competitive'&'contract'funding'from'public'sector'(sectoral'ministries)Share,of,total,funding,for,research

N6.'Collaboration'with'industry'"'competitive'&'contract'research

Raw,Data

Contract,research,from,industryTotal'funding'from/for'cooperation'with'industry

Shares,of,the,totalOP,RDI,3,Regional,R&D,centres

Contract,research,from,international,public,bodiesTotal'international'funding'"'competitive'&'contract'research

Shares,of,the,total

Competitive,funding,3Other,state,resources,–,Ministry,of,EnvironmentCompetitive,funding,3Other,state,resources,–,Ministry,of,CultureContract,research,from,ministries

European,Funding

Contract,research,from,other,public,sector,institutions,(in,the,Czech,Republic)

Total'funding'from/for'cooperation'with'industryShare,of,total,funding,for,research

Activity,profile,of,the,RU,based,on,funding,sources

N4.'International'funding'"'competitive'&'contract'research

Raw,Data,(million,CZK)

Contract,research,from,industry

Total'competitive'&'contract'funding'from'public'sector'(sectoral'ministries)

Funding,from,other,foreign,public,sourcesContract,research,from,international,public,bodiesTotal'international'funding'"'competitive'&'contract'researchShare,of,total,funding,for,research

N5.'Funding'from'public'sector'(sectoral'ministries)'"'competitive'&'contract'research

Raw,Data,(million,CZK)

European,FundingFunding,from,other,foreign,public,sources

Page 134: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

National'funding

2009 2010 2011 2012 2013 20140.00 0.34 1.46 4.20 3.68 3.970.00 0.00 0.00 0.00 0.00 0.001.21 1.55 0.00 0.00 0.00 0.000.00 0.00 2.54 1.33 2.00 2.021.2 1.9 4.0 5.5 5.7 6.0

2009 2010 2011 2012 2013 20140.0% 18.0% 36.4% 76.0% 64.8% 66.3%0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

100.0% 82.0% 0.0% 0.0% 0.0% 0.0%0.0% 0.0% 63.6% 24.0% 35.2% 33.7%

100.0% 100.0% 100.0% 100.0% 100.0% 100.0%19.8% 27.6% 52.3% 56.6% 55.2% 58.0%

2009 2010 2011 2012 2013 20140.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.0

2009 2010 2011 2012 2013 20143 3 3 3 3 33 3 3 3 3 33 3 3 3 3 33 3 3 3 3 33 3 3 3 3 3

0.0% 0.0% 0.0% 0.0% 0.0% 0.0%TotalShare,of,total,funding,for,research

Shares,of,the,totalOP,RDI,3,European,Centres,of,ExcellenceOP,RDI,3,Regional,R&D,centresOP,Prague,Competitiveness,(OPPC),Other,–,OP,1

OP,RDI,3,Regional,R&D,centresOP,Prague,Competitiveness,(OPPC),Other,–,OP,1Total

TotalShare,of,total,funding,for,research

National'funding'from'structural'funds

Raw,DataOP,RDI,3,European,Centres,of,Excellence

Science,FoundationTechnology,AgencyOther,state,resources,–,Ministry,of,EnvironmentOther,state,resources,–,Ministry,of,Culture

Other,state,resources,–,Ministry,of,EnvironmentOther,state,resources,–,Ministry,of,CultureTotal

Shares,of,the,total

Technology,Agency

National'competitive'funding'(‘targeted’'funding)

Raw,DataScience,Foundation

Page 135: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Research,excellence

H.'Research'excellence''''

H1.'Peak'quality

Referees,–,review,of,submitted,research,outputs

Bibliometrics

Page 136: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

NONE'RECORDED

NONE'RECORDED

NONE'RECORDED

Societal,relevance

I.'Knowledge'&'technology'transfer'activities

I1.'Income'from'societal'relevant'research

Relevant'information'in'other'sections:

Name'of'Institution Type'of'organisation Contact'person'in'the'partner'org.

Funding,from,public,sector,(sectoral,ministries),,(N5)Collaboration,with,industry,3,competitive,&,contract,research,(N6)

N36.''Income'from'the'commercialisation'of'research'outputs

Major,area,of,value,in,research,and,development,of,the,activities,in,RU,from,a,scientific,or,socio3economic,perspective.,

N37.'Non"academic'collaborations

Most,important,non3academic,collaboration,with,societal,actors,(max,10).

National,collaborations,and,partnerships,(N28)Value,of,the,research,activities,(N36)

I2.'Collaboration'with'other'societal'actors

Relevant'information'in'other'sections:Shared,or,collaborative,use,of,RI,(N16,and,N17)Involvement,in,Centres,or,RI,(N27)

RS,DYNAMICS,s.r.o.,,Starochodovská,1359/76,,Chodov,,CZ314900,Prgue,,Czech,Republic small,company RNDr.,Jiří,BláhaDescription'of'collaboration'(150'words)Development,of,advanced,portable,system,for,explosive,agent,sampling,,detection,and,analysis.,A,new,technology,for,trace,explosives,detection,applied,into,hand3held,portable,instrument,with,extreme,sensitivity,(picogram,level),,very,low,false,rate,and,excellent,resistance,against,interferents.,

N38.'Participation'in'incubators'and'clusters

Participation,in,incubators,or,clusters,3,maximum,of,10,different,cases.

I2.'National'research"industry'collaboration

I3.'Non"traditional'scholarly'research'&'IPR'outputs

See,bibliometric,report

N39.'Membership'of'advisory'boards'outside'academia

Most,important,memberships,of,scientific,advisory,boards,outside,academia,(max,10,cases)

I3.'Reputation'and'esteem'beyond'academia

I4.'Income'from'commercialisation'activities

Page 137: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

N40.'Societal'impact

I5.'Societal'impact'narratives

The,economic,,social,,cultural,,policy,and,other,impacts,of,the,work,conducted,at,the,Research,Unit,on,Czech,society.

The,research,carried,out,within,the,RU,as,well,as,in,the,frame,of,various,co3operations,had,societal,impact,on,our,society,,especially,in:,economic,area,,public,policy,and,service,,health,and,welfare,,and,the,environment.,The,orientation,to,development,of,new,methods,and,instrumentation,for,the,chemical,analysis,was,fruitful,for,the,business,and,public,policy,sectors,when,the,co3operation,with,small,company,such,as,RS,DYNAMICS,s.r.o.,,Prague,induced,a,production,of,the,portable,systems,for,explosive,agent,sampling,,detection,and,analysis.,The,partnership,with,University,of,Defense,,Brno,resulted,in,the,novel,portable,device,for,the,analysis,of,energetic,materials,,nowadays,prepared,for,practical,use,in,security,and/or,army,areas.,The,results,acquired,within,frame,of,inhalation,projects,of,small,animals,and,sampling,of,air,in,heavily,polluted,regions,of,the,Czech,Republic,,followed,by,chemical,analysis,of,fine,particles,and,nano3particles,stimulated,a,public,debate,concerning,the,health,risks,of,nano3particles,and,fine,particles,from,the,point,of,quality,of,life,of,human,population.,International,co3operation,with,partners,in,Spain,and,in,Belgium,illuminated,some,questions,arising,fromthe,presence,of,atmospheric,oxidants,in,typical,urban,and,semi3rural,air,in,the,Czech,Republic.

Page 138: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Please&provide&the&name&and&contact&details&of&a&person&at&the&Research&Unit.

Name%&%Surname Doc.&PhDr.&Csaba&Szaló,&Ph.D.Telephone%number 549&49&6854E2mail%address [email protected]

Telephone%number 549&49&7555E2mail%address [email protected]

Type%of%the%Research%Organisation

Research&Unit&1

Research%Unit%Field

5.&Social&Sciences&F&5.4&Sociology

Contact%person%for%the%Research%Unit

Scientific&Research&Organisation

viceFrector&for&academic&affairs,&Masaryk&University;&department&head,&Office&for&Population&StudiesTelephone%number 549&49&1037E2mail%address [email protected]

Contact%person%for%the%Evaluated%Unit

Name%&%Surname Ing.&Alena&RaisováPosition coordinator&for&project&support,&Office&for&Research&&&Development&and&Doctoral&Study&Programmes

Panel%Data%Report

Name%&%Surname prof.&PhDr.&Ladislav&Rabušic,&CSc.

Name%of%the%EvU: Masaryk%University,%Faculty%of%Social%Studies

ID%of%the%EvU: orjk:14230

Head%of%the%internal%evaluation%committee

Position

Page 139: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

2009 2010 2011 2012 2013 201437.0 35.3 28.6 32.3 30.3 27.8

174.5 173.4 200.4 216.0 269.9 234.40.0 0.0 0.0 0.0 0.0 0.0

2009 2010 2011 2012 2013 2014

0.0 0.0 0.0 0.0 0.0 0.0

16% 16% 16% 29% 23% 21%

(in&million&CZK)

2009 2010 2011 2012 2013 20143.1 2.4 8.0 11.5 12.2 12.60.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 1.9 2.5 3.30.6 0.2 0.0 0.0 0.0 0.03.6 2.6 8.0 13.4 14.7 15.9

2009 2010 2011 2012 2013 20140.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.9 0.70.0 0.0 0.0 0.0 0.9 0.7

2009 2010 2011 2012 2013 20140 0 0 0 7.44 5.64

0.14 0 0 0 0 00.14 0 0 0 7.44 5.64

2009 2010 2011 2012 2013 20140.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 0.00 0.00 0.00

Contract&research&from&industryContract&research&from&ministriesContract&research&from&other&public&sector&institutions&(in&the&Czech&Republic)Contract&research&from&international&public&bodiesOther&contract&researchTotal

N5.%Competitive%international%funding

N6.%Funding%from%contract%research

A1.%Institutional%funding

N1.%Institutional%funding%2%EvU

Total&amount&of&institutional&funding&for&research&received&by&the&Evaluated&Unit&(in&million&CZK).

Institutional&funding&for&research

Background&information

A2.%Ability%to%attract%external%funding%(RU)

N3.%National%Competitive%Funding

Science&FoundationTechnology&AgencyMinistry&of&CultureMinistry&of&Education,&Youth&and&SportsTotal

N4.%National%funding%from%Structural%Funds

European&FundingFunding&from&other&foreign&public&sourcesTotal

Other&institutional&funding&Total%institutional%funding%EvU

Share&of&EvU&institutional&funding&for&research

N2.%Institutional%funding%for%research%2%RU

Total&amount&of&institutional&funding&for&research&received&by&the&Evaluated&Unit&(in&million&CZK).

Institutional&funding&for&research&(estimate)

OP&RDI&F&European&Centres&of&ExcellenceOP&RDI&F&Regional&R&D&centresOP&Prague&Competitiveness&(OPPC)Other&–&OP&1Total

Page 140: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Headcounts%

2009 2010 2011 2012 2013 201498 97 95 115 130 13240 55 52 54 68 69

138 152 147 169 198 201

2009 2010 2011 2012 2013 201471.0% 63.8% 64.6% 68.0% 65.7% 65.7%29.0% 36.2% 35.4% 32.0% 34.3% 34.3%100% 100% 100% 100% 100% 100%

FTE

2009 2010 2011 2012 2013 201498.0 88.3 90.5 99.7 106.7 113.036.0 35.8 38.7 41.3 52.7 53.8

134.0 124.1 129.2 141.0 159.4 166.8

2009 2010 2011 2012 2013 201473.1% 71.1% 70.0% 70.7% 66.9% 67.7%26.9% 28.9% 30.0% 29.3% 33.1% 32.3%100% 100% 100% 100% 100% 100%

Universities&and&university&hospitals

Raw&data 2009 2010 2011 2012 2013 201413.0 11.8 12.7 12.7 12.3 11.017.0 19.7 18.8 21.6 19.4 21.041.0 41.0 44.0 47.3 56.6 62.018.0 7.5 6.5 10.4 9.0 9.09.0 8.3 8.5 7.7 9.4 10.036.0 35.8 38.7 41.3 52.7 53.8

134.0 124.1 129.2 141.0 159.4 166.8

2009 2010 2011 2012 2013 201410% 9% 10% 9% 8% 7%13% 16% 15% 15% 12% 13%31% 33% 34% 34% 36% 37%13% 6% 5% 7% 6% 5%7% 7% 7% 5% 6% 6%

27% 29% 30% 29% 33% 32%100% 100% 100% 100% 100% 100%

A3.%Research%capacity

Research&capacity&in&the&EvU

N7.%R&D%personnel%in%the%EvU%2%Overview

Raw&dataTotal&research&staff

Total%overall

Shares&of&the&totalTotal&research&staff

Institutional&management&and&development&potential

A.%Quality%adequacy%of%the%research%environment

Total&support&staff&(i.e.&scientific/technical/admin)Total%overall

Associate&professorAssistant&professor&Assistant&Lecturer&Scientific&staff&Total

Total&support&staff&(i.e.&scientific/technical/admin)

N8.%R&D%personnel%in%the%EvU%2%Detailed%(FTE)

Shares&of&the&totalTotal&research&staffTotal&support&staff&(i.e.&scientific/technical/admin)Total%overall

Total&support&staff&(i.e.&scientific/technical/admin)Total&research&staff

Raw&data

Total%overall

Professor&Associate&professorAssistant&professor&Assistant&Lecturer&Scientific&staff&Total

Shares&of&the&totalProfessor&

Page 141: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Headcounts

2009 2010 2011 2012 2013 201435 34 31 35 43 41

138 152 147 169 198 20125.4% 22.4% 21.1% 20.7% 21.7% 20.1%

FTE

2009 2010 2011 2012 2013 201429 28 29 30 33 32

134 124 129 141 159 16722% 23% 23% 21% 21% 19%

2009 2010 2011 2012 2013 201421.7% 22.7% 22.8% 21.5% 21.0% 19.1%78.3% 77.3% 77.2% 78.5% 79.0% 80.9%21.7% 22.7% 22.8% 21.5% 21.0% 19.1%

Ratio&FTE&/&HC

2009 2010 2011 2012 2013 201483.1% 82.9% 95.0% 86.6% 77.7% 78.8%97.1% 81.6% 87.9% 83.4% 80.5% 83.0%

Key&pieces&of&experimental&equipment,&scholarly&resources&such&as&libraries&or&databases.&Maximum&of&10&different&cases.

Raw&dataResearchers&in&the&RUResearchers&in&the&EvURU&share&of&total&researchers&in&the&EvU

Shares&of&the&total

N10.%Researchers%in%the%RU

Raw&dataResearchers&in&the&RUResearchers&in&the&EvURU&share&of&total&researchers&in&the&EvU

In&the&RUIn&the&EvU

A4.%Infrastructure%for%research%at%the%RU

Research&infrastructure&(description&&&assessment)

N12.%Key%experimental%equipment

Researchers&in&the&RUResearchers&in&the&EvURU&share&of&total&researchers&in&the&EvU

N11.%Level%of%researchers'%full2time%employment%

The&EvU&systematically&builded&a&high&quality&social&science&library&from&its&establishment.&Recently&the&library&collection&contains&all&basic&internationally&relevant&social&science&literature,&books&as&well&as&journals.&The&library&provides&important&services&related&to&buying&books,&providing&electronic&reseources&or&interlibrary&loan.

Name%of%the%equipment Is%the%equipment%or%resource%older%than%10%years?%

Sage&Journals&Online 0

Name%of%the%equipment Is%the%equipment%or%resource%older%than%10%years?%

Library yesDescription%of%equipment%or%resource%(50%words)%

Description%of%equipment%or%resource%(50%words)%

This&is&just&one&of&the&softwares&for&qualitative&data&analysis&used&in&social&science&research&provided&by&the&RU&to&its&researchers.

Description%of%equipment%or%resource%(50%words)%

This&is&just&one&of&the&several&electronic&resources&that&are&provided&by&the&EvU.&The&access&to&recent&international&publications&is&crucial&for&conceiving&relevant&research&topics,&preparing&up&to&date&projects&and&performing&high&quality&research.

Name%of%the%equipment Is%the%equipment%or%resource%older%than%10%years?%

Atlas.ti 0

Page 142: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Details&of&the&type&of&infrastructure&and&purpose&F&no&more&than&10&different&cases

Shared&or&collaborative&use&of&scholarly&resources&such&as&libraries&or&databases&F&maximum&of&10&different&cases.

Usage&of&research&infrastructures&of&other&Czech&research&organisations&F&max&10&cases

Description%of%equipment%or%resource%(50%words)%

This&is&just&one&of&the&softwares&for&quantitative&data&analysis&used&in&social&science&research&provided&by&the&RU&to&its&researchers.

Name%of%the%equipment Is%the%equipment%or%resource%older%than%10%years?%

IBM&SPSS&Statistics 0

Name%of%the%equipment Who%did%you%share%the%equipment%with?

Library colleagues,&studentsDescription%of%equipment%or%resource%(50%words)%

Shared&or&collaborative&use&of&RI

N15.%Shared%or%collaborative%use%of%RU%equipment%with%other%research%organisations

N13.%Adequacy%of%the%research%infrastructure%(self2assessment)

Both&facilities&and&the&infrastructure&of&research&is&adequate&for&performing&research&the&RU.&The&library,&through&its&up&to&date&collection&of&international&social&science&books&and&journals,&as&well&as&the&instant&access&to&electronic&journals&and&books&via&global&electronic&publication&databases&generates&inevitable&basis&for&preparing&and&performing&research&at&internationally&relevant&level.

N14.%Usage%of%major%national%or%international%RI%(competitive%access)

Usage&of&national/international&RI&awarded&(based&on&competition)

NONE%PROVIDED

Name%of%the%equipment Who%did%you%share%the%equipment%with?

Atlas.ti colleagues,&students

Resources&available&in&our&library&serve&as&a&ground&for&conceiving&any&kind&of&workshops&dealing&with&project&preparation&or&interpretive&analysis&in&the&research&process.

Name%of%the%equipment Who%did%you%share%the%equipment%with?

Sage&Journals&Online colleagues,&studentsDescription%of%equipment%or%resource%(50%words)%

Resources&available&in&our&electronic&resources&serve&as&a&ground&for&conceiving&any&kind&of&workshops&dealing&with&project&preparation&or&interpretive&analysis&in&the&research&process.

N16.%Use%of%RI%in%other%Czech%research%organisations%(non2competitive)

NONE%PROVIDED

Description%of%equipment%or%resource%(50%words)%

Beside&direct&collaboration&it&is&shares&via&the&institutional&licence&of&this&software.

Description%of%equipment%or%resource%(50%words)%

This&software&enables&direct&cooperation&with&other&researcher&mainly&in&interpretive&phase&of&the&research.

Name%of%the%equipment Who%did%you%share%the%equipment%with?

IBM&SPSS&Statistics colleagues,&students

Page 143: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Frequency&of&performance&review&/&Frequency&employees'&feedback&on&potential&concerns,&issues,&challenges&related&to&their&jobs

B1.%Research%strategy

N17.%Description%of%the%research%strategy%for%the%RU

B.%Research%strategy%and%(HR)%management%

B2.%Human%resource%development

The&research&strategy&of&the&Research&Unit&is&in&line&with&the&long&term&strategy&of&the&Faculty&which&follows&the&logic&that&aims&to&institutionalize&research&and&concentrate&human&resources&on&specific&fields&while&at&the&same&time&gives&chance&to&other&streams&of&research&as&a&part&of&academic&Departments&as&well.&In&this&sense,&insitutional&funding&for&research&that&is&received&by&the&RU&is&fully&concentrated&in&a&relatively&newly&established&(2012)&research&centre,&that&is&the&Office&for&Population&Studies&(UPS).&UPS&integrates&cuttingFedge&interdisciplinary&knowledge&from&the&fields&of&sociology&and&demography,&and&draws&on&their&diverse&viewpoints&in&applied&studies&on&population&affairs&and&processes.&Methodologically,&UPS&emphasizes&hypothesis&and&theory&testing&based&on&empirical&data&analysis,&including&UPS&primary&research&data,&aggregate&statistical&data&and&data&from&international&databases.&Nevertheless&in&its&research&work&the&RU&including&the&UPS&applies&a&wide&range&of&appropriate&methodological&approaches&including&qualitative,&quantitative&and&mixedFmethods&techniques.&The&core&research&topics&the&UPS&deals&with&are&the&following:&(a)&Changes&in&reproductive&behaviour&and&their&effects,&with&particular&focus&on&biological&and&social&factor&interactions&in&reproductive&behaviour.&(b)&Population&ageing&and&its&impacts&on&the&future&development&of&society.&(c)&International&migration&and&its&impact&on&labour&market&and&national&identities.&(c)&Anthropological&and&sociological&study&of&various&aspects&of&human&sexuality.&(d)&Changes&in&value&structures&F&as&the&Czech&partner&in&the&European&Values&Study&longFterm&international&research.&(e)&Study&of&interconnections&between&demographic&and&stratification&processes.&There&are&other&streams&of&research&in&the&RU&which&are&also&play&a&crucial&role&in&the&research&strategy&of&the&RU.&These&research&topics&can&be&located&in&the&intersection&of&Cultural&Sociology,&Urban&Studies&and&Migration&Studies.&This&part&of&our&strategy&build&on&research&experiences&of&RU&members&who&participated&in&several&projects&funded&by&the&Czech&Science&Foundation&(e.g.&"Collective&Memory&and&the&Transformation&of&Urban&Space")&or&funded&internationally&like&the&7th&Framework&Program&of&the&European&Commission&(e.g.&"Ethnic&Difference&in&Education&and&Diverging&Prospects&for&Urban&Youth&in&an&Enlarged&Europe&F&EDUMIGROM").&In&this&ways&the&research&potential&of&those&RU&members&who&cannot&work&primarily&as&researchers&but&focus&on&teaching&at&the&graduate&and&postFgradute&study&program&of&Sociology,&including&graduate&study&profiles&in&Social&Anthropology,&Genders&Studies&and&Urban&Studies&can&be&included&into&the&long&term&development&of&the&RU.

N18.%Processes%for%career%development%in%the%EvU

Appraisal&and&monitoring&system&/&Competency&framework&for&different&job&roles

The&performance&of&individual&researchers&is&reviewed&one&a&year&by&means&of&the&above&mentioned&evaulation&interview.&Employees&provide&feedback&minimally&related&to&their&jobs&once&a&year.

There&is&an&appraisal&and&monitoring&system&both&at&the&level&of&Evaluated&Unit&and&the&Research&Unit.&The&core&of&this&system&at&he&EvU&level&consist&in&regular&meetings&held&in&the&beginning&of&the&year&in&which&the&last&year's&performance&of&particular&Departments&and&Research&Institutions&is&evaluated.&This&evaluation&meeting&is&based&on&statistical&data&that&are&available&for&both&evaluators&(the&Dean&and&Vice&Deans&of&the&Faculty&as&well&as&the&Bursary)&and&the&representatives&i.e.&Heads&of&the&Departments&and&Research&Institutions.&This&meeting&has&a&clear&agenda&dealing&with&educational,&scientific,&administrative&and&economic&performace&of&the&particular&Departments&and&Research&Institutions.&Discussions&over&particular&agendas&usually&involve&an&assesment&of&possible&tracks&of&development&for&particular&Departments&and&Research&Institutions.&The&appraisal&and&monitoring&system&at&the&level&of&Research&Units&is&centred&on&evaluation&interviews&held&by&individual&researchers&regularly&at&the&begining&of&year.&Individual&researcher&have&to&fill&out&an&online&evaluation&form&that&then&enables&Heads&of&the&RU&to&base&the&evaluation&interview&on&data&about&the&educational,&scientific&and&administrative&preformance&of&individuals&in&the&last&five&years.&The&agenda&of&these&interview&also&contains&the&debate&about&the&future&plans&and&individual&academic&prospects&of&individual&researchers.&All&types&of&academic&positions&at&the&university&has&a&separate&outline&of&competence&requirements&and&responibilities.

Career&development&policy&in&the&EvU

Page 144: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Promotion&criteria&/&Individual&targets&/&Other&ways&to&tackle&the&career&development&of&employees

Approaches&for&EvU&to&develop&and&maintain&structures&and&practices&that&foster&good&research

Approach&by&EvU&to&help&earlyFcareer&researchers&making&their&way&into&the&profession

Policy&and&practice&concerning&PhD&education&and&the&role&of&postFdocs

N19.%Career%development%of%PhDs%and%post2docs

The&promotion&criteria&are&clearly&linked&to&academic&performance&of&individuals,&that&is&achieving&Ph.D.,&habilitation&and&full&professorship.&Individual&targets&are&conceived&in&terms&of&these&academic&titles.&Usually&it&means&to&plan&4F7&years&for&finishing&the&Ph.D.&degree,&then&7F9&years&to&attain&habilitation,&accopmlishing&the&full&professorship&much&more&depends&on&individual&life&strategies,&so&there&are&usually&no&target&dates&set.&Research&Units&offer&to&their&researchers&a&scheme&of&financial&incentives&which&serves&as&a&motivating&force&for&(a)&publishing,&(b)&supervising&doctoral&students&in&the&Ph.D.&study&program,&and&(c)&actively&participating&in&international&conferences.&All&these&supported&activities&contribute&also&to&the&individual&career&development&of&researchers.

The&main&strategy&of&the&EvU&in&this&respect&is&the&establishment&of&separate&institutions&focusing&on&particular&fields&of&research.&Already&in&the&second&half&of&the&1990's&there&were&three&research&streams&established&at&the&EvU&(Sociology&and&Social&Policy,&Psychology,&and&Political&Science),&nevertheless&these&were&administratively&linked&to&academic&Departments.&These&research&streams&were&in&the&beginning&of&2000's&reorganized&into&Research&Institutions&(the&Insitute&for&Social&Reproduction&and&Integration,&the&Institute&for&Comparative&Political&Research,&and&the&Institute&for&Research&on&Children,&Youth&and&Family).&As&far&as&the&state&funding&of&these&Research&Institutions&were&finished&in&2011,&they&were&reorganized&into&four&Research&Institutions&funded&on&funds&the&University&receives&for&its&research&performance.&These&recent&institutions&are:&The&Office&for&Population&Studies;&The&International&Institute&for&Political&Science;&The&Institute&for&Research&on&Children,&Youth&and&Family;&and&The&Insitute&for&Public&Policy&and&Social&Work.&Beside&these&research&institutes&there&is&still&substantial&proportion&of&research&activities&conducted&at&the&level&of&academic&Departments,&nevertheless&this&mode&of&division&of&labour&enables&to&concentrate&resources&to&achieve&excellence&in&particular&fields.

Objectives&and&outcomes&of&the&PhD&programme&(in&particular&mission&of&the&programme&and&career&destination&of&PhDFgraduates)

There&are&two&main&objectives&of&the&Sociology&Ph.D.&programme:&(a)&to&develop&Ph.D.&candidates'&research&skills&and&their&ability&to&carry&out&a&whole&research&process&from&conceiving&a&project&to&writing&an&article&or&a&book;&(b)&to&form&Ph.D.&candidates&teaching&skills&to&be&able&to&lead&a&seminars&and&workshops.&The&primary&outcome&of&the&Ph.D.&programme&is&a&dissertation,&as&well&as&two&academic&papers&submitted&to&publication.&Our&Ph.D.&candidates&regularly&work&as&teaching&assitants&both&in&undergraduate&and&graduate&courses.&The&mission&of&the&Ph.D.&programme&is&clearly&academic,&to&prepare&sociologists&who&will&work&in&academic&institutions&as&lecturers/assitant&professors&as&well&as&researchers.&In&general&our&Ph.D.&programme&by&forming&analytical&and&research&skills&of&Ph.D.&candidates&as&well&as&their&ability&to&cooperate&with&other&colleagues&enables&them&to&participate&in&institutional&processes&of&problem&solving&and&project&based&management&forms&in&a&wider&context&of&career&prospects.

For&years&the&main&mode&of&support&for&early&career&researcher&was&to&enable&them&to&participace&in&the&Czech&Science&Foundation's&postFdoctoral&funding&scheme&by&an&institutional&guarantee&in&application&process.&Recent&form&of&junior&scholar&scheme&does&not&make&possible&to&follow&these&form&of&support.&The&university&provided&for&the&last&years&and&EU&based&funding&for&postFdoctoral&fellowships&which&were&aimed&at&recruiting&young&researchers.&However,&this&support&scheme&ends&next&year&and&recently&there&is&no&insitutionalized&form&of&support&at&level&of&EvU&for&helping&excellent&doctoral&students&after&finishing&their&degree&which&would&enable&them&to&make&their&way&into&their&profession&as&postFdoctoral&researchers.&Typically&this&kind&of&early&carreer&support&occurs&at&the&level&RU's&human&resources&policy&and&have&a&form&of&offering&part&time&jobs&to&these&junior&researchers.

Page 145: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Supervision:&rights&and&obligations&of&both&supervisors&and&PhD&candidates

Educational&components&for&the&training&of&PhDs

Headcounts

2009 2010 2011 2012 2013 201435 34 31 35 43 412 1 2 5 5 5

6% 3% 6% 14% 12% 12%

FTE

2009 2010 2011 2012 2013 201429 28 29 30 33 322 1 2 5 5 5

5% 4% 7% 15% 13% 14%

Total&researchers&in&the&RU&Total&former&PhD&students

It&is&the&Board&of&doctoral&studies&which&decides&about&the&acceptance&of&a&Ph.D.&candidate&on&the&basis&of&her&research&proposal.&Supervisors&have&a&right&to&consent&in&respect&to&their&supervision&of&individual&Ph.D.&candidates'&research&project.&The&Board&of&doctoral&studies&evaluates&every&academic&year&individual&Ph.D.&candidate's&academic&performance&on&the&ground&of&their&supervisor's&reports.&Supervisors&consult&the&research&work&as&well&as&dissertation&writing&of&her&Ph.D.&candidates&and&every&term&they&are&supposed&to&give&them&a&feedback&also&by&issuing&course&credits&for&their&academic&performance.&Ph.D.&candidates&have&right&to&appeal&to&a&Board&of&doctoral&studies&in&the&case&they&would&like&to&change&their&supervisor.&They&are&obliged&to&obtain&at&least&20&ECTS&credits&for&their&academic&performance&mainly&related&to&their&research&work,&including&writing&up&their&dissertation&in&the&last&terms&of&their&studies.

All&courses&Ph.D.&candidates&have&to&go&through&are&related&to&their&research&activities&and&dissertation&writing.&These&are&usually&courses&having&a&form&of&several&workhops&shared&by&other&Ph.D.&candidates&from&the&RU,&aimed&at&developing&their&dissertation&project,&giving&a&methodological&feedback&to&their&ongoing&research&or&developing&their&academic&writing&skills.

Share&of&total&researchers

Total&researchers&in&the&RU&Total&former&PhD&studentsShare&of&total&researchers

N20.%Employment%of%former%PhD%students%in%the%RU

Level&of&inbreeding&in&the&RU

Page 146: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

USA

UK

Germany

USA

USA

USA

Japan

C.%International%research%presence%and%collaboration%%

C1.%International%mobility

N21.%Close%collaborations%with%institutions%in%other%countries

Membership&of&the&(world)&research&community

International&collaborations&and&partnerships

Name%of%partner%organisation Contact%person%in%the%partner%org.

Center&for&Cultural&Sociology,&Yale&University

Description%of%collaboration%(150%words)

Most&important&research&organisations&/&close&collaboration&(max&10)

Name%of%Visitor Institution%of%affiliation Country% Total%months%of%visits

Incoming&and&outgoing&study&visits

N22.%Incoming%study%visits

The&European&Values&Study&is&a&largeFscale,&crossFnational,&and&longitudinal&survey&research&program&on&basic&human&values.&It&provides&insights&into&the&ideas,&beliefs,&preferences,&attitudes,&values&and&opinions&of&citizens&all&over&Europe.&It&is&a&unique&research&project&on&how&Europeans&think&about&life,&family,&work,&religion,&politics&and&society.&Ladislav&Rabusic&is&a&member&of&the&Board&of&Program&Directors.&Our&RU&is&the&main&partner&of&the&project&&for&the&Czech&Republic.

Prof.&Jeffrey&C.&AlexanderDescription%of%collaboration%(150%words)

There&is&a&long&term&collaboration&in&the&field&of&memory&studies.&In&our&RU&the&graduate&study&program&in&Cultural&Sociology&provides&a&background&for&developing&shared&research&interest&and&projects.&There&are&two&postFdoctoral&researchers&recently&in&the&RU&who&finished&their&degree&in&the&CCS&at&the&Yale&University.

Name%of%partner%organisation Contact%person%in%the%partner%org.

Tilburg&University Dr.&Loek&Halman

Patricia&H&Dyk University&of&Kentucky USA 3

Marelize&MarxUniversity&of&Port&Elizabeth South&Africa 6

Incoming&studyFvisits&(max&10&cases)

Radim&Marada 10

Naday&Jaworsky 3

N23.%Outgoing%study%visits

Outgoing&studyFvisits&(max&10&cases)

Name%of%visiting%res. Total%months%of%visits

CCS&Yale&University

UCL&London

Institution%of%affiliation Country

Kateřina&Lišková 12

Kateřina&Lišková 3

Dominik&Bartmanski 5

Kateřina&Nedbálková 4

New&York&University

Columbia&University

UC&Berkeley

Technische&Universtat&Berlin

Tomáš&Katrňák 6Tohuku&University,&Sendai

Page 147: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

C2.%International%co2publications

See&bibliometric&report.

C3.%Reputation%and%esteem

The&usage&of&major&national&or&international&RI&(competitive&access)&(N14)The&international&positioning&based&on&funding&data&(N4)Relevant%information%in%other%sections:

Short%description%(150%words)

This&conference&aimed&to&highlight&the&challenges&we&face&in&educating&young&people&about&sex&and&the&impact&that&current&social&and&educational&discourses&and&practices&may&have&on&young&people’s&gender&and&sexual&identities&and&practices.&In&particular,&it&was&to&draw&attention&to&the&necessity&of&addressing&issues&of&gender&and&sexual&diversity&and&identity&in&sex&education.

Title%of%the%conference Dates% Number%of%participantsEducating&young&people&about&sex:&addressing&issues&of&gender,&sexuality&and&diversity 16F18.10.2014 14

N24.%International%conferences%organised%by%the%RU

Most&important&international&conferences&with&RU&as&the&main&organiser&(maximum&of&5&cases)Title%of%the%conference Dates% Number%of%participants

14th&Annual&Conference&on&Identities&in&Conflict&F&Conflict&in&Identities 11F13&April&2014 75

N25.%Editorship%in%international%journals

Membership&of&editorial&boards&of&journals&indexed&by&ThomsonFISI,&Scopus,&ERIH&or&Engineering&Village&(max&10)

Type%of%membership% Title%of%the%journal% Name%of%the%researcher% Czech%ID%of%researcher Period%of%membership

Short%description%(150%words)

The&aim&of&the&conference&was&to&discuss&the&concept&of&identity&in&a&broad&variety&of&contexts&(for&example,&nationality,&religion,&ethnicity,&gender,&generations,&and&class,&and&also&memory,&social&rituals,&everyday&encounters,&the&arts,&consumption,&etc.),&and&to&test&the&concept's&analytical&power&and&limitations.&With&this&goal&in&mind,&we&invited&the&submission&of&historical&and&theoretically&framed&conceptual&analyses&as&well&as&papers&that&contribute&to&a&critical&examination&of&the&field&of&cultural&sociology&in&general.

MemberSociologický&časopis/Czech&Sociological&Tomáš&Katrňák 8301840 2012F

MemberSociologický&časopis/Czech&Sociological&Csaba&Szaló 4775856 2006F

MemberSociologický&časopis/Czech&Sociological&Martin&Kreidl 2271044 2012F

Page 148: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Name%of%centre Focus%of%research Main%project%partners

D.%National%research%presence%and%collaboration%%%

D1.%%National%collaborations%and%partnerships

N26.%Involvement%in%Centres%or%RI

Participation&in&a&Research&Centre&/&CoE&/&Competence&Centre&or&Research&Infrastructure&(maximum&10&cases)

Relevant%information%in%other%sections:

Shared&or&collaborative&use&of&RI&(N15&and&N16)PhDs&trained&in&the&RU&(N33)

Member&of&Scientic&Board 2843854

Member&of&Scientic&Board 4775856Institute&of&Sociology&of&the&ASCR

Czech&Science&Foundation

Csaba&Szalo

Ladislav&Rabušic

Most&important&memberships&of&scientific&advisory&boards&in&academia&(max&10&cases)

Type%of%membership ResearcherID%(from%IS)%

Member&of&Scientic&Board 2843854Masaryk&University

Name%of%organisation

Ladislav&Rabušic

Name%of%the%researcher

D2.%National%co2publications

See&bibliometric&report.

D3.%Reputation%and%esteem

N29.%Panel/board%memberships

N28.%Scientific%awards

Scientific&Prizes&Scientific&Prizes,&honours&and&scientific&positions&of&trust&awarded&to&researchers&(max&10&cases)

Name%of%prize%/%position Org.%awarding%prize Name%of%the%researcher ResearcherID%(from%IS)%

Marvin&B.&Sussman&Prize&for&Best&Dissertation

International&Sociological&Association Dominik&Bartmanski 0

There&is&a&long&term&cooperation&in&various&research&projects,&mainly&those&supported&by&the&Czech&Science&Foundation,&dealing&with&social&stratification,&inequality&(e.g.&the&project&"Social&inequalities&and&the&market&risks&following&from&housing&consumption.&The&real&and&desirable&response&of&state&fiscal&and&monetary&policies"&2008F2010&headed&by&Martin&Lux).&Recently&there&is&a&collaboration&in&the&project&aimed&to&promote&excellence&in&basic&research&i.e.&Dynamics&of&Change&in&Czech&Society&also&supported&by&the&Czech&Science&Foundation.

Dynamics&of&Change&in&Czech&Society Process&of&socialization&in&householdsInstitute&of&Sociology&of&the&ASCREconomics&Institute&of&the&ASCR

Within&the&framework&of&a&Household&Panel&Survey,&models&of&social&change&will&be&examined&using&individuals’&attitudes,&preferences&and&behaviour&within&households.&Time&use&data&and&repeated&surveys&of&children&will&provide&evidence&of&how&daily&life&is&spent.&Our&participation&is&present&in&all&aspects&of&the&project.

Describe%your%role%in%the%Centre%(50%words)%

N27.%National%collaborations%and%partnerships

Most&important&research&organisations&with&which&you&have&established&close&collaboration&(max&10).

Name%of%partner%organisation Contact%person%in%the%partner%org.

Institute&of&Sociology&of&the&ASCR RNDr.&Tomáš&Kostelecký,&CSc.Description%of%collaboration%(150%words)

Page 149: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

2009 2010 2011 2012 2013 2014

25 18 16 32 28 02 4 3 0 5 012 2 11 5 14 04 5 1 0 0 0

0 0 0 0 0 0

0 0 0 0 0 0

0 0 0 0 0 0

0 0 0 0 0 00 0 0 0 0 00 0 0 0 0 0

Plant&variety&(Zodru)Animal&breed&(Zplem)

See&bibliometric&report&for&more&data.

Book&chapter&(C.)Conference&proceedings&/&Article&in&proceedings&(D)Non2traditional%research%outputs

Results&used&by&the&funding&provider,&i.e.&&into&legislation&or&norm,&&into&nonFlegislative&or&strategic&

Research&report&containing&classified&information&(V)

Patents%and%other%forms%of%IP

Patent&(P)

Certified&methodologies,&art&conservation&methodologies,&specialized&map&works&(N)

Research&performance

Scholarly%outputs

Article&in&a&periodical&(J)Monographs&and&books&(B)

E.%Research%productivity%%%%

E1.%Research%outputs

Page 150: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

2009 2010 2011 2012 2013 20147 6 11 14 19 318 16 20 21 26 3215 22 31 35 45 63

2009 2010 2011 2012 2013 201446.7% 27.3% 35.5% 40.0% 42.2% 49.2%53.3% 72.7% 64.5% 60.0% 57.8% 50.8%100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

2009 2010 2011 2012 2013 201435 34 31 35 43 4115 22 31 35 45 63

0.43 0.65 1.00 1.00 1.05 1.56

2009 2010 2011 2012 2013 201415 22 31 35 45 634 4 9 6 3 0

0.27 0.18 0.29 0.17 0.07 0.00

PhD&students&who&did&a&substantial&part&of&their&thesis&work&(>50%)&at&the&RU&but&enrolled&in&another&Research&Organisation.

2009 2010 2011 2012 2013 20141 1 4 3 5 52 3 6 6 5 63 4 10 9 10 11

2009 2010 2011 2012 2013 201433.3% 25.0% 40.0% 33.3% 50.0% 45.5%66.7% 75.0% 60.0% 66.7% 50.0% 54.5%

100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

PhD&students&who&did&a&substantial&part&of&their&thesis&work&(>50%)&at&the&RU&but&enrolled&in&another&Research&Organisation.

2009 2010 2011 2012 2013 201435 34 31 35 43 413 4 10 9 10 11

0.09 0.12 0.32 0.26 0.23 0.27

Total

Total&researchers&in&the&RU&(HC)Total&PhD&students&enrolled

F.%Ability%to%attract%PhD%students

RU&ability&to&attract&PhD&students

N30.%PhD%student%enrolment

Raw&dataMaleFemaleTotal

Shares&of&the&totalMaleFemale

Raw&dataMale&Female

N31.%PhD%students%versus%total%researchers

Ratio%PhD%students%/%total%researchers

RU&ability&to&train&PhD&students

N32.%PhDs%awarded%in%the%RU

Total&PhD&student&enrolledAwarded&PhDsShare%of%PhDs%enrolled

N33.%PhDs%trained%in%the%RU

Ratio%trained%PhDs%/%researchers

Trained%PhDs

N34.%Level%of%investment%in%PhD%training

Raw&dataNumber&of&researchers&in&the&RU&(FTE)Trained&PhDs

FemaleTrained%PhDs

Shares&of&the&totalMale&

Page 151: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

See&bibliometric&report.

2009 2010 2011 2012 2013 201433.8 33.4 36.6 70.9 70.2 54.73.6 2.6 8.0 13.4 14.7 15.90.0 0.0 0.0 0.0 0.9 0.70.1 0.0 0.0 0.0 7.4 5.60.0 0.0 0.0 0.0 0.0 0.0

37.6 36.0 44.6 84.3 93.3 77.0

2009 2010 2011 2012 2013 201489.9% 92.7% 82.1% 84.1% 75.3% 71.1%0.0% 0.0% 0.0% 0.0% 0.9% 0.9%0.4% 0.0% 0.0% 0.0% 8.0% 7.3%0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

International%positioning

2009 2010 2011 2012 2013 20140.0 0.0 0.0 0.0 7.4 5.60.1 0.0 0.0 0.0 0.0 0.03.0 0.0 0.0 0.0 4.0 0.0

3.1 0.0 0.0 0.0 11.4 5.6

2009 2010 2011 2012 2013 20140.0% F F F 65.0% 100.0%4.5% F F F 0.0% 0.0%

95.5% F F F 35.0% 0.0%

100.0% 2 2 2 100.0% 100.0%

8.3% 0.0% 0.0% 0.0% 12.3% 7.3%

Contract&research&from&international&public&bodiesTotal%international%funding%2%competitive%&%contract%

research

Shares&of&the&totalEuropean&Funding

G.%Overall%quality

G1.%International%citations

Relevant%information%in%other%sections:

The&international&positioning&based&on&funding&data&(N4)The&usage&of&major&national&or&international&RI&(competitive&access)&(N14)

G2.%Research%outputs%quality%assessment

Total%funding%for%research

Shares&of&the&total

N3.%Overview%funding%sources%2%categories

Raw&DataInstitutional&funding&for&researchNational&competitive&funding&(‘targeted’&funding)National&funding&from&Structural&Funds

N35.%Value%of%the%research%activities

The&main&area&of&value&associated&to&research&in&our&RU&is&clearly&academical,&in&the&sense&that&research&enriches&our&understanding&of&the&world.&By&giving&priority&to&empirical&research&we&promote&those&forms&of&institutionalized&understanding&which&are&grounded&on&rationality&and&realism.&This&is&crucial&as&far&as&one&acknowledges&the&role&of&knowledge,&especially&expert&systems&in&the&constitutions&and&reproduction&of&social&structures.&In&other&words&our&RU&is&dedicated&to&the&rational&potential&of&the&social&sciences&which&can&be&developed&by&focusing&on&historically&and&geographically&specific&empirical&cases.

Major&area&of&value&in&research&and&development&of&the&activities&in&RU&from&a&scientific&or&socioFeconomic&perspective.&

A2.%Ability%to%attract%external%funding%(RU)

Overview

Total%funding%for%research

Competitive&international&funding

Institutional&funding&for&researchNational&funding&from&Structural&FundsCompetitive&international&funding

Funding&from&contract&research

Funding&from&contract&research

Activity&profile&of&the&RU&based&on&funding&sources

N4.%International%funding%2%competitive%&%contract%research

Raw&Data

Funding&from&other&foreign&public&sourcesContract&research&from&international&public&bodiesTotal%international%funding%2%competitive%&%contract%

research

Share&of&total&funding&for&research

European&FundingFunding&from&other&foreign&public&sources

Page 152: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Relevance%for%the%public%sector

2009 2010 2011 2012 2013 20140.0 0.0 0.0 1.9 2.5 3.30.6 0.2 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.0

0.0 0.0 0.0 0.0 0.0 0.0

0.6 0.2 0.0 1.9 2.5 3.3

2009 2010 2011 2012 2013 20140% 0% 0% 100% 100% 100%

100% 100% 100% 0% 0% 0%0% 0% 0% 0% 0% 0%

0% 0% 0% 0% 0% 0%

100% 100% 100% 100% 100% 100%

2% 1% 0% 2% 3% 4%

Relevance%for%industry

2009 2010 2011 2012 2013 20140.0 0.0 0.0 0.0 0.0 0.0

2009 2010 2011 2012 2013 20140.0% 0.0% 0.0% 0.0% 0.0% 0.0%

National%funding

2009 2010 2011 2012 2013 20143.1 2.4 8.0 11.5 12.2 12.60.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 1.9 2.5 3.30.6 0.2 0.0 0.0 0.0 0.03.6 2.6 8.0 13.4 14.7 15.9

2009 2010 2011 2012 2013 201484.5% 90.8% 99.9% 85.6% 83.1% 79.0%0.0% 0.0% 0.0% 0.0% 0.0% 0.0%0.0% 0.0% 0.0% 14.4% 16.9% 21.0%

15.5% 9.2% 0.1% 0.0% 0.0% 0.0%100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

9.7% 7.3% 17.9% 15.9% 15.8% 20.7%

2009 2010 2011 2012 2013 20140.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.0 0.00.0 0.0 0.0 0.0 0.9 0.70.0 0.0 0.0 0.0 0.9 0.7

2009 2010 2011 2012 2013 2014F F F F 0.0% 0.0%F F F F 0.0% 0.0%F F F F 0.0% 0.0%F F F F 100.0% 100.0%F F F F 100.0% 100.0%

0.0% 0.0% 0.0% 0.0% 0.9% 0.9%

Other&–&OP&1Total

Total

Share&of&total&funding&for&research

Shares&of&the&totalOP&RDI&F&European&Centres&of&ExcellenceOP&RDI&F&Regional&R&D&centresOP&Prague&Competitiveness&(OPPC)Other&–&OP&1

National%funding%from%structural%funds

Raw&DataOP&RDI&F&European&Centres&of&ExcellenceOP&RDI&F&Regional&R&D&centresOP&Prague&Competitiveness&(OPPC)

Technology&AgencyMinistry&of&CultureMinistry&of&Education,&Youth&and&SportsTotal

Share&of&total&funding&for&research

Ministry&of&CultureMinistry&of&Education,&Youth&and&SportsTotal

Shares&of&the&totalScience&Foundation

Technology&Agency

Shares&of&the&totalCompetitive&funding&F&Ministry&of&CultureCompetitive&funding&F&Ministry&of&Education,&Youth&and&SportsContract&research&from&ministriesContract&research&from&other&public&sector&institutions&(in&the&Czech&Republic)Total%competitive%&%contract%funding%from%public%

sector%(sectoral%ministries)

Share&of&total&funding&for&research

N6.%Collaboration%with%industry%2%competitive%&%contract%research

Raw&DataTotal%funding%from/for%cooperation%with%industry

Shares&of&the&total

National%competitive%funding%(‘targeted’%funding)

Raw&DataScience&Foundation

Competitive&funding&F&Ministry&of&CultureCompetitive&funding&F&Ministry&of&Education,&Youth&and&SportsContract&research&from&ministriesContract&research&from&other&public&sector&institutions&(in&the&Czech&Republic)

Share&of&total&funding&for&research

NO%DATA%PROVIDED

Total%competitive%&%contract%funding%from%public%

sector%(sectoral%ministries)

N5.%Funding%from%public%sector%(sectoral%ministries)%2%competitive%&%contract%research

Raw&Data

Page 153: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Research&excellence

H.%Research%excellence%%%%

H1.%Peak%quality

Referees&–&review&of&submitted&research&outputs

Bibliometrics

Page 154: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

Relevant%information%in%other%sections:

Shared&or&collaborative&use&of&RI&(N16&and&N17)Involvement&in&Centres&or&RI&(N27)

Societal&relevance

I.%Knowledge%&%technology%transfer%activities

I1.%Income%from%societal%relevant%research

Relevant%information%in%other%sections:

Name%of%Institution Type%of%organisation Contact%person%in%the%partner%org.

Funding&from&public&sector&(sectoral&ministries)&&(N5)Collaboration&with&industry&F&competitive&&&contract&research&(N6)

N36.%%Income%from%the%commercialisation%of%research%outputs

Major&area&of&value&in&research&and&development&of&the&activities&in&RU&from&a&scientific&or&socioFeconomic&perspective.&

N37.%Non2academic%collaborations

Most&important&nonFacademic&collaboration&with&societal&actors&(max&10).

National&collaborations&and&partnerships&(N28)Value&of&the&research&activities&(N36)

I2.%Collaboration%with%other%societal%actors

I2.%National%research2industry%collaboration

Museum&of&Romany&Culture,&Brno Cultural&Institute Mgr.&Michal&SchusterDescription%of%collaboration%(150%words)

This&cultural&institute&is&a&crucial&partner&in&RU's&research&project&(LEPERIBEN)&dealing&with&the&migration&of&Romany&workers&from&the&Slovak&part&of&Czechoslovakia&to&the&industrial&centres&in&its&Czech&part.&The&core&of&this&research&is&collecting&oral&history&testimonies&of&those&migrants&who&recently&live&in&the&Czech&republic.&The&project&aims&to&connect&social&science&research&with&public&communication&of&its&results&via&exhibitions&and&documentaries.&The&project&is&funded&with&the&Ministry&of&Culture.

Name%of%organisation Name%of%the%researcher ResearcherID%(from%IS)%

I3.%Reputation%and%esteem%beyond%academia

N38.%Participation%in%incubators%and%clusters

Participation&in&incubators&or&clusters&F&maximum&of&10&different&cases.

NO%DATA%PROVIDED

NONE%PROVIDED

I4.%Income%from%commercialisation%activities

Membership&in&the&Commission&for&Social&Sciences&and&Humanities

Govermnent&of&the&Czech&Republic Ladislav&Rabušic 2843854

I3.%Non2traditional%scholarly%research%&%IPR%outputs

See&bibliometric&report

N39.%Membership%of%advisory%boards%outside%academia

Most&important&memberships&of&scientific&advisory&boards&outside&academia&(max&10&cases)Type%of%membership

Page 155: Draft of the Third Interim Report / Background report 10: The Small Pilot Evaluation – Feedback and Results

The&economic,&social,&cultural,&policy&and&other&impacts&of&the&work&conducted&at&the&Research&Unit&on&Czech&society.

The&societal&impact&of&our&research&can&be&conceived&in&terms&of&thres&sphere&of&potential&influence:&(i)&impacts&on&public&sphere,&(ii)&impacts&on&public&policy,&(iii)&impact&on&welfare.&In&the&first&case&the&beneficaries&can&be&individuals,&associations,&organisations&or&local&communities&whose&understanding&of&contemporary&world&and&their&situation&in&these&conditions&has&been&improved&both&through&theri&collaborative&involvment&in&the&research&process&as&well&as&by&means&of&stimutating&a&public&debate&which&can&recognize&their&particular&situation.&For&instance&our&research&dealing&with&biographies&of&Romany&migrant&workers&living&in&big&industrial&cities&can&be&conceived&in&these&terms.&The&second&case,&that&is&impacts&on&public&policy,&where&beneficaries&can&be&governmental&institutions,&nonFgovernmental&organizations,&movements,&is&built&on&possible&influence&of&documented&evidence&of&social&changes&and&their&mechanisms&on&those&who&have&specific&interests&in&these&processes&of&change.&For&instance&our&research&dealing&with&reproduction&of&social&inequalities&through&selective&forms&secondary&education&can&be&informative&both&for&government&agencies&and&NGO's&dealing&with&educational&processes.&In&the&third&case,&in&the&sphere&of&welfare,&the&beneficaries&can&be&individuals&and&groups&whose&quality&of&life&depends&on&the&influence&of&various&expert&systems&like&medicine,&social&care,&public&health&services.&For&instance&our&research&dealing&with&population&ageing,&especially&with&organizations&supporting&active&seniors&can&lead&to&changes&to&various&institutional&guidelines.

N40.%Societal%impact

I5.%Societal%impact%narratives