Embed Size (px)
DESCRIPTION
About IPM and ERC of CAS IPM Found in June 1985 Devoted to the studies of development strategy, development and reform policy, public administration, S&T management and state-of-the-art theories and methodologies of related disciplines Faculty: 138 ERC of CAS (also affiliate to IPM) Faculty: 14 Mission Management Innovation and Evaluation Research, institutes evaluation
Citation preview
Review of Quantitative Monitoring System of Chinese Academy of
SciencesZHENG Haijun, GUAN Zhongcheng, WANG Biaoxiang, WU Jianmei
Management Innovation and Evaluation Research Center, CASInstitute of Policy and Management, CAS
AEA 2013, Washington DC
About IPM and ERC of CAS• IPM• Found in June 1985• Devoted to the studies of development strategy, development
and reform policy, public administration, S&T management and state-of-the-art theories and methodologies of related disciplines• Faculty: 138
• ERC of CAS (also affiliate to IPM)• Faculty: 14• Mission
• Management Innovation and Evaluation Research, institutes evaluation
Quantitative Monitoring is Necessary1
Quantitative Monitoring• Peer Review• Quantitative Monitoring(QM)：widely used
management tool for international institutions• NIH• MPG• INRIA
• SoSP
Quantitative Monitoring in CAS2
Development of Quantity Monitoring of CAS
>Performance+Status>Rank and classify
>Contribution
>Rank by one system
>Outcome+ Contribution>Different systems>Rank by system
EV. = QM EV. = QM+Peer Review EV. = Peer Review， ref. QM
Rank with Mainly Numbers No ranking, more quality
Blue Book Ev.
White Book Ev.
Yellow Book Ev.
Comprehensive Quality Ev.
Major Outcomes Oriented Ev.
>Innovation Capacity>No more rank
>Key indicators+Innovation Capacity>No more rank
Blue Book Evaluation（ 1993-2001）
• QM=Evaluation• Totally Quantity Evaluation• 1993-1999: rank within one system• 1998-2001: rank within three category series• Basic Research• Applied Research• Development and High Tech Research Industrialization
Category Parent Indicator Child Indicator
Performance Evaluation
Basic Research Performance
Natural Science Awards
Papers (amount and per amount)
Citations
Other Publications
Applied Research PerformanceS&T Progress Awards and Invention Award
Patents (Applied and Granted)
Development and High Tech Research Industrialization
Institution Company's Total Profit Tax
Development Income
Commercialization of Research Findings
Blue Book Evaluation（ 1993-2001）
Category Parent Indicator Child Indicator
Status Evaluation
Research Status
Capability of Undertaking Government and Local Governments' MissionsOutcomes (amount and per amount)International Cooperation (Funds and # of Programs)
Talents
Academician of CASProportion of Senior Researchers under 45 years oldProportion of holding PhD’s and Master’s Degree# of Graduate Students
FundingInput FundsFunds per ResearcherOutside funding
Equipment and labs
Advanced EquipmentProportion of value of Advanced Equipment from 90’ in all Equipment# of Opening Labs and Engineering Center
Blue Book Evaluation（ 1993-2001）
• Evaluation results were revealed according to indicators• Rank by score into four categories A, B, C and D within
each indicators
• Performance and Status in forms of Category• Results of one institute from different years are not
comparable
Blue Book Evaluation（ 1993-2001）
White Book Evaluation（ 1999-2001）
• Objective completion status + basic, strategic and prospective contribution• Objective completion status Ev., Peer Review• S&T objective Ev.
• Contribution of work• Influence to subject, etc.
• Management objective Ev.• Does fixed staff turnover rate exceed 5%?• Does the usage of Special Funds for KIP exceed 60%? Etc.
White Book Evaluation （ 1999-2001）
• Evaluation of basic, strategic and prospective contribution: QM
White Color Book（ 1999-2001）
Parents Indicators Child Indicators
Undertaking of Strategy S&T MissionNational Major ProgramsInternational Programs
Papers High IF journals by subjective
Major Lectures of International ConferencePlenary LecturesInvited Lectures
Significant social and economic benefits
Research Outcomes transferStart-upowner's equityMajor Consultation ReportsExcellent labsIP(Issued patens, Software copyrights)
Talents
Academician and Major Projects investigators, etc. S&T ConsultantsInternational TalentsYoung Talents
Major Awards National Awards
White Book Evaluation （ 1999-2001）
•Qualitative evaluation has been introduced to White-Book evaluation•Evaluation = QM+Peer Review•Scores, not classification results•Turned qualitative evaluation results into quantitative data, then got the ranking Scores based on all quantitative data•Characteristics of institutes disciplines is not enough, and QM is still a primary method
Yellow Book Evaluation （ 2002-2004）
• Started to focus on quality over quantity• Evaluation System• Major Innovation Contributions Evaluation, Peer Review• Basic Indicators Evaluation, conducted by CAS headquarter
bureaus• Classification Oriented Evaluation, for all institutes in CAS• basic research• high technology research and development• resources, environment and sustainable development• Industrialization
Yellow Color Book（ 2002-2004）
Series Indicators Remarks
Basic Research
Papers Top journalMajor International Conference Lectures S&T Awards National Science Awards
Research Programs NSFC, 973, etc.
Major Innovation Contributions
Resources, Environment and Sustainable Development
Papers Top journal
Major International Conference Papers
Consultant Reports Adopted by national leaders or the central ministries and commissions, with substantive instructions
S&T Awards National or Provincial awards of Science Awards, Invention Awards, and Progress Awards
Standards International, national, or industry standards
Research Programs NSFC, 863,973, etc.
Major Innovation Contributions
Series Indicators Remarks
High Tech Research and Development
Patents
Software Copyright
New medicines, new pesticides, new crop varieties
S&T Awards National Progress Awards and Invention Awards
Standards International, national, or industry standards
Research outcomes transfer Amount to the account
Research programs 863, 921, etc.
Major Innovation Contributions
Industrialization
Owner’s equity Total Owner’s equity
Research outcomes transfer Amount to the account
Research programs Major Industrialization Programs and Enterprises Programs
Major Innovation Contributions
Yellow Book Evaluation（ 2002-2004）
• Evaluation = QM + Peer Review (similar to White Book Evaluation)• Started to pay more attention to• major innovation contributions according to series• subject’s specialty
• Monitoring results is reflected as score within each category series
Comprehensive Quality Evaluation (2004 - 2011)
• Ten key process are included in the evaluation• self-evaluation of institutes• strategic planning for next stage• peer review for existing outcome• comprehensive analysis of previous evaluation• Communication review• on-site assessment• president office conference etc.
• Final conclusions and opinions are based on results of diverse contents of strategic decisions• Qualitative Evaluation, and QM play an important part in
decision making
•Before 2004• quantitative monitoring gradually returns to its original function• a powerful tool to compare among institutes by providing year-based information• cannot display developing trend of institutes within certain years in a more comprehensive way
Comprehensive Quality Evaluation (2004 - 2011)
Innovation Capacity Indicator (ICI) Monitoring3
Design of Innovation Capacity Indicator (ICI)
• Law of Comparative Advantage, David Ricardo, 1817• In economics, comparative advantage refers to the ability of a party to produce a particular good or service at a lower marginal and opportunity cost over another. Even if one country is more efficient in the production of all goods (absolute advantage in all goods) than the other, both countries will still gain by trading with each other, as long as they have different relative efficiencies.
• Indicators• Status• Outcomes• Impact
•Weights• Weights of indicator are based on simulation of years
historical data• Evaluate the importance of parents indicators• Set up the price according to data of a certain year
Design of Innovation Capacity Indicator (ICI)
How to Choose Indicators•CAS has 100+ institutes, and covers most of disciplines• Indicators should reflect•common characteristics of institute• respective expertise and features
How to Choose Indicators•Expert group discussion, and approval of most institutes•7 Indicators to monitor
Status
OutcomesImpact
Researches FundsExcellent Talents
High Quality PapersIPCompleted Major Programs
S&T AwardsConsultant Reports
•To some degree, the indicators of ICI consider different characteristics of different institutes• excellent talents and scientific research fund are necessary and common indicators• high quality paper is the common indicator to reflect the output of fundamental research institutes
How to Choose Indicators
• Intellectual Property is an important output indicator in high technology institute• Completed Major Programs can reflect how it meets
the demand of the nation• In addition, the characteristics indicators are included in the child indicators• National Natural Science Awards• National Technological Invention Awards• National S&T Progress Awards
How to Choose Indicators
• Number of outcomes increasing rapidly
How to Choose Indicators
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
1877
9500
16550
# of SCI Papers of CAS
Year
•Higher Quality Requirements in Choosing Indicators•For example, papers• published in journal whose impact factor is among top 15% within this discipline (JCR&SJCR)
How to Choose Indicators
Weights of Indicators• Management experts and strategy experts made a group
decision making on the weights of ICI, based on evaluation orientation, management experience and a large amount of simulation computation
Indicators WeightsExcellent Talents 20%Research Funds 10%High Quality Papers 20%IP 15%Completed Major Programs 25%Major Consultant Reports 5%Major Research Awards 5%
•Using the weights of 7 indicators, we can determine the price of parent indicators based on the data of 2003 and 2004.•Weights of child indicators, determined by its importance•One 973 project worth 5 points， and one NSFC program worth 2 points.
Weights of Indicators
Parent indicators Child Indicators
ICI
Excellent Talents, 20% Top talentsGraduate education
Research Funds, 10% Funds from outsideHigh Quality Papers, 20%
IP, 15%PatentsSoftware CopyrightsStandards
Major Consultant Reports, 5%
Completed Major Programs, 25%NSFCMOSTOther Programs
Major Research Awards, 5%Major International AwardsNational AwardsProvincial Awards
Computation
,
• • is the value indicator of institute • is the price of indicator • is the weights of indicator • is the capacity of institute
How to Consider Efficiency•total amount of seven indicators are divided by the number of innovation positions or regular budget•Regular funding for KIP• If Parent Indicator represent money
•# of Positions for KIP• If Parent Indicator does not represent money
ICI for CAS• Great support for management of CAS• Institutes can also recognize their positions among
similar institutes• Development trend of CAS and its institutes in recent
years
ICI for CAS
2004' 2005' 2006' 2007' 2008' 2009' 2010' 2011'
8100
12541 10839 11659 12390
15098
20012 17801
Conclusion4
Conclusion•QM systems fits the improvement of the research quality of CAS Institutes•Rapidly promoted the outcomes of CAS• ICI is a useful management tool
Limitations of ICI• Some indicators is bad for some institutions• High Quality Paper• Small discipline
• ICI does not reflect all the characteristics of an institute• Mainly common indicators• Some meaningful works are difficult to quantify• Large-scale scientific facility• Scarce Resources
Conclusion• Peer review is getting more attentions from CAS• ICI becomes a tools of monitoring, neither ranking or
comparing among institutes• 2012, Major Outcomes-Oriented Evaluation, QM is an
important part of the evaluation system• Monitoring Key indicators + ICI• Focus on diagnostic function of their own development
