Upload
keely
View
36
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Stakeholder Involvement and the Choice of Science & Technology Policy Outcome Evaluation Methods. Presenter : Shan-Shan Li STPI, NARL Presentation on 2009 AEA conference 2009.11.14. Outline. Introduction Methodology Analysis Research Fundings Conclusions. Introduction. - PowerPoint PPT Presentation
Citation preview
Stakeholder Involvement and the Choice of Science & Technology Policy Outcome Evaluation Methods
Presenter : Shan-Shan LiSTPI, NARL
Presentation on 2009 AEA conference 2009.11.14
2
Outline
IntroductionMethodologyAnalysis Research FundingsConclusions
3
IntroductionIntroduction
4
The importance of stakeholder
• Stakeholder involvement plays an indispensable part in science & technology management.
• Stakeholder involvement can…– help to clarify the root causes of problems and to
develop effective strategies for addressing them during different phrases of the S&T management process
• Evaluation methods are the means for solving such problems or meeting stakeholders’ requirements.
5
Comparison of 10 globally used policy outcome evaluation methods
Sources: Summarized and organized from Fahrenkrog, Gustavo, Polt, Wolfgang, Rojo, Jaime, Tubke, Alexander, & Zinocker, Klaus. (eds.) 2002
6
Research questions
Stakeholders’ demand-side requirements
Stakeholders’ supply-side issues
which S&T policy outcome evaluation methods are most suited for measuring stakeholders’ interests and in assessing their impacts in each phase of publicly funded S&T programs?
which S&T policy outcome evaluation methods are most appropriate to use in measuring their interests and in assessing their impacts in each phrase of publicly funded S&T programs?
7
Stakeholders in S&T management
ProfitOrganization
S&T Funding Organization
Internal Environment
Competitors
Contractors
End user
Reviewers
Mass media
Government/decision makers
Consultants
Customers/community
Industrial strategic Partners
Research team
Administrative personnel
Researchers
External environment
The internal environment: which is the environment that is related to S&T management and assignment, used to describe individuals or a group of stakeholders that participate directly in the internal S&T environment, such as customers, contractors, etc.
The external environment: This is formed by species elements and is related to the S&T management including technology, law, economy, politics, physics, labor supply and demand, culture and environments, etc.
8
MethodologyMethodology
9
Methodology
funders A
funders B
S&T executive institutions A
cooperative institutions A
cooperative institutions C
cooperative institutions B
cooperative institutions D
the end-users utilizing the final products or services of S&T
research A
the end-users utilizing the final products or services of S&T
research C
the end-users utilizing the final products or services of S&T
research B
the end-users utilizing the final products or services of
S&T research D
Allocation of funding Science community’s research output, Outcomes and impact
Research fields’ collaboration societal impact (impact of research utilization)
the end-users utilizing the final products or services of
S&T research E
S&T executive institutions B
S&T executive institutions C
Sources: Academy of Finland (2006)
we will use the 4 phases explained in the Academy of Finland’s report to further derive the issues that the stakeholders are concerned with in order to draw a strategic plan for the future
10
The properties of the research funding impact phases
Phase Concerns Properties
Allocation of public funds
Can be seen from the point of view of a nation, sponsoring organization itself, applicants and impact
Understand the impact caused by the approval of the funding and the allocation ratio
Outputs and outcomes of S&T communities
The impact of the modality caused by scientific research
Number of research output and impact elementsNumber of research output provided by the research community to non-academic users for utilization and developmentThe impact of the research executor’s participation on the research program
Cooperation of research communities
Cooperative system between research organizations or between research organization and research users
Scientific community in the form of knowledge spillovers effectThe formation of international cooperation networkThe level of knowledge flows between research communities and research usersSimple calculation of cooperation agreementVisualization of network frameworkInvestigate formation of cooperative relations
Impact of research utilization
The impact of research results is often not formed by a single program, institution, enterprise etc. but rather, it’s the stakeholders that cause the impact, such as social, economic etc.
Evaluation on relevance of the users’ researchesTarget creation of the research result commercializationInvestigation on the change in users systemGreat impact caused by the evaluation research results
Sources: Academy of Finland (2006)
11
The 4 groups of stakeholders within S&T management
•Academic system•Industrial strategic partner
•Government•Investigator•Contractor•Competitor•Decision-maker
•Research team•Administrators•Consultants•Profit Organization (e.g. insurance)
•End users•Customers•Academic staff•Community
Sponsors
S&T R&D management environment
S&T R&D implementation
system+
Cooperative network system
S&T R&D result users
12
Stakeholders’ supply-side and demand-side
PhasesStakeholders Main concerns
Supply-side Demand-side Supply-side Demand-side
Allocation of public funds
FundersS&T research and development implementation system
S&T research and development management processS&T research and development planning implementation detailReasonability of the funds allocationComparison between actual expenditure and budgeted amountExternality problemAcademic research institute financial enhancementSupport and promote high-quality research
Investigation of system fairness and transparencyCommunication process
Outputs and outcomes of S&T
communities
S&T executive institutions
Cooperation network systemCustomers/ community/ end-users
Resource allocationExternal network connectionImplementation process efficiencySchedule arrangement and planningFuture expected outcomeOutput produced meet expectation
Cooperation directionUtilization of research results
Academic staffS&T research and development energy development
Cooperation of research
communities
Cooperation network system
Academic staff/ end-users
Cooperation direction and systemThe role of the systemThe level of relevance between industry, government, academic and research
Energy upgrade
Impact of research utilization
Cooperation network system Academic staff/end users
Community
Increase productionEnhance economic growthCreate legal environmentLevel of result commercializationThe impact that result commercialization has on the industrySpillover effectProducts sold at a profit and impact on employmentBenefits of technologyOutstanding innovation procedures provide feedback to the S&T research and development result
Research utilization
13
AnalysisAnalysis
14
• Through the analysis of the two dimensional matrix of :– stakeholders’ interests
– evaluation methods
• Comparisons (demand-side and supply-side) are shown in p.16, p.17.
Analysis
15
Direction of discussion : example
Phases Sides Meanings Methodspublic fund allocation
supply-side Funders (including governments, decision makers, contractors, competitors, etc.) are the main stakeholders who provide S&T related funds for some publicly funded executors (including S&T institutions and end-users). Under the circumstance, they have own selective criteria on who is more appropriate institution performing publicly funded S&T programs, including program management efficiency, appropriateness and relevance of programs, rationality of funds allocation, whether the funds additionally improve S&T institutions’ financial situation or not, or whether the funds support and promote excellent research.
Take expert review, a qualitative method
public fund allocation
demand-side S&T institutions (including team members, administrators, program officers/managers, consultants, etc.) are the main stakeholders who want to apply for publicly funds. Their concerns include the efficiency of communication between managers and members, and the transparency and fairness of the auditing mechanism.
Take expert review as an example also to discuss these issues.
16
Comparisons between stakeholders, and evaluation methods (Demand-side)
Issue form Allocation of public funds Outputs and outcomes of S&T communitiesCooperation of
research communities
Impact of research
utilization
Stakeholders S&T research and development implementation system
Cooperation network system
Academic staff Research result users Community
Concerning issues Investigation of system fairness and
transparency
Communication process
Cooperation direction
S&T research and development
energy development
Research utilization/
safety/ function/ quality
Energy upgrade Research development utilization/
safety/ function/ quality
Evaluation methods
Macroeconomic Modeling
& Simulation*
Micro-econometric Models
* * *
Cost-benefit Analysis * * *
Case Studies * * * * * *
Benchmarking * *
Expert Review * * * * *
Control Group * * *
Innovation Survey * * *
Network Review * * * *
Foresight/Technology Assessment
* *
17
Comparisons between stakeholders, and evaluation methods (Supply-side)
18
Research findingsResearch findings
19
Macroeconomic modeling and simulation
demand sidedemand sidesupply sidesupply side
The funders’ concern about the externalities problem created by research resources (e.g. knowledge spillover effect or industrial innovation spillover effect etc.)
The community concern about impact of R&D input on the economy (e.g. economic output and employment creation etc.)
It can be applied together to measure policy’s direct outcome and between research fields, inter-industries spillover effects’ direct outcome.
20
Micro econometric model
This method can directly measure the outcome of public policies towards company or individual project.
supply sidesupply side demand sidedemand side
In order to answer questions about rationality of fund allocation and external outcome etc. and also help S&T research and development implementation system into comparing similar projects implementation results, for S&T research and development results users, the model can evaluate researches for instance R&D activities’ achievements (e.g. relationship between R&D budget and company’s ability to earn profit) or spillover effect of individual project’s R&D activity towards industries etc
As for the community, evaluation of the effect caused by government’s science and technology policy measures can be done using the method (e.g. implementation and authentication results of scientific projects)
21
Cost benefit analysis
It calculates the socio-economic impacts when the implementation plan is carried out, and through the cost benefit analysis result, suitable project, case or policy can be selected. It helps the manager to think about the cost and benefit in a more structural and systematic way.
supply sidesupply side demand sidedemand side
It is suitable in measuring the appropriateness of the resources and funds allocation and the efficiency and the outcome of the project implementation
It can evaluate academic staff’s S&T research and development energy development, research output quality and also S&T research and development result users and the community in the level of research output utilization
22
Expert panels/ Peer review
They are able to not only carry out ex-post measurement, but also ex-anti evaluation.
supply sidesupply side demand sidedemand side
experts can use the rationality of the expected budget allocation to further analyze the ex-anti evaluation section. It can play an important part in terms of S&T research and development implementation system’s investigation fairness & transparency, communication process and the utilization of the energy development.
with research result development and utilization, they can also provide credible and effective suggestions.
23
Control group approaches
supply sidesupply side demand sidedemand side
This method is mainly under the control of other variables, explore the effect single manipulated variable has on different samples; the main manipulated variables are sponsorship funds and applicants.
It is appropriate to apply control group approaches on measuring and evaluating private enterprises economic activities for example: the impact of governmental department’s S&T research and development grants on private enterprises R&D, the impact of government fiscal measures (lowering taxes), and the effectiveness of loaning between emerging companies and banks etc.
It is able to help provide analytical information on the demand-side’s fund allocation issues and also additionality effect, such that academic staff’s energy development can be further analyzed.
24
Innovation surveys
supply sidesupply side demand sidedemand side
It mainly focus on the evaluation of the corporations’ innovation activities.
Funders can monitor whether policy tools can help to achieve product innovation target or not through funds grant formula or rationality of funds allocation
Stakeholders will be very much concern about economic and industrial growth; level of commercialization and product sales due to innovation etc
25
Network analysis
supply sidesupply side demand sidedemand side
It is mainly a relationship structure between each node and also the behavioral change result at each node through the end of the decision-making process.
Cooperation network system’s stakeholders have always been the target of the network analyze explore. As a result, necessary information can be obtained through network analysis; also it can explore the condition of knowledge distribution
It can fully perform its function when looking at the fund allocation communication process and network system cooperation direction on the demand-side. It explores the interaction and state between each node; therefore the academic staff’s S&T research and development energy development and users’ energy enhancement on the demand-side can also be
analyzed and explored
26
Foresight/ technology assessment
supply sidesupply side demand sidedemand side
The transfer of foresight and technology onto the evaluation is mainly a comprehensive outline of the whole fund allocation direction and S&T research and development project content.
The stakeholders on the supply-side will care most about the S&T research and development planned implementation procedures and rationality of the allocation of the public funds
It would be more interested in knowing the safety of the research and development result and whether or not future demand is met etc
27
Field/ Case study
supply sidesupply side demand sidedemand side
The survey method is qualitative and is suitable for those innovative projects that haven’t been analyzed in depth yet and is also suitable on the policy measures that can have complicated socio-economic impacts, though it is not very appropriate to use it to test the causal relationship.
Due to explore the details of S&T implementation. It is suitable for some issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality
In order to respond the S&T impact. It is the application of these issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality
28
Benchmarking
supply sidesupply side demand sidedemand side
It can apply on product, service, procedure, method, framework, and the whole organization etc. In the process of benchmarking, target selection is very important.
The choices of indicators can be applied in some issues, Investigation of system fairness and transparency, S&T research and development energy development
The choices of indicators can be applied in some issues, Investigation of system fairness and transparency, S&T research and development energy development
29
Conclusions
• Stakeholders are a determinate element in the choice of outcome evaluation methods. The level of attention paid towards each issue will depend on each stakeholder ultimate goals causing a constant need of coordination in conflicted S&T policy decision makings and evaluation systems. A combination of multiple S&T policy outcome evaluation methods is necessary.
• Government organizations can carefully formulate a suitable policy through interacting with the stakeholder ; resolving concerning issues and planning future development of the policies together, in order to achieve a realm of ‘mutual benefit’.
• The identification of stakeholders indeed have influence on the whole process of evaluation.
• When we make the choices of evaluation methods, we need consider the indispensable element, the stakeholders.
Li, Shan [email protected]
http://www.stpi.org.tw/STPI/index.htm
31
Thanks For Your Attention!Thanks For Your Attention!