Unintended consequences of innovation policy programmes: Social evaluation of technological projects programme in Croatia

Copyright © eContent Management Pty Ltd. Innovation: Management, policy & practice (2011) 13: 77–94.

Volume 13, Issue 1, April 2011 INNOVATION: MANAGEMENT, POLICY & PRACTICE 77

Unintended consequences of innovation policy programmes:

Social evaluation of technological projects programme in Croatia

JADRANKA ŠVARC AND JURAJ PERKOVIC’Institute of Social Science Ivo Pilar, Zagreb, Croatia

JASMINKA LAŽNJAKDepartment of Sociology, University in Zagreb, Zagreb, Croatia

ABSTRACTThe paper presents empirical results of social evaluation of the fi rst innovation policy programme, technology projects (TEST), in Croatia to identify and explain the main bottlenecks of the pro-gramme and put them in perspective in terms of the entire innovation system. The motivation is the growing criticism of innovation system’s effi ciency and perception of poor return on public investment in innovation policy programmes. The proposed method of social evaluation is based on combination of the institutional theory and the sociological approach of intended and unintended consequences. It reveals that expected outcomes of the programme such as commercialisation of research and tech-nological outputs are exchanged, in the majority of projects, for essentially scientifi c results such as scientifi c publications and extension of the on-going scientifi c projects. The reasons are found in the institutional defi cits that point to the persistence of some common ‘systemic’ problems in the innova-tion system. Their drivers are identifi ed in three types of institutional defi cits: administrative rules and procedures, some aspects of social capital and broader socio-economic environment that com-monly determine behaviour and interaction of the main stakeholders of the programme.

Keywords: innovation policy programme; social evaluation; institutional theory; technology projects; Croatia

INTRODUCTION

The evaluation of science, technology and innovation (STI) policies and related

government-supported programmes is a rela-tively new phenomenon which dates from the late 1960s in the United States and the late 1970s in Europe (Luukkonen 2002; Roessner 2002; Molas-Gallart & Davies 2006). The inter-est for STI policy evaluation is mainly driven by the need of policy makers to legitimise R&D

expenses and to demonstrate the impacts of STI policies on productivity, competitiveness and economic and social welfare in general.

Nowadays, within the ‘broad-based’ innovation policy (Edquist et al. 2009) and ‘systemic approach’ (Smits & Kuhlmann 2004) which underlines inno-vation as a contextual and endogenous process, the performance-based evaluation suffers certain limitations in assessing innovation policy measures (Perrin 2002). It assumes a direct relationship

Jadranka Švarc, Jasminka Lažnjak and Juraj Perkovic

INNOVATION: MANAGEMENT, POLICY & PRACTICE Volume 13, Issue 1, April 201178

between input and output, whereas innovation is mediated through context and interaction with many other activities. Therefore, the evaluation of innovation policy instruments should go beyond standard quantitative measures to more formative approach to perceive wider socio-economic con-texts which determine the outcomes and possible impacts of policy instruments (Kuhlmann 2003, 2006). The new trends in evaluation practice and methods ranging from quantitative to qualitative measures and from summative to formative types of evaluation have been introduced and elaborated by many authors (Georghiou & Roessner 2000; Perrin 2002; Roessner 2002; Kuhlmann 2003; Arnold 2004; Bozeman 2005; Molas-Gallart & Davies 2006).

The main argument for social evaluation of the government-supported innovation pro-grammes comes from the ‘new innovation para-digm’ (Mytelka & Smith 2002: 1469) drawing on the evolutionary and institutional view of innovation which is broadly elaborated by schol-ars like Nelson and Winter (1982), Freeman (1988), Lundvall (1992), Lundvall and Borras (1997), Edquist (1997) and others. In this new context, the ability of businesses to be competi-tive increasingly depends on their capacity to apply new knowledge and innovation shaped by the partnerships and interactivity among many actors of the innovation system, primarily com-panies and research institutes/universities. The underlying theoretical idea was that innovation through interaction occurs in specifi c institu-tional contexts in which government has a critical role by defi ning the ‘rules of the game’ through various incentive measures and polices like legal rules on intellectual property, fi nancial assistance, transfer institutions etc. It has provided a rapid rise to development of the innovation policy pro-grammes to stimulate the commercialisation of publicly funded research and science–industry cooperation. A numerous studies on university-based spin-off companies, university patenting/licensing, public-private joint ventures etc. have been produced to illustrate the evolution of

entrepreneurial activity of both the universities in general and individual scientists in particular. Despite the clash of business and scientifi c culture that is rooted in the classic normative structure of science (Merton 1968) many studies like those of Etzkowitz (1998, 2002), Hsu et al. (2007) Jain et al. (2009), Stuart and Ding (2006), Van Looy et al. (2004) depicted the shifts in scientists’ atti-tudes toward commercial involvement.

Although commercial activity in academia is not without controversy it is commonly perceived that ‘academic entrepreneur’ has now achieved taken-for-granted status in the scientifi c commu-nity (Stuart & Ding 2006) while the notion of the eentrepreneurial university and its ‘third mis-sion’ has become institutionalized within the tri-ple helix model of science–industry– government cooperation (Etzkowitz 2008). Scientists, espe-cially those at prestigious universities seek to capitalize on their research through the different technology transfer activities (e.g. consulting, con-tract research, joint ventures, university spin-offs) and they manly pursue their commercial interests in tandem with their academic work (Etzkowitz 2002). Entrepreneurial scientists ‘ typically adopt a hybrid role identity that comprises a focal aca-demic self and a secondary commercial persona’ (Jain et al. 2009: 923).

Some studies revealed that engagement in entrepreneurial activities usually is not made on the expense of more basic research but coincides with increased publication outputs (Van Looy et al. 2004). The effi ciency of the scientifi c entre-preneurship usually depends on the strength of the national innovation system and its institu-tions to support complex interactions between the actors that play a role in the process of knowl-edge generation and diffusion (Etzkowitz 2008).

The concept of NIS has an astonishing take-up and has been rapidly adopted by the national governments around the world as an analytical framework and practical tool on how to manage innovation processes in local economies (Albert & Laberge 2007). Infl uenced by the similar strategic scenarios within the European Union

https://www.researchgate.net/publication/222694144_Academics_or_Entrepreneurs_Investigating_Role_Identity_Modification_of_University_Scientists_Involved_in_Commercialization_Activity?el=1_x_8&enrichId=rgreq-c2ba9f9e-e5a6-47d6-9499-fc094ee20770&enrichSource=Y292ZXJQYWdlOzI3NDU0NjEyMjtBUzoyMTc4MDY5NTMwMjk2MzJAMTQyODk0MDYzMjE2Nw==

Unintended consequences of the innovation policy programmes in Croatia


complete information about the programme per-formance the collected data we considered suf-fi ciently indicative for the critical analysis of the programme implementation, especially because it is based on fi rsthand experience of the principal participants of the programme.

Social evaluation can be understood as a part of formative evaluation. Its aim was to identify the main bottlenecks of the programme implementa-tion in order to assess the structural defi ciencies or system failures which hinder the effi ciency of the entire innovation system. The motivation is the growing criticism of the innovation system perfor-mance and the perception of poor return on public investment in the innovation policy programmes. It goes beyond standard evaluation methods focused on the quantitative measurement of out-puts since social evaluation takes into account also wider socio-economic, cultural and political con-texts that determine the effi ciency of the system.

Dimensions of social capital like values, trust and norms rarely make a subject of standard eval-uations of innovation policies despite the grow-ing recognition of socio-cultural embeddedness of national innovation capacities (Akcomak & Ter Weel 2009) innovation system (Fagerberg & Srholec 2008) and socio-technical system (Geels 2004).2 Some pioneering studies on social capital in Croatia revealed that Croatia suffers from the defi cit of social capital mainly due to the lack of generalised trust, trust in institutions and percep-tion of corruption (Štulhofer 2004). On the other side, there is also limited civil society develop-ment related to restrictive legislation and a weak of social responsibility on the part of the corpo-rate sector (Bežovan 2003).

Within our social evaluation we tried to pro-vide the insight into the general value orientation, trust and other relevant attitudes of researchers – project leaders – related to the TEST programme.

depicted in the Lisbon agenda (European Council 2000), the Croatian government launched the fi rst innovation policy programme, ‘Programme for Innovative Technological Development’ (HITRA), at the beginning of 2001. It has marked a turning-point in science policy since it presents the fi rst deliberate and institutionalised action of public administration to connect science and eco-nomic development (Švarc 2006). Although the initial two sub-programmes, the RAZUM, aimed at supporting knowledge-based companies, and the TEST, focused on developing new technolo-gies, have been running for almost eight years, standard performance-based evaluations to reveal the outputs in terms of patents, start-up compa-nies, jobs, royalties, etc. have not been performed or presented to a wider public. The lack of evalu-ation studies produced negative feedback on the programmes themselves and innovation policy as a whole and has given rise to disputes about the transparency of the programmes, their effi ciency, socio-economic impact, grant policy, and even corruption of the administration and researchers.

Therefore, an attempt to develop a method of social evaluation of the TEST programme has been carried out in order to observe the effi -ciency, position and functioning of the innova-tion policy in Croatia.1 The evaluation has been limited only to the TEST programme because it had suffi cient number of publicly available data accessible for the statistical analysis (212 projects) via the Ministry web site. Unlike TEST, the num-ber of projects in the RAZUM programme was relatively small (ca. 30), still in progress and not publicly available because the business companies were the principal stakeholders.

The social evaluation is based on the analyses of the data collected by questionnaire and by this limited to the perception of the project leaders. Although the perception does not provide the

1 The empirical results presented in this paper are part of the wider research project on social evaluation of the Croatian innovation system fi nanced by the Ministry of Science, Education and Sports (MSES) in the period 2005–2009.

2 For an overview of research on innovation culture and the differences in innovation culture across Europe see the Transform project (Didero et al. 2008).

https://www.researchgate.net/publication/223572230_Socio-political_factors_and_the_failure_of_innovation_policy_in_Croatia_as_a_country_in_transition?el=1_x_8&enrichId=rgreq-c2ba9f9e-e5a6-47d6-9499-fc094ee20770&enrichSource=Y292ZXJQYWdlOzI3NDU0NjEyMjtBUzoyMTc4MDY5NTMwMjk2MzJAMTQyODk0MDYzMjE2Nw==



set of unintended and failed consequences, which shifted the focus of the TEST programme from research commercialisation and science–industry cooperation (intended consequences) towards standard research projects which mainly serve the scientifi c interests of researchers (undesirable unintended and failed consequences).

RESEARCH DESIGN, METHODOLOGY AND SAMPLEThe social evaluation of Croatian innovation policy is an ex post evaluation performed only on the one innovation policy instrument – the TEST programme. The TEST programme was the fi rst government programme for direct incentives and fi nancing of research activities in both public and private sectors focused on development of new technologies (products, processes, services) up to the stage of original solutions (prototype/pilot stage). Although the overall goals of the HITRA programme were defi ned broadly in the strategic document (MOST 2002a) in terms of technol-ogy development and revival of industry, the Directives for the implementation of the HITRA Programme (MOST 2002b)3 emphasised that the research results of the programme are expected to be commercialised through patents/licences, strategic partnership with companies, new spin-offs/start-up companies within the RAZUM programme or will be exploited in other ways. Besides, the TEST programme has been assigned a pioneering role to establish a new institutional framework, i.e. the legal basis, procedures and rules for supporting science-driven innovations and science–industry cooperation. In addition, a new ministerial body, the Technology Council was established in 2001 to administer and man-age the TEST programme.

These project leaders, being the fi rst in Croatia who applied for technology-oriented projects, are perceived as the agents of socio-cultural and institutional change. Therefore, the investigation of their motives and benefi ts, their attitudes about the programme, commercialisation of science, etc. provides a valuable contribution for analysing the positively inclined socio-cultural environment of the current innovation policy.

This social evaluation of innovation policy draws on the combination of two theoretical back-grounds. The fi rst is the theory of unintended con-sequences (Merton 1936) which states that social actions in a complex system creates intended and unintended consequences. Unintended conse-quences are the outcomes that are not originally planned and could be either benefi cial or dysfunc-tional for a designated system.

The next theory is the concept of institutional defi cits developed by Edquist (1997: 24–26; 2001). It considers institutions to be crucial ele-ments in all versions of the systemic innovation approach: the confi guration and relationships among organisations (consciously created formal structures such as funding agencies) and institu-tions (sets of common habits, routines, established practices, rules, or laws) vary signifi cantly with the national innovation systems causing differences in their performance and success. Similarly to the the-ory of market failures, which tends to eliminate the defi ciencies of markets in generating innovation, the institutionalist approach tends to reveal insti-tutional failures linked to the formal institutions (rules and regulations) and informal institutions (socio-political context, value orientations, norms), as well as their interaction with organisations.

The research starts from the thesis that imple-mentation of the TEST programme produced a

3 All offi cial documents of the HITRA programme were published in the booklet ‘Collected programmatic papers’ (MOST 2002c) and consists of the (1) Croatian Program for Innovative Technological Development (HITRA) (strategic document), (2) Directives for the Implementation of the HITRA Program, (3) Regulation on the Procedure for the Implementation of the Program for Development of Knowledge-Based Companies, (4) Agreement on the Cooperation and implementation of the HITRA with research institutions, (5) public calls for project proposals. All the documents were adopted by the Government of Croatia on April 5, 2001.

https://www.researchgate.net/publication/243680747_Merton_R._The_Unanticipated_Consequences_of_Purposive_Social_Action.American_Sociological_Review?el=1_x_8&enrichId=rgreq-c2ba9f9e-e5a6-47d6-9499-fc094ee20770&enrichSource=Y292ZXJQYWdlOzI3NDU0NjEyMjtBUzoyMTc4MDY5NTMwMjk2MzJAMTQyODk0MDYzMjE2Nw==



from expected results towards standard research projects which mainly serve the scientifi c interests of researchers and cause so far low performance of the TEST programme.

The consequences were operationalised as a set of indicators grouped in nine dimensions of the TEST programme (Table 1). The data were col-lected by a self-administrated questionnaire sent by mail and with return control. It consisted of 53 questions grouped in the six thematic categories that enable measurement of the selected dimen-sions of the TEST programme. The respondents were exclusively project leaders who provided the information on requested data about projects and also about their motivation, attitudes and value orientations. The standard Likert scale (1 – Strongly disagree; 2 – Disagree; 3 – Cannot decide; 4 – Agree; 5 – Strongly agree) was used to mea-sure the statements concerning attitudes and value

The barriers to successful programme imple-mentation are captured indirectly by measuring intended and unintended outcomes of the TEST programme. In this research, the intended or expected consequences of the programme are those outputs which meet the requirements of the offi cial goals of the programme set out in the ‘Directives’ (MOST 2002b) that are originally formulated such as: ‘quick and effect support to applied and development research relevant for direct use in the industry and economy’, ‘stimulation of scientists to initiate projects that may encourage additional investments by the industry’, ‘permanent coopera-tion between scientifi c and industrial sector’, etc. Such goals are, obviously related to the production of new technologies, research commercialisation and science–industry cooperation. Conversely, the unintended and failed consequences are those out-puts that shifted the focus of the TEST programme

TABLE 1: THE LIST OF INDICATORS FOR MEASURING THE SELECTED DIMENSIONS OF THE TEST PROGRAMME

Dimensions of the IndicatorsTEST programme

1. Motivation • The origin of the project idea (researcher, entrepreneur, industrial partner); • Additional motives to apply besides developing new technology;

2. Financing • Sources of project funding; • Budget spending;

3. Realisation • Main outputs ( reports, feasibility studies, patents, scientifi c papers); • Follow-up activities; • Benefi ts for the researchers;

4. Quality performing • Satisfaction with results of the project; • Control over the budget spending; • Dedication to commercialisation;

5. Feasibility • Reasons for commercialisation failure; • Willingness to submit new project

6. Impact of the TEST • List of statements about infl uence of TEST on transition towards on the changing role of science Mode 2 of science production (Likert scale)

7. Impact of TEST’s new • List of statements about new measures of project application, individual instruments on evaluation, monitoring (Likert scale)standard policy

8. Attitudes towards science • List of statements on commercial application, science–industry commercialisation and science– cooperation and general value orientation (Likert scale)industry cooperation

9. General value orientations • Trust in institutions • Egalitarianism, statism (Likert scale)



biotechnical sciences and biomedicine (28%) and 14% are affi liated to natural science and 10% to agronomy and the veterinary fi eld. Some of the respondents (7.5%) possess their own company and some of them (9.2%) are working parallel in another fi rm or institution. The sample illustrates that some Croatian researchers are quite familiar with entrepreneurship.

RESULTS AND DISCUSSION: INTENDED AND UNINTENDED CONSEQUENCES AS AN ELEMENT OF FORMATIVE EVALUATIONThe social evaluation of intended and unintended consequences reveals three categories of policy effects of the TEST programme:1. Realised intended consequences;2. Failed intended consequences;3. Unintended consequences.

Realised intended consequencesThe following fi ve results have been identifi ed as the expected and intended consequences of the TEST programme. They are mostly derived from the analysis of the policy documents (MOST 2002c) and partly from the interviews with the policy makers and researchers:1. Fulfi lment of the formal outputs of the

programme;2. Share of projects with commercialised research

results;3. Strengthening of research capabilities for

cooperation with industry;4. Impact of TEST on the changing role of

science;5. Impact of particular measures of TEST on the

standard science policy.

The analysis revealed that all the projects have fulfi lled the three formal requirements of the

orientation (dimensions 6–9 in the Table 1). To process the data we have used the SPSS 13.0. The missing values were excluded from the analysis.

In addition to the questionnaire, a brief analy-sis of the Collected programmatic papers for the HITRA programme (MOST 2002c) that deter-mine the strategic goals as well as the guidelines for the implementation of the Programme was carried out in order to identify the infl uence of formal procedures and rules on the success of the programme. Finally, a total of 12 semi-structured interviews have been conducted, audio-recorded and transcribed in order to understand the ratio-nales of the programme from both the policy makers and researchers. The interviews involve so far nine projects leaders and three policy makers – the members of the Technological council. The future research will be extended also by the repre-sentatives of the companies.

The research was conducted through a question-naire-based survey in the winter of 2007 targeting the 212 project leaders4 who have taken part in the TEST programme and have successfully completed the projects till 2005. In the period 2001 to 2007 there were 605 applied projects, of which 298 were accepted for fi nancing. Therefore, the sample is a self-selected purposive sample of successful appli-cants to TEST projects. One hundred and twenty researchers responded to the questionnaire, giving a response rate of almost 57%.

The information about the projects and proj-ect leaders has been identifi ed from the web-based Inventory of the TEST projects provided by the Ministry of Science, Education and Sports (MSES). The majority of respondents in the sample are from university departments (76%), whereas only 17% are from public research insti-tutes. Some 65% of respondents are male and 35% are female researchers. A large majority of them (93.3%) have a PhD in science. Most of them come from technical sciences (45%),

4 All the project leaders of the TEST programme were coming from the public research institutes and universities since the offi cial rules and procedures of the TEST programme allowed the business companies to submit project proposals only through the public research institutes.



6% of projects were used for submission of a new project proposal to the HITRA–RAZUM pro-gramme aimed at establishing a start-up company based on research results. Also, 25% of projects were declared as being used as a basis for new col-laborative projects with business. Given that com-mercialisation is an extremely complex process with usually only a modest success rate, it could be stated that 16% of projects with commercial results provide a platform for justifi cation of the TEST programme as the initial and pioneering innovation policy programme.

The strengthening of research capabilities for cooperation with industry is the next positive output of the TEST as revealed by the analysis of benefi ts by researchers participating in the TEST programme (Figure 2). Researchers estimated that their main benefi ts consist in testing new ideas (M = 4.06), new contacts with companies (M = 3.63) and experience in cooperation with industry (M = 3.48).

Extra material resources and procurement of new equipment are also highly appreciated by

programme specifi ed in the forms on fi nal achieve-ments on the project: (a) fi nal report of project which assumes standard description of research project implementation (72.5% of projects), (b) feasibility studies (20% of projects) and (c) sub-mission of the new project proposal to the pro-gramme RAZUM5 (7.5%). Since these ‘offi cially required’ results do not say much about com-mercialisation or technological application of the project after its completion, a new question with a scale of nine possible follow-up activities has been posed. Three of the follow-up activities were categorised as commercial use of project results: (1) sale of patent/licence, (2) commercialisation through partnership with existing companies and (3) commercialisation in another way (e.g. online education (Figure 1).

Although the analysis revealed that the major-ity of project results have been used for continua-tion of the scientifi c project which was, in essence, a failed intended consequence, about 16% of projects still declared commercialisation in one of the three given ways. In addition, more than

FIGURE 1: FOLLOW-UP ACTIVITIES (WHAT HAVE THE PROJECTS’ RESULTS BEEN USED FOR?) (in percentage, multiple choice of answers was allowed)

5 The RAZUM programme is the follow-up phase of the TEST programme aimed at establishing a start-up company/research spin-off.



Finally, the fi fth positive result is related to the individual policy measures introduced by the TEST programme which are complete novelties in traditional science policy and aimed at higher standards in project evaluation and monitoring (Figure 4). Detailed fi nancial reporting (with copies of relevant documents), public defence of project proposal and permanent monitoring by the expert groups nominated by the MSES were largely appreciated. They are followed by other measures such as two-phase project applications which sort out valuable ideas for full application procedures, frequent reporting in three-month cycles, return of 21% of income to MSES in case of successful commercialisation, etc.

Failed intended consequencesRegardless of the positive results of the TEST pro-gramme, some measures have not produced the expected effects. One of the main incentives for this evaluation was directed toward identifi cation of such failures and the underlying reasons and to fi nd the main obstacles to programmes’ successful implementation. Four failures of the TEST pro-gramme have been indentifi ed:1. Programme fails to mobilise companies and

entrepreneurs;

researchers. The analyses of the follow-up activi-ties (Figure 1) and use of the equipment (Figure 8) revealed that these resources were actually used after project completion for the continuation of research projects, i.e. for scientifi c purposes, not for commercialisation.

The fourth positive result of the TEST pro-gramme is related to its socio-cultural impact on the role of science. It contributed to the transition of dominant ‘elite-type’ science in Croatia towards the productive-type known generally as the Mode 2 of science production (Gibbons et al. 1994). As a consequence, a standard science policy was extended with the innovation policy based on the triple helix model (Etzkowitz 2003). The analysis reveals (Figure 3) that respondents have recognised the TEST as a promoter of commercialisation of scientifi c research, change in the traditional role of science and cooperation between research and industry. They noticed, however, that TEST suf-fers serious shortcomings such as the lack of infra-structure needed for commercialisation of research results (e.g. technology transfer centres) and the lack of fi nal users (companies and industry) made it not fully functional. They are also not sure about its cost-effi ciency, i.e. the return on investment in the programme is doubtful.

FIGURE 2: BENEFITS FROM PARTICIPATION IN THE TEST PROGRAMME (MEAN∗)*1 – STRONGLY DISAGREE; 2 – DISAGREE; 3 – CANNOT DECIDE; 4 – AGREE; 5 – STRONGLY AGREE



FIGURE 3: ATTITUDES TOWARDS TEST PROGRAMME AS A POLICY PROGRAMME (MEAN∗)*1 – STRONGLY DISAGREE; 2 – DISAGREE; 3 – CANNOT DECIDE; 4 – AGREE; 5 – STRONGLY AGREE

FIGURE 4: ATTITUDES TOWARDS INDIVIDUAL MEASURES OF THE TEST PROGRAMME (MEAN∗)*1 – STRONGLY DISAGREE; 2 – DISAGREE; 3 – CANNOT DECIDE; 4 – AGREE; 5 – STRONGLY AGREE

2. Project results serve mainly scientifi c, not commercial purposes;

3. Programme fails to attract investments from companies;



Since the participation of companies in the TEST programme was rather modest, the share of their co-fi nancing of projects was also fairly small (Figure 5). More than 62% of projects have not received any funds from fi rms, whereas 26% of projects have received some fi nancial means up to the amount of 30% of total projects costs. Only about 12% of projects were co-fi nanced by fi rms with more than 30% of project total costs. That roughly corresponds to the percentage of around 16% of commercialised projects. Respondents also indicated that 54% of projects were fi nanced only by the MSES, whereas the next 38% received more than 50% of funds from the MSES. A small share of fi nancial resources was provided by agen-cies and local governments. The analysis of the budget resources shows that the programme fails to attract investments from companies.

The next failed consequence of the TEST pro-gramme was that commercialisation of research results has not been in the focus of research-ers who have submitted projects. Only 17% of respondents have developed concrete plans for commercialisation prior to project submission. The remaining 83% have had vague ideas of com-mercialisation, of whom 46% were just ‘thinking about commercialisation’, whereas another 35% expressed intentions to commercialise research results, but had developed no concrete plans (Figure 6).

The majority of those who had developed some kind of commercialisation strategy based their commercialisation plans on the follow-ing (Figure 6): (a) extension of the contract research with industry (21%), (b) developing business services like testing or quality control (16.7%), (c) launching production either with strategic partner (11.7%) or with the compa-nies where researchers work part time (5%). A few respondents wanted to sell a patent/licence and launch their own company. Almost half of respondents (49%) did not respond to this question at all, stressing, thus, that com-mercialisation was not really a priority for the majority of respondents

4. Commercialisation of research results has not been the focus of submitted projects.

The target groups of the TEST programme were both researchers and entrepreneurs who wanted to develop or test new ideas with com-mercial potential within the cooperative projects. It was expected that a signifi cant number of the project proposals would come from the compa-nies since they can benefi t not only from public grants but also from research expertise, equip-ment and knowledge at universities, institutions and government labs. Although companies are always eager to collect public money, it turned out that the majority of projects were initiated by ideas coming from researchers (73.3%) and were in essence the extension of scientifi c projects fi nanced by the government. Only about 20% of ideas are generated by industrial partners or entrepreneurs from small companies searching for technological solutions or concrete applications. The fi gure roughly corresponds to the percent-age of commercialised projects as presented in the analysis of the follow-up activities (Figure 1). The follow-up activities also illustrate that project results serve mainly scientifi c and not commercial purposes since 37.5% of projects were used for the continuation of scientifi c projects and another 27.5% have not been used for any of the listed follow-up activities. That makes a high propor-tion of projects (65%) which have met formal cri-teria of TEST ( potential commercialisation), but were not continued to the more concrete phase of technological application or commercialisation.

Also, the analysis of published results within TEST projects shows that scientifi c outputs largely outnumbered technological outputs like pat-ents. Eleven respondents protected their research results with patents and produced altogether 21 registered patents. For comparison, scientifi c outputs include 417 published works, primar-ily scientifi c papers (66%) published in foreign (48%) and Croatian journals (18%). Professional papers contribute with 28% and books with 6% to total scientifi c publishing.



2. The TEST programme has fi nanced mainly standard research activities, not technological outputs and cooperation with industry.

The ambiguity of motives was identifi ed by asking the respondents about their ‘additional intentions to apply for the TEST project besides presumed technological results’ (Figure 7). It turned out that motives of researchers to apply for TEST projects were split between three main interests: to develop additional experience of cooperation with industry (65.8% of respondents), to buy new equipment (64.2% of respondents) and to gain additional fi nancial means for scientifi c research (50% of respondents).

The combination of such mixed motives indi-cates the ambiguous way researchers understand the TEST programme. Their prime interest was, in essence, to secure additional funding and equip-ment for their scientifi c research, which was the raison d’être of their vocational ethos and profes-sional existence. The analysis of budget spending supports this fi nding since the majority of project budgets was spent on new equipment (majority of respondents or 23.4% have spent more than 50% of the budget on equipment; Table 2). The remain-ing research grants were spent primarily on mate-rial and personal costs, whereas the least amount was spent on business trips and cooperation with companies. This illustrates that cooperation

Unintended consequencesAlthough the intention of the policy makers when designing the programme was to promote commercialisation and immediate industrial application (that was quite clear from the offi cial documentation), yet some rules of the programme were defi ned broadly (e.g. request for pre-com-mercial projects in the public call) which allow ambiguous understanding of the programme’s requests from the applicants point of view. As a consequence, the two unintended consequences of the programme were identifi ed, as follows:1. The motives for participation in the TEST

programme were ambiguous and mainly science-driven;

FIGURE 5: THE STRUCTURE OF THE PROJECTS’ BUDGET RESOURCES

FIGURE 6: INTENTIONS AND PLANS FOR COMMERCIALISATION (% OF RESPONDENTS)



commercialisation of project results. The majority of TEST projects were caught in a backward loop, going from scientifi c ideas to technology results and back to scientifi c activities. The exchange of technological outputs for essentially scientifi c results appeared as the common feature of many projects pointing, thus, to the persistence of some problems in the innovation system. The origin of these systemic problems can be found in the institutional frameworks that determine behav-iour and interaction of the main stakeholders of the programme. In the light of the analysis of the questionnaire and offi cial documentation of the TEST programme (public call, application rules, etc.), the three main institutional defi cits can be identifi ed, as follows:1. Administrative defi cits;2. Lack of social capital;3. Defi ciency of the broader socio-economic

environment.

1. Administrative defi cits are recognised as formal institutional defi cits, which include the offi cial rules and procedures of the TEST programme defi ned by the offi cial documentation (MOST 2002c). They suffer a rather important defi cit which discourages

with industry and commercialisation was a wel-come ingredient, but rarely the prime motive of researchers for participating in the programme.

Since the analysis of follow-up activities (Figure 1) revealed that the majority of projects ended up as continuations of scientifi c projects, it is fair to say that the grants for technology projects were mainly used for additional fi nancing of standard research and educational activities. Almost all projects (93%) used the purchased equipment for scientifi c research and education, whereas only 19% of projects used it also for commercial purposes. More than a half of purchased equipment (52%) is shared among all research staff within research institutions and treated as a common asset, which is typical of the scientifi c ethos. Only 19% of projects use the equipment only within project teams (Figure 8).

FROM SOCIAL EVALUATION TO INSTITUTIONAL DEFICITSThe social evaluation of the TEST programme has identifi ed a range of unexpected and failed results, which points to bottlenecks in the implan-tation of the programme. The bottlenecks could be summarised as low mobilisation of companies to take part in technological projects and weak

FIGURE 7: ADDITIONAL MOTIVES BESIDE DEVELOPMENT OF NEW TECHNOLOGIES (% OF RESPONDENTS)

Getting start-up capital to establish afirm

2,5

10,8

18,3

50

64,2

65,8

0 20 30 40 50 60 70

Improving patent/licence for selling

Financing research cooperation with acompany

Additional financial means for scientificwork

Experience of cooperation with industry

Buying equipment

2,5

10,8

18,3

50



entrepreneurship in the domain of MELE. It also signifi cantly reduced the potential of inno-vation policy to provide a common ground for harmonisation of government policies towards knowledge-based economy. The simple lack of a coordinated innovation system illustrates also the low ability of governance and public administra-tion to manage innovation processes and the cor-related economic development.2. The lack of social capital, recognised as infor-mal institutional defi cits, is one of the serious obstacles to an effi cient NIS and has been analysed in the previous papers on the Croatian NIS (Švarc et al. 2009). The standard indicators of social capital such as values, norms and trust in institu-tions were operationalised and measured through four aspects: (1) attitudes toward traditional and entrepreneurial universities; (2) attitudes toward commercialisation of science; (3) general value orientations; and (4) trust in institutions.

participation of companies/entrepreneurs in the programme. It is related to the rule that project leader should be only from research institute or university registered by the MSES. Under this rule, research organisations are the main fi nancial con-tractor, which bears all the responsibilities for proj-ect implementation whereas companies are reduced to second-hand participants, which deterred many companies from participation.

This rule is also a consequence of a frag-mented innovation policy, since the innova-tion policy in Croatia is still remarkably defi ned by the division of competences between the MSES and the Ministry of Economy, Labour and Entrepreneurship (MELE). The lack of a common policy platform and communication made the MSES restrict their policy measures to science-driven innovation based on the scientifi c capacities of the public research sector and avoid interfering with the business-led innovation and

FIGURE 8: USAGE OF THE PURCHASED EQUIPMENT (%, MULTIPLE CHOICE IN ANSWERING WAS ALLOWED)

On the disposal only toresearch team

18

52

19

93

On the disposal to theinstitution

Commercialisation

Research and education

0 20 40 60 80 100

TABLE 2: STRUCTURE OF THE PROJECT BUDGET EXPENDITURE

Range (%) Business Material Equipment (%) Personal Collaborative trips (%) costs (%) costs (%) fi rm (%)

50.1–100 0 10.9 23.4 10.8 0

30.1–50 0.8 19.2 15 20.8 2.5

10.1–30 20.8 36.7 25.8 32.5 5.8

0.1–10 53.3 25.8 23.3 19.2 13.3



3. Defi ciencies in the broader socio-economic environment are identifi ed as the lack of institu-tions for knowledge transfer and the lack of busi-ness demand for innovation and cooperation with research sectors. It is clearly shown in the answers to the question which explicitly asked respon-dents to identify the barriers to commercialisation (Table 3). The most relevant barrier is the lack of business partners, since 32.5% of respondents think that the lack of business partners is the rea-son for the failure of commercialisation ‘greatly’ and ‘very greatly’. The next barriers are the lack of support in the environment (30%) and the lack of business opportunities (30%).

The previous analysis of social capital and innovation system in Croatia (Švarc et al. 2009) also revealed that the defi cit of technology trans-fer infrastructure (e.g. university offi ces for tech-nology transfer) and lack of interest of industry in cooperation with research organisations are perceived as the main obstacles to more intensive science–industry cooperation and commercial use of science (Table 4). These factors are perceived as more important obstacles than the dominant culture of elitist science focused on scientifi c pub-lishing and ethical principles of scientifi c freedom distant from application. The lack of necessary infrastructure is also emphasised as an important shortcoming of the TEST programme in the gen-eral statements about the TEST as a new policy programme (Figure 3). The respondents think that the most minor problem of commercialisa-tion is the low quality of science.

Finally, it is worth mentioning that 30% per-cent of respondents stated that they would not apply for TEST again mainly because of the dif-fi culties in commercial implementation of the results and collaboration with fi rms (Figure 9).

CONCLUSIONThe social evaluation of the TEST programme based on the analysis of intended and unintended consequences indicates that innovation policy in Croatia does not work well in the given socio-eco-nomic and cultural circumstances and institutional

The analysis revealed that participants in the TEST programme, although recognised as the agents of change towards science–industry coop-eration and commercialisation of research, share the rather traditional values such as statism, anti-globalism and egalitarianism, also typical of the wider Croatian population. As expected, this gen-eral value system is accompanied by the greatest trust in science and the education system and the least trust is placed in the judicial system, political parties and public administration. Statism refers to the traditional paternalistic role of the state (not a mediator between the stakeholders) while egalita-rism refers to the re-distributive ethics and anti-entrepreneurial spirit (Županov 1987). Therefore, the traditional values discourage individual initia-tive, willing to take risk, cooperation and network-ing which build social cohesion needed for the effi cient innovation system in Croatia. The lack of social cohesion is also sustained by the lack of con-fi dence in public administration and their policy programmes such as TEST, meaning that research-ers cannot be adequately mobilised to follow gov-ernment actions for fostering science–industry cooperation. Although they prefer the entrepre-neurial university, which is in essence based on the Triple helix paradigm and the Mode 2 of sci-ence production, they are more inclined toward the protectionist role of government that would protect them from market competition and glo-balisation processes. The strong value orientation towards statism seems to be in contradiction with their preferences for the commercialisation of sci-ence and the entrepreneurial university. Statism is probably driven, however, by the lack of a research market, which simply does not work in the pres-ent socio-economic circumstances. The reason is the lack of industrial research centres, which were destroyed during the privatisation process, and a lack of companies interested in research since they have largely lost their technology capabilities during transition. The only way to keep science alive is through the protective role of the state and by maintaining a restricted and closed market as refl ected in the value orientation of respondents.



the broader socio-economic environment in terms of the lack of knowledge transfer institutions and weak business demand for innovation.

The institutional defi cits are rooted in the broader socio-cultural and economic environment and cannot be overcome by mere redesign of the single innovation policy programmes. In contrast, the entire policy and institutional context should be changed in order to attain development based on knowledge and innovation that is essentially the long-term goal of the TEST programme. Within the given context, the most important outcome of the TEST programme consists, instead of impact on technological development, in bringing socio-cultural change in understand-ing the new commercial role of science, which can be described also as transition from the standard (Mode 1) to the Mode 2 of knowledge produc-tion (Gibbons et al. 1994). It also highlighted the principles of the triple helix model in Croatia (Etzkowitz 2003).

The TEST programme has introduced the institutions, which made a ground for the national innovation system in Croatia and started the irreversible processes of market orientation of science. However, after the seven years of expe-rience in innovation policy it is high time for reviewing the achievements and for making the adequate breakthrough in institutional frame-work needed for accelerating economic growth based on knowledge and innovation.

set up. Instead of expected technological outputs with possible commercial effects, the majority of TEST projects were caught in a backward loop, going from scientifi c ideas to technology results and back to scientifi c activities. The explanation of the bottlenecks of the implementation of the TEST programme visible in this backward loop draws on the institutionalist approaches developed by Edquist (1997: 24–26; 2001).

This points to the persistence of some common or system problems, which are ascribed to the three institutional defi cits that determine behav-iour and interaction of the main stakeholders of the TEST programme, as follows: (1) administra-tive defi cits (formal institutions), which consists of narrowly defi ned programme procedures and rules which have not attracted entrepreneurs, (2) pat-tern of researchers’ behaviour (informal institu-tions) driven by value orientations and norms determined by the ambiguity between scientifi c ethos, on the one side, and the lack of research market, on the other side: and, (3) defi ciency of

TABLE 3: THE BARRIERS TO RESEARCH COMMERCIALISATION

Answers ‘greatly’ + ‘very greatly’ in %

Lack of business partner 32.5

Lack of support in the 30environment

Lack of business 30opportunities (interests)

Lack of fi nancial resources 20.9

Occupation with other 24.3job (teaching)

Lack of potential market 20

Lack of management 16.7capabilities

Uncoordinated legal rules 15between ministries

Lack of pursuing 14.2commercialisation

Idea was not 14.2profi table enough

Intellectual property rights 4.1

TABLE 4: ATTITUDES TOWARD COMMERCIALISATION OF SCIENCE

M∗

Industry is not interested in 3.8739cooperation with science

Science in Croatia is dominated 3.7871by elite-type of science

Science in Croatia is inert and 3.4804of low quality

Technology transfer infrastructure 4.1134is defi cient

∗1 – Strongly disagree; 2 – Disagree; 3 – Cannot decide; 4 – Agree; 5 – Strongly agree.



Albert, M. and Laberge, S. (2007) The legitimation and dissemination process of the innovation system approach: The case of the Canadian and Quebec science and technology policy, Science, Technology & Human Values 23(2), 221–249.

Arnold, E. (2004) Evaluating research and innova-tion policy: A systems world needs a systems evaluation, Research Evaluation 13(1), 3–17.

Bežovan, G. (2003) Indicators of civil society development in Croatia, Društvena Istraživanja 12(3–4): 495–518 (in Croatian).

Bozeman, B. (2005) Public value mapping of science outcomes: Theory and method. A Monograph of the public value mapping project, Center for Science, Policy and Outcomes, Washington, DC. Columbia University.

Didero, M., Gareis, K., Marques, P. and Ratzke, M. (2008) Differences in innovation culture across Europe – A discussion paper (available on line: http://www.transform-eu.org/publications/doc-uments/Differences%20in%20Innovation%20Culture.pdf. (Last access: June 2, 2010).

Edquist, C. (1997) Systems of innovation approaches – Their emergence and charac-teristics. In: Edquist, C. (Ed.), Systems of innovation – Technologies, institutions and orga-nizations, pp. 41–63, Pinter/Cassell Academic, London.

It assumes also a change of the current ‘science-push’ and ‘supply-side’ innovation policy towards more down-to-earth policy measures, which build on existing technological, socio-cultural and gov-ernance capacities. From the strategic perspective, the focus of such policy should be transferred from research-driven innovations and transla-tion of research into business to other areas seri-ously neglected in Croatia that generate structural impediments to innovation. Such areas include low technology capabilities of companies, weak international competitiveness and regional com-petences, neglect of the new service sectors, etc. A fresh model of innovation policy and manage-ment, which assumes a combination of supply-driven policy at the national level and user-driven regional innovation policies based on clusters, is worth exercising. An intensive process of policy learning and management of innovation is essen-tial for such strategic changes.

ReferencesAkcomak, S. and Ter Weel, B. (2009) Social

capital, innovation and growth: Evidence from Europe, European Economic Review 53, 544–567.

0 2 4 6 8 10 12 14 16 18 20

Difficulties in commercial implementation ofthe results

19,2

13,3

12,5

10,8

10

9,2

8,3

0,8

To much effort concerning collaboration withfirms

Taking away the time for scientific work

To much effort concerning project preparation

To much effort in delivering the results

Not useful for my scientific promotion

Other

I have no idea at the moment

13,3

12,5

10,8

10

9,2

8,3

0,8

FIGURE 9: THE REASONS NOT TO PARTICIPATE IN THE TEST PROGRAMME AGAIN (%)



Hsu, D. H., Roberts, E. B. and Eesley, C. E. (2007) Entrepreneurs from technology-based universities: Evidence from MIT, Research Policy 36, 768–788.

Jain, S., Georg, G. and Maltarich, M. (2009) Academics or entrepreneurs? Investigating role identity modifi cation of university scientists involved in commercialization activity, Research Policy 38, 922–935.

Kuhlmann, S. (2003) Evaluation as a source of strategic intelligence, In: Shapira, P. and Kuhlmann, S. (Eds.), Learning from science and technology policy evaluation, pp. 352–381. Edward Elgar, Cheltenham.

Kuhlmann, S. (2006) Evolution of research evalua-tion, Lecture in R&D Evaluation Course 2006. University of Twente, The Netherlands.

Lundvall, B. A. (1992) National system of innova-tions: Towards theory of innovation and interactive learning, Pinter, London.

Lundvall, B. A. and Borras, S. (1997) The globaliz-ing learning economy: Implications for innova-tion policy. Report based on contributions from seven projects under TSER Program. DG XII, Commission of the European Union.

Luukkonen, T. (2002) Research evaluation in Europe: State of the art, Research Evaluation 11(2): 81–84.

Merton, R. K. (1936) The unanticipated conse-quences of purposive social action, American Sociological Review 1(6), 894–904.

Merton, R. K. (1968) Social theory and social struc-ture, Free Press: New York.

Molas-Gallart, J. and Davies, A. (2006) Toward theory-led evaluation: The experience of European science, technology, and innovation policy, American Journal of Evaluation 27(1), 64–82.

MOST (2002a) Croatian program for innova-tive technological development (HITRA), The Ministry of Science and Technology of the Republic of Croatia, Zagreb.

MOST (2002b) Directives for the implementation of the HITRA Programme, The Ministry of Science and Technology of the Republic of Croatia, Zagreb.

MOST (2002c) Collected programmatic papers of the HITRA programme, The Ministry of Science and

Edquist, C. (2001) The systems of innovation approach and innovation policy: An account of the state of the art, Lead paper presented at the DRUID Conference, Aalborg, June 12–15, 2001.

Edquist, C., Luukkonen, T. and Sotarauta, M. (2009) Broad-based innovation policy, Ministry of Education: Evaluation of the Finnish National Innovation System – Full Report, Helsinki University Press, Helsinki.

Etzkowitz, H. (1998) The norms of entrepre-neurial science: Cognitive effects of the new university-industry linkages, Research Policy 27, 823–833.

Etzkowitz, H. (2002) MIT and the rise of entrepre-neurial science, Routledge, London.

Etzkowitz, H. (2003) Learning from transition: The triple helix as an innovation system, In: Švarc, J., Lažnjak, J., Šporer, Z. and Polšek, D. (Eds.), Transition countries in the knowledge soci-ety: Socioeconomic analysis, pp. 39–61, Institute of Social Sciences, Ivo Pilar, Zagreb.

Etzkowitz, H. (2008) The triple helix univer-sity–industry–government Innovation in action, Routledge, London.

European Council. (2000) Conclusions of the Presidency of the Lisbon European Council of 23 and 24 March 2000, Council document 100/1/00, European Council, Brussels.

Fagerberg, J. and Srholec, M. (2008) National innovation systems, capabilities and economic development, Research Policy 37, 1417–1435.

Freeman, C. (1988) Japan: A new national innova-tion system, In: Dosi, G. et al (Eds.), Technical change and economic theory, pp. 330–349, Pinter Limited, London.

Geels, F. W. (2004) From sectoral systems of inno-vation to socio-technical systems, insights about dynamics and change from sociology and insti-tutional theory, Research Policy 33, 897–920.

Georghiou, L. and Roessner, D. (2000) Evaluating technology programs: Tools and methods, Research Policy 29, 657–678.

Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P. and Trow, M. (1994) The new production of knowledge: The dynamics of science and research in contemporary societies, Sage, London.



Štulhofer, A. (2004) Perception of corruption and the erosion of social capital in Croatia 1995-2003, Politiška Misao 41(5), 74–86.

Švarc, J. (2006) Socio-political factors and the failure of innovation policy in Croatia as a country in transition, Research Policy 35(1), 144–159.

Švarc, J., Lažnjak, J. and Šporer, Z. (2009) Social capital and innovation policy in Croatia: Scientifi c community as a source of innova-tion, In: Tripp, G., Payne, M. and Diodorus, D. (Eds.), Social capital, pp. 15–48, Nova Science, New York.

Van Looy, B., Ranga, M., Callaert, J., Debackere, K. and Zimmermann, E. (2004) Combining entrepreneurial and scientifi c performance in academia: Towards a compounded and reciprocal Matthew-effect?, Research Policy 33, 425–441.

Županov, J. (Ed.). (1987) Egalitarism and industri-alism. In: Sociology and self-management, p. 281, Školska knjiga, Zagreb (in Croatian).

Received 23 April 2010 Accepted 02 November 2010

Technology of the Republic of Croatia, Zagreb, p. 78.

Mytelka, L. K. and Smith, K. (2002) Policy learning and innovation theory: An interactive and co-evolving process, Research Policy 31, 1567–1479.

Nelson, R. R. and Winter, S. G. (1982) An evolu-tionary theory of economic change, The Belknap Press of Harvard University Press, Cambridge, MA.

Perrin, B. (2002) How to – and how not to – Evaluate innovation, Evaluation 8(1), 13–28.

Roessner, J. D. (2002) Outcome measurement in the USA: State of the art, Research Evaluation 11(2), 85–93.

Smits, R. and Kuhlmann, S. (2004) The rise of systemic instruments in innovation policy, International Journal of Foresight and Innovation Policy 1(172), 4–32.

Stuart, T. E. and Ding, W. W. (2006) When do scientists become entrepreneurs? The social structural antecedents of commercial activity in the academic life sciences, American Journal of Sociology 112(1), 97–144.

SMALL ENTERPRISE RESEARCH – ISSN 1321-5906

Public policy and enterprise development + general sectionA special issue of Small Enterprise Research – Volume 17 Issue 1

ISBN 978-1-921729-45-4 – October 2010

Guest Editors: David Storey (University of Sussex, Brighton, UK) and Francis Greene (Warwick Business School, University of Warwick, Coventry, UK)

http://ser.e-contentmanagement.com/archives/vol/17/issue/1/marketing/

Editorial Announcement: New publishing arrangements for Small Enterprise Research – Brian Gibson, Beth Walker and Kate Lewis

Guest Editorial: Enterprise policy and practice – Francis Greene and David J Storey

Competition law, enforcement and the Australian small business sector – Michael T Schaper

The competitive experience of UK SMEs: Fair and unfair – David J Storey

Evaluation of a New Zealand business support programme using fi rm performance micro-data – Michele Morris and Philip Stevens

The goal achievement of federal lending programs – Matthias G Raith, Thorsten Staak and Christoph Starke

General sectionInternationalisation of small and medium New Zealand wineries: An exploratory study – Abel Duarte Alonso

A discursive approach to entrepreneurial marketing: Integrating academic and practice theory – Alex Maritz, Howard Frederick and Michael Valos

Perceived usefulness of business succession planning and chartered accountants’ involvement in the process – Deborah Sawers and Rosalind H Whiting

N O W AV A I L A B L E

eContent Management Pty Ltd, PO Box 1027, Maleny QLD 4552, AustraliaTel.: +61-7-5435-2900; Fax. +61-7-5435-2911;[email protected] www.e-contentmanagement.com

Contentmanagement

PTYLTD

Documents

Unintended consequences of innovation policy programmes: Social evaluation of technological projects programme in Croatia