11/2/13

1

Impact of Funding on Scientific Output and Collaboration

Ashkan Ebadi Hadi Shahidi Nejad

Andrea Schiffauerova

2  November  2013  

1  /  16  

Outline

q  Funding q  Collaboration q  Impact of funding on collaboration q  Impact of funding on scientific output q  Research gaps and conclusion

11/2/13

2

2  /  16  

Funding

One of the most crucial factors for improving the research performance •  Huge annual investment on R&D

•  Importance of the link between funding and scientific development

•  Evaluation is needed! (King, 1987)

(Martin, 2003)

3  /  16  

Role of Funding

Affects size, efficiency and productivity of R&D sector

(Jacob & Lefgren, 2007)

Benefits of the funded research: Ø  Increasing the level of available knowledge Ø  Motivating collaboration networks Ø  Training skillful graduates Ø  Creating new jobs and companies Ø  Augmenting the problem-solving ability of the researchers

(Martin et al., 1996)

11/2/13

3

4  /  16  

Why researchers collaborate?

(Lee & Bozeman, 2005)

Collaboration!

Higher  produc-vity  

Be2er   and   faster   access   to  exper-se  

Idea   exchange   among   scien-fic  disciplines  

Be2er  access  to  funding  resources  

(Beaver, 2001)

(Katz, et al., 1997)

(Heinze, et al., 2008)

(Beaver, 2001)

How  to  measure  it  

Ø  Co-authorship Advantages  

Prac9cal  

Invariant  

Verifiable  

Inexpensive    

Quan9fiable    

Disadvantages  

Collabora9on  not  necessarily  results  in  a  joint  ar9cle  

Par9cipa9on  share  

(Subramanyam, 1983; Katz & Martin, 1997)

Ø  Sub-authorship •   Coordina9on  cost  •   Finding  right  partners  •   Delays  

(He, Geng, & Campbell-Hunt, 2009)

5  /  16  

Funding Impact on Collaboration

Author(s) Year Methodology Data Target area Result(s) Beaver and Rosen

1979 Indicators (no of authors per article)

24 scientific fields Positive relation Heffner 1981 Statistics · 395 articles, 28 journals

· 1974-1975 4 scientific fields Positive relation found in 2 (out of

4) disciplines Lewison and Dawson

1998 Statistical descriptive analysis

Research Outputs Database (ROD) 12,925 UK papers 1988-1994

Biomedical (gastroenterology)

Collaboration and number of funding bodies increase paper quality

Bozeman and Corley

2004 Questionnaire Regression analysis

451 scientists and engineers in the US

Scientists’ collaboration choices

Significantly positive

Adams et al. 2005 Regression analysis

2.4 million papers of 110 US universities 1981-1999

Top US universities

Positive effect of funding on team size

Gulbrandsen and Smeby

2005 Questionnaires Logistic regression analysis

1,967 records Tenured university professors in Norway

A positive relation observed

Lundberg et al. 2006 Indicators Industrial funding to a medical university 1993-2003

Co-authorship between university and industry

Incomplete results Some signs of positive collaboration

11/2/13

4

6  /  16  

Funding Impact on Collaboration

Author(s) Year Methodology Data Target area Result(s)

Thune 2007 Qualitative analysis

Interviews with 29 researchers and R&D managers

Collaborative R&D projects in two academic fields

Important, but other factors are also involved

Rosenzweig et al. 2008 Logistic regression 5,728 articles published in 4 American journals 1994-2003

US Emergency Medicine (EM)

No significant relation

Defazio et al. 2009 Regression analysis

· Panel of 294 scientists · 39 EU research networks

Researchers in the EU funding program framework

Funding may affect the formation of more effective collaboration networks

7  /  16  

Funding Impact on Collaboration

Literature Review

Article Ctrl.

Grp

Independent Variables

Type Various

sources of funding

Different funding periods?

Have

network structure variables?

Fund Gender Prestige/career Scientific

fields Regional

share Past prod.

H e f f n e r (1981) ü ü Desc.

analysis No No No

Bozeman and Corley (2004)

ü ü ü ü OLS

Linear regression

No No No

Adams et al. (2005) ü ü ü Linear

regression No No No

Gulbrandsen and Smeby (2005) ü ü ü ü ü ü Logistic

regression Yes No No

Rosenzweig et al. (2008)

ü ü Logistic regression Yes No No

Defazio et al. (2009) ü ü ü Linear

regression No Yes No

11/2/13

5

8  /  16  

Funding Impact on Collaboration

Highlights: •  Relatively new •  From simple indicators to statistical analysis •  No study done in Canada! •  Relatively limited data sets •  Lack of using a control group •  Vague net impact of funding on collaboration

9  /  16  

Scientific Output

Number  of  publica-ons  as  a  measure  of      output  quan-ty  • Very  reliable  measure  for  large-­‐scale  data  • Quan9fiable  

Number  of  cita-ons  as  an  indicator  of  quality  • A  good  index  of  the  mean  impact  at  the  aggregate  level    • Easy  to  calculate  • Drawbacks:  • Papers  of  famous  scien9sts  are  more  likely  to  be  cited  • Self  cita9on  • Etc.  

(Gingras, 1996)

(Okubo, 1997)

11/2/13

6

10  /  16  

Funding Impact on Scientific Output

Author(s) Year Methodology Data Target area Result(s)

McAllister and Narin

1983 Bibliometric indicators

NIH funded researchers

Strong positive relationship High quality schools are more productive

Peritz 1990 Statistical analysis Articles published in two British journals in 1978 and 1979

Economics Funded researchers are more being cited

Gingras 1996 Bibliometric indicators

NSERC funded research 1984-1993

Feasibility study of using bibliometrics

Feasible to apply at disciplines or specialties level

Arora et al. 1998 Econometrics OLS regression

Research funded by CNR 1989-1993

Biotechnology, bio-instrumentation

Italy

More unequal distribution of funds may increase the output in the short term

Lewison and Dawson

1998 Statistical analysis Research Outputs Database (ROD)

12,925 UK papers 1988-1994

Biomedical (gastroenterology)

Positive relation between the number of funding bodies and research output impact

Boyack and Borner 2002 VxInsight 3D map 33,448 grants 4,549 outputs

1975-2001

BSR program output NIA grants

Positive relation between funding and output in most of their cases

Godin 2003 Bibliometrics Science Citation Index (SCI) database 1990-1999

NSERC funded research

Positive relation between funding and productivity, but no impact on quality

Payne and Siow 2003 Regression analysis Federal funding 1972-1998

74 research universities

Positive effect on output No impact on research quality

11  /  16  

Funding Impact on Scientific Output

Author(s) Year Methodology Data Target area Result(s) H u f f m a n and Evenson

2005 Econometrics USDA 1970-1999

Agricultural research productivity

Negative impact

Carayol and Matt

2006 Statistical analysis OLS regression

Louis Pasteur University (ULP) 1993-2000

Faculty members No significant effect of contractual funding

Jacob and Lefgren

2007 OLS regression 1980-2000 NIH Positive impact on the output

Crespi and Geuna

2008 Econometric Thomson ISI database 1981-2002

Higher education in 14 OECD countries

A significant impact on the time lag structure of the output

Albrecht 2009 Bibliometric indicators PubMed database, 1994-2003 CANSA No conclusion

Leydesdorff and Wagner

2009 Main S&T indicators of OECD (2008)

Thomson’s Web of Science M a c r o - l e v e l comparisons

Different patterns of the link between inves tment and wor ld-share of publication

Campbell et al.

2010 Bibliometrics Thomson Reuters’ Web of Science (WoS) database

NCIC Positive relation between funds and output quality

Shapira and Wang

2010 Bibliometric indicators Thomson ISI database 2008-2009

C r o s s - c o u n t r y evaluation

Positive impact of China’s investment on output quantity, but no major effect on the quality

B e a u d r y and Clerk-lamalice

2010 Regression analysis 1985-2005 3,678 articles

C a n a d i a n b i o t e c h n o l o g y academics

Positive effect of strong network position and individual funding on scientific output

B e a u d r y and Allaoui

2012 Regression analysis Articles and patents 1985-2005

3,724 articles, 566 patents

C a n a d i a n n a n o t e c h n o l o g y researchers

Positive effect of public funding, no impact of private funding

11/2/13

7

Godin (2003) studied the impact of NSERC funding on the productivity and papers’ quality of the supported researchers:

Ø  Bibliometric evaluation Ø 1990-1999 Ø  Science Citation Index (SCI) database Ø  Positive relation between funding and productivity Ø  No impact of funding on papers’ quality

12  /  16  

Funding Impact on Output - Canada

Literature Review

Gingras (1996) discussed the feasibility of bibliometric evaluation of funded research:

Ø  1984-1993 Ø  Two grant selection committees of NSERC (ME and EE) Ø Bibliometric indicators are good measures for investigating the

relation among funds and productivity but not at individual level

1996

Gingras

2003 Feasibility study

Godin

•  Positive impact on productivity •  No impact on quality

2009-2010

Campbell et al.

Few case studies done by Campbell et al. in 2009-2010

Ø  Evaluated the impact of funded research Ø  National Cancer Institute of Canada (NCIC) Ø  Thomson Reuters’ Web of Science (WoS) database Ø  Bibliometrics Ø  Positive relation between NCIC funds and scientific performance

•  Case studies •  Positive impact on performance •  No impact on quality

Ø  Evaluated the impact of funded research Ø  Canadian Forest Service (CFS) Ø  Bibliometrics Ø  Assessed CFS internal, national and international position

Ø  Evaluated specifically the selection procedure of Genome Canada Ø  Thomson Reuters’ Web of Science (WoS) database Ø  Bibliometrics Ø  Peer-review process was successful in researchers selection Ø  Higher scientific impact of the funded researchers’ papers

2012 •  Network variables •  Positive impact of public funding •  No impact of private funding

Beaudry and Allaoui

Beaudry and Allaoui (2012) studied the impact of funding on the productivity of Canadian nanotechnology researchers:

Ø  Negative binomial regression Ø 1985-2005 Ø  3,724 articles and 566 patents Ø  Positive relation between public funding and productivity Ø  No impact of private funding

13  /  16  

Funding Impact on Scientific Output

Article Ctrl.

Grp.

Independent Variables

Type

Various sources

of funding

Different funding periods?

Network structure variables? Fund

Other productiv-ity shocks

Prestige/ career/

Age

Content/ scientific

fields

Regional share

Group size

Paper quality

Peritz (1990) ü ü ü Descriptive analysis No No No Arora et al. (1998) ü ü ü ü ü OLS regression No No No

Lewison and Dawson (1998) ü ü ü ü Statistical analysis No No No

Godin (2003) ü ü ü ü Descriptive analysis No No No

Payne and Siow (2003) ü ü ü Linear regression No No No

Huffman and Evenson (2005) ü ü Econometrics Yes No No

Carayol and Matt (2006) ü ü ü ü OLS regression No No No

Jacob and Lefgren (2007) ü ü ü ü OLS regression No Yes No

Crespi and Geuna (2008) ü ü ü Econometrics No Yes No

Campbell et al. (2010) ü ü ü ü Descriptive analysis No No No Beaudry and Clerk-lamalice (2010)

ü ü Negative binomial

regression Yes No Yes

Beaudry and Allaoui (2012) ü ü ü

Negative binomial regression Yes No Yes

11/2/13

8

14  /  16  

Funding Impact on Scientific Output

Highlights: •  Recently attracted more attention •  Simple indicators and statistical analysis •  Several studies in Canada •  Relatively limited data sets •  Lack of using a control group •  Vague net impact of funding on output •  Less studies done at the individual level

15/  16  

Research Gaps

Ø Lack of a comprehensive study Ø The importance of the impact of network structure Ø Bibliometrics as the main methodology Ø Consider different periods of funding Ø Relatively old results Ø Adjust funding!

Ø Vague relation among funding, collaboration and scientific production

Ø Lots of contradictory results

Ø Very limited knowledge about the effects of funding on collaboration

Ø Limited scope of the studies (collaboration among the university

professors or the cooperation between universities and industry)

Ø No test group

Ø Simplified linear regressions

Ø Score collaborations! (Linking to the same institute gets lower score)

11/2/13

9

 Ø  The need for developing a clear link between the funding,

multidisciplinary collaboration and knowledge production

Ø Most of the studies have used bibliometrics or statistical

methods for performing the analysis

Ø  The need for a comprehensive study that covers several

scientific fields and aspects

17  16/16  

Research Summary

18  /  22  

Suggested Solution

Bibliometrics  

Sta9s9cal  Analysis  

Visualiza9on  Techniques  

Social  Network  Analysis  

Data  &  Text  Mining  

Survey  

11/2/13

10

Bibliometric Analysis Ø  Scientific output and research quality Ø  Funding trend and funding efficiency Ø  Universities (regional) performance Ø  Collaboration trend and research group structure Ø  Measuring the complexity of research

19  /  22  

Methodologies

Data Mining Ø  Dimensionality and size reduction Ø  Clusters of researchers Ø  Pattern and rules detection

Text Mining Ø  Pre-processing and data cleaning Ø  Keyword extraction Ø  Co-authorship trend (probabilities of a connection)

Collaboration Network Analysis Ø  Network structure variables Ø  Trend of collaboration Ø  Collaboration patterns

Statistical Analysis Ø  Statistical validation of the results

Survey Ø  Qualitative investigation to check the quantitative results

20  /  22  

Methodologies

Why several methods Ø  A multi-dimensional phenomenon! Ø  “Triangulation” (cross examination), using multiple methods are

used in a study to empower the validation of results Ø  Enabling comparisons of different explanations (Fulop et al., 2001)

11/2/13

11

22  /  22  

Evaluation Methods

     

Scientometrics  

Bibliometrics  Peer  review  

Econometric  Cost-­‐benefit  analysis  

Surveys  Case  studies  

Data  and  text  mining  

Visualiza9on  techniques  Social  network  analysis  

Peer  review  

Ø  Fast and low cost Ø  High quality works are most of the time detected Ø  Highly depends on the experts and criteria Ø  More  fame  will  result  in  geZng  higher  funds  Ø  High administra9ve  cost

(King, 1987)

Ø  Simple and easy to use Ø  Not integrated and simplified (Ruegg, 2007)

Sciento-­‐metrics  

Ø  Good tool for testing the findings Ø  Simplified assumptions for creating the model

(Salter & Martin, 2001) Surveys  Case  studies  

Ø  Narrative, easy to understand Ø  Less convincing Ø  Could be inconsistent (Ruegg, 2007)

Econometric  

Ø  Modern technique Ø  Relatively expensive (Ruegg, 2007)

Data  mining  

Ø  Give a big picture of the subject Ø  Limited flexibility

Visualiza9on  

Ø  Useful tool for studying collaboration Ø  Useful tool for studying impact of a program Ø  The generated network can be time limited (Ruegg, 2007)

11/2/13

12

23  

Funding Impact on Collaboration

Literature Review

Ø  Higher financial investment can change the structure of research Ø The efficient collaboration network will not be stable due to huge

coordination cost Ø Funding enables researchers to cover the collaboration costs

Ø  Co-authorship as a measure of scientific collaboration Ø Studying the Impact of governmental funding on scientific collaboration

and formation of scientific networks is relatively new (Katz & Martin, 1997; Lee & Bozeman, 2005)

(Ubfal & Maffioli, 2011)

Ø Funding may help the central actor(s) to make a balance between the new knowledge creation and the management of the existing collaborative relationships in the network

(Porac et al., 2004)

24  

Funding Impact on Collaboration

Literature Review

Heffner (1981) analyzed the relation between funding and multiple authorship:

Ø  500 articles published in 28 journals in four scientific fields Ø  1974-1975 Ø  Statistical analysis Ø  Positive relation between financial support and size of the research teams Ø  Statistically significant just in chemistry and biology

Researchers of top US universities with larger amounts of funding tend to work in larger scientific groups (Adams et al., 2005)

Defazio et al. (2009) studied the impacts of funding on collaborative behavior and productivity of the researchers:

Ø  Used a panel of 294 scientists in 39 EU research networks Ø  15-year time period Ø  Funding may play an important role in forming more effective collaboration

networks

11/2/13

13

25  

Funding Impact on Collaboration

Literature Review

Gulbrandsen and Smeby (2005) studied the effect of industrial funding on the performance of university professors:

Ø  Norway Ø  Used questionnaires to collect data from all tenured professors Ø  Statistical analysis Ø A Positive relation observed Ø  Funded professors tend to collaborate more with other researchers from

both universities and industries

A significant positive effect of funding on the collaboration of the US university researchers (Bozeman & Corley, 2004; Lee & Bozeman, 2005)

26  

Funding Impact on Output

Literature Review

Ø  Number of publications as a measure of output quantity Ø  Number of citations as an indicator of quality

Peritz (1990) found that even if both funded and unfunded researches are published in a high-impact journal, the funded research will be more cited.

Lewison and Dawson (1998) used journal impact as a quality measure. They concluded that the number of authors per article and the number of funding bodies have a great effect on the impact of research output.

McAllister and Narin (1983) found a positive relation between NIH’s funding and number of publications of the U.S. medical schools.

11/2/13

14

27  

Funding Impact on Output – Cross Country

Literature Review

Leydesdorff and Wagner (2009) analyzed the relation between research macro-level investment and world share of publication:

Ø  Different efficiencies observed among examined countries Ø  Different schemes of funding

Crespi and Geuna (2008) analyzed impact of investment on scientific productivity of 14 OECD countries:

Ø  Used Thomson ISI database Ø  1981-2002 Ø  Focused on the time lag Ø  Found a significant impact of investment

Shapira and Wang (2010) investigated impact of nanotechnology funding:

Ø  Used Thomson Reuters database Ø  August 2008 to July 2009 Ø  Very basic bibliometric indicators Ø China is getting closer to the U.S. in terms of the number of

publications but still lower quality

28  

Publications

Conferences: 1.  Ebadi, A. & Schiffauerova, A., 2011. Investigating Climate Change Patterns in Montreal: A

Data Mining Approach, Third International Conference on Climate Change, Brezil. 2.  Ebadi, A., Moazami, A. & Schiffauerova, A., 2012. Analyzing the Collaboration Trend of

Canadian Researchers in Natural Sciences and Engineering: The Impact of Geographical Proximity and Research Funding, Proximity Days Conference - 7th Edition, Montreal, Canada.

3.  Ebadi, A. & Schiffauerova, A., 2012. Impact of Funding on Scientific Output and Collaboration Patterns of Canadian Researchers: A Bibliometric and Text Mining Approach”, 13th Collnet Conference, Seoul, South Korea.

Journal: 1.  Eslami, H., Ebadi, A. & Schiffauerova, A., 2012. Effect of collaboration network structure on

knowledge and innovation productivity: The case of biotechnology in Canada.

11/2/13

15

29  

Data Gathering Procedure

144,156 articles 184,770 authors

1996-2010

381,197 grantees