MEASURING FIRM-LEVEL INNOVATION USING SHORT QUESTIONNAIRES.

EVIDENCE FROM AN EXPERIMENT IN DEVELOPING COUNTRIES

Xavier Cirera and Silvia Muzi

September 21, 2016

OECD Blue Sky Conference

Ghent, Belgium

1. MOTIVATION

Little evidence on extent and impact of firm level innovation in developing countries

o Lack of comparable innovation data

o Issues with reliability: subjective nature of some key concepts is a challenge, especially in developing countries (more incremental and less radical innovation)

o Implementing innovation surveys is costly, especially in developing countries

>>As a result, there is a complete lack of longitudinal data

2

2. OBJECTIVES

WBG-DECEA launched the Innovation Enterprise Surveys with the aim to:

• Increase the coverage of innovation surveys in developing countries

• Assessing the robustness of the Oslo framework (testing short form questionnaire vs traditional long questionnaire):o Which survey instrument is preferable?o Are the innovation data collected through these two survey

instruments different? If so, why are they different?

Main finding: Short questionnaire severely mis-measures innovation outcomes

3

3. THE EXPERIMENT

Two stage approach to data collection.

• First stage: a short innovation section of 9 selected questions (short questionnaire) is incorporated into the main ES questionnaire.

• Second stage, an in-depth innovation module (long questionnaire) is administered to 75% randomly selected ES respondents.

Both questionnaires are based on the standard Community Innovation Survey (CIS) but the long questionnaire was expanded to

o capture more details on innovation outcomes – when innovation was reported

o to integrate other important questions on organization, management practices, use of ICT, and innovation inputs.

4

4. THE DATA

• The dataset covers more than 11,000 firms in 15 countries –

• 11 in Sub-Saharan Africa - DRC, Ghana, Kenya, Namibia, Nigeria, Sudan, South Sudan, Tanzania, Uganda, Malawi and Zambia

• 4 in South Asia - Bangladesh, India, Pakistan, and Nepal

• Firms in manufacturing and services

• Only formal (registered) firms with 5+ employees (ES classification small (5-19), medium (20-99) and large (100+) - micro firms excluded

5

5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE TWO SURVEYS

• Large differences between innovation rates reported in the two questionnaires

• In more than 30% of cases firms change their mind regarding innovation outcomes between the two questionnaires

6

BGDDRC

GHA

IND

KEN

MWINAM

NGANPL

PAK

SDNSSDTZA

UGA

ZMB

020

4060

80S

hort

ques

tionn

aire

0 20 40 60 80Long questionnaire

Product Innovation

BGD

DRC

GHA

IND

KEN

MWINAM

NGANPL

PAK SDNSSD

TZA

UGA

ZMB

020

4060

80S

hort

ques

tionn

aire

0 20 40 60 80Long questionnaire

Process Innovation

BGDDRC

GHA IND

KEN

MWINAM

NGA NPL

PAK

SDNSSD

TZA

UGA

ZMB

020

4060

80S

hort

ques

tionn

aire

0 20 40 60 80 100Long questionnaire

Marketing Innovation

BGDDRCGHA

INDKENMWI

NAM

NGANPL

PAKSDN

SSDTZA

UGAZMB

020

4060

80S

hort

ques

tionn

aire

0 20 40 60 80Long questionnaire

R&D

5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE TWO SURVEYS

• More importantly, both surveys overstate the extend of innovation – when this can be verified looking at the explanation of the innovation introduced –

• Short questionnaire severely overestimates for most countries – but still very high innovation rates once cleaned!!!

7

5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE TWO SURVEYS

5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE TWO SURVEYS

9

5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE TWO SURVEYS

• How similar are firms in these groups? No-no likely non-innovators, but other non-innovators we could not say

10

5. DIFFERENCES ON INNOVATION OUTCOMES BETWEEN THE TWO SURVEYS

• How similar are firms in these groups? No-no likely non-innovators, but other non-innovators we could not say

11

6. EXPLAINING DIFFERENCES BETWEEN THE TWO SURVEYS

Three sets of elements may affect respondents’ answers to innovation questions

• Cognitive problems due to different understanding of innovation and quality of the interview

• Respondent framing - firm characteristics; context in which firms operate; quality of the interview

• Recall period – time elapsed between the two interviews conducted for the experiment

Estimation based on

- probit model (dependent variable = 1 if the answer in the two surveys is different and 0 otherwise).

- multinomial logit model, (Yi = 0: no difference between the two surveys; Yi =1 firm i reports innovation in the short questionnaire and no innovation in the long questionnaire; Yi =2: firm i reports no innovation in short questionnaire and innovation in the long questionnaire)

12

6. EXPLAINING DIFFERENCES BETWEEN THE TWO SURVEYS

• Discrepancies between surveys:

(–) same respondent; size; potential innovator; report bribes payment

(+) time interval; quality of the interview

• No effect of explanatory cards

13

6. ACCURACY SHORT QUESTIONNAIRE– controlling for selection

•(+) quality of the interview

•(-) size, report bribes payment

•no explanatory card effects,

14

7. MAIN FINDINGS

• The results from the survey experiment are clear: a few short questions in a more general firm-level survey do not provide an accurate picture of firm-level innovation.

• More context to the questions is likely to be needed; although the innovation rates found in the longer innovation questionnaire also requires substantial cleaning and present innovation rates that are unreasonably very high for the level of development in these countries – we need a new approach to better measure innovation outcomes and to disentangle “new” from “significantly improved”

Also:

• The results show that lack of information does not seem to play a key factor explaining inaccuracies, and neither the use of explanatory cards nor the experience of the respondent seem to matter.

• The quality of the interview is likely to influence the accuracy of responses (lower quality interview> inaccurate responses), while whether the firm reports to pay informal payments and the size of the firm are likely to reduce inaccuracies.

• In the case of explaining discrepancies between surveys, the time elapsed between interviews is a strong predictor of differences, likely the result of the recall effect.

15

THANK YOU

16