Relation between the Quality of the Review Process of Papers and the Impact of Artificial Intelligence

Journals

José GonzálezJosé-Ángel SosaJorge AlarcónCongress on Research Methodology, MUI-TICOctober 27, 2011

• INTRODUCTION

• RELATED WORK

• RESEARCH QUESTION

• OBJECTIVES

• RESEARCH METHODOLOGY

• THEORETICAL MODEL

• EVALUATION

• ANALYSIS OF ANOMALIES

• CONCLUSIONS AND FUTURE WORK

• REFERENCES

Index

Introduction

• The acceptance or rejection of an item is directly related tothe quality of the referees [1].

• The level of importance of a journal is often measured by theimpact factor of the Journal Citation Reports (JCR).

• The quality of the review article by a journal is positivelyrelated to the impact factor of this journal .

• There are conferences supported by sponsors, where"quantity" is preferred over "quality” [2].

3

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Related Work

• Quality of the review process of paper has been studied in the last 50 years [1].

• Peer review has generated opinions for and against [1].

• Alternative ways of performing the review process, like the "open review" [3] or using a public voting system [1] [4].

• Some researchers think that JCR does not include research conducted in other languages [5].

• We did not find studies that relate the quality of the review process versus the impact factor of publications on Computer Science.

4

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Research Question

Is there a positive co-relation between theimpact of Artificial Intelligence Journals and

their quality in the review process?

5

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Objectives

• Global Objective:

• To determine if there is a positive co-rrelation between the quality ofthe review process of papers and the impact of journals in theArtificial Intelligence field.

• Specific Objectives:

• To analyze the factors that are involved in the quality of the reviewprocess of articles in Artificial Intelligence.

• To propose a theorical model to measure the quality of the reviewprocess against the impact factor of a journal of Artificial Intelligence.

• To evaluate the theoretical model with quantitative and qualitativetechniques.

• To analyze the possible anomalies (cases where no positive co-relation occurs) by case studies and interpret the results. 6

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

7

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

7

Theoretical Model

8

• Quality of the review process• Comprised by three main factors:

• Factor 1: Quality of the review process as perceived by authors who submitted an article to the journal. Two sides of the same coin [6].

• Factor 2: Quality of the reviewers of a journal. Expertise of the reviewers.

• Factor 3: Quality of the review criteria used in a journal. Procedures, criteria and practices a journal follows to do the review process.

• We will define the “quality of the review process” as follows:

• Impact• “The average number of times articles from the journal published in

the past two years have been cited in the JCR year” [7].

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

9

Evaluation

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Quantitative and qualitative evaluation methodology.

15 journals selected randomly.

1. Measure of the Quality of the review process:a. Factor 1: Survey to authors: Using open-ended questionnaires, the

results will be graded from 0 to 1

b. Factor 2: Mean of the H-Index of the reviewers, average value willbe normalized to obtain a value from 0 to 1

c. Factor 3: Survey to reviewers: Using closed-endedquestionnaires, results will be graded from 0 to 1

Quality = Factor 1 + Factor 2 + Factor 3

2. Measure of the Impact of a journal:• Impact factor found in ISI JCR 2010.

10

Evaluation

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Qu

alit

y o

f th

e r

evi

ew

pro

cess

Impact

Quality of the review vs. Impact

Journal

Linear (Journal)

• We may find that some anomalies or strange cases appear.

• The anomalies may provide us with a counter-example for ourhypothesis.

• We will thoroughly analyze the journals with anomalies in aqualitative case study to find the reasons for the anomaly.

• The results of the case study will give us information tocomplete and refine our theoretical model.

11

Analysis of Anomalies

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Conclusions

• Research.

Relevant, Original and Feasible• Relevant, we may end up finding that the impact is not a good

indicator to ensure the quality of a publication

• Original: no previous studies on this topic, and

• Feasible: The study involves low-cost research methods (onlinesurveys, data obtained via the Internet, etc).

• We expect to validate that the quality of the review and itsimpact index are positively correlated for Artificial Intelligencejournals.

12

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Conclusions

• We think that the relation between quality of the review andits impact will be positive.

• The completion of the study will determine the possibleexistence of anomalies.

• High impact journals with low quality of review in their papers.

• Low-impact journals with a high quality review process in theirpapers.

• As a limitation we are aware that the factors of the quality area rough approximation to the real value of quality.

13

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Future Work

• This research is work in progress.

• We reached the phase where we propose the theoreticalmodel.

• Our future work will pursue the completion of the evaluationphase.

• Find the relation between Impact and Quality of thepublication itself.

14

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

References

1. CAMPANARIO, J. M. El Sistema De Revisión Por Expertos (Peer Review): MuchosProblemas y Pocas Soluciones., 2002. Available from<www2.uah.es/jmc/an24.pdf>.

2. OLSEN, K.A. The Economics of International Conferences, Jun. 2004. pp. 89-91.

3. WING, J. M.; CHI, E. H. Reviewing Peer Review. Communications of theACM, 2011, vol. 54, no. 7. pp. 10-11.

4. KESAHAV, S. Editor's Message. ACM Sigcomm Computer CommunicationsReview, 2011, vol. 41, no. 3. pp. 3-3.

5. Universidad Nacional de Educación a Distancia de España. Factor De ImpactoDe Las Publicaciones Periódicas. 08/2008 [cited 10/23/2011] Availablefrom:<http://www.uned.es/biblioteca/referencia/impacto.htm#que>.

6. SORENSEN, Karsten. This is Not an Article — just some Thoughts on how toWrite One. . ed. Penti Kerola, Antti Juustila, and Janne Järvinen. Oulu Universityed., Syöte, Finland ed. , August 6–9, 1994, 1994.

7. HARZING, Anne-Wil. Publish Or Perish. [cited 10/23/2011] Availablefrom:<http://www.harzing.com/pop.htm>.

15

INTRODUCTION

RELATED WORK

RESEARCH QUESTION

OBJECTIVES

RESEARCH METHODOLOGY

THEORETICAL MODEL

EVALUATION

ANALYSIS OF ANOMALIES

CONCLUSIONS AND FUTURE WORK

REFERENCES

Relation between the Quality of the Review Process of Papers and the Impact of Artificial Intelligence

Journals

José GonzálezJosé-Ángel SosaJorge AlarcónCongress on Research Methodology, MUI-TICOctober 27, 2011