A Systematic Mapping Study on High-level Language Virtual ...design/vmil/2010/slides/p05-durelli-slide… · Systematic Mapping Threats to Validity Concluding Remarks A Systematic

MotivationSystematic Mapping

Threats to ValidityConcluding Remarks

A Systematic Mapping Study on High-levelLanguage Virtual Machines

Vinicius H. S. Durelli, Katia R. Felizardo, andMarcio E. Delamaro

Computer Systems DepartmentUniversity of Sao Paulo (ICMC-USP)13560-970 – Sao Carlos – SP – Brazil{durelli,katiarf,delamaro}@icmc.usp.br

October 17, 2010

1 / 24



Agenda

1 MotivationResearch on HLL VMsFirst Step Towards Filling in Such a Gap

2 Systematic MappingOverviewStepsData Extraction and Mapping

3 Threats to Validity

4 Concluding Remarks

2 / 24



Research on HLL VMsFirst Step Towards Filling in Such a Gap

Research on High-level Language Virtual Machines

A great deal of the contemporary high-level languages havetheir execution environment based upon high-level languagevirtual machines (HLL VMs).

There is a large body of literature on research in virtualmachine for high-level languages.

A mature research area means a sharp increasein the number of results made available, thus itbecomes essential to summarize and provide anoverview of such area.

To the best of our knowledge there are no comprehensivestudies focusing on an overview of this research area andits most investigated subjects.

3 / 24



Research on HLL VMsFirst Step Towards Filling in Such a Gap

Motivation: First Step Towards Filling in Such Gap

In order to fill in such a gap it is needed to ascertain thenature, extent, and quantity of published research papers.

Contributions:

1 Areas that have been most subjected to investigation.

Side effect: Areas that require further research.

2 The relevant publication forums.

3 HLL VM implementations that are the most widely usedwithin the academic community.

4 / 24



OverviewStepsData Extraction and Mapping

Evidence-based Paradigm

Definition → Systematic Mapping

Methodology that involves searching the literature in order to aggre-gate and categorize primary studies, thereby yielding a synthesizedview of the research area under consideration [Petersen et al., 2008].

Advantages:

The approach used for searching and inclusion and exclusioncriteria are defined in a research protocol and reported asan outcome.

Side effects: Transparent;Replicable;Updatable.

5 / 24




Systematic Mapping Process: Overview

Figure: The systematic mapping process [Petersen et al., 2008].

6 / 24




Research Questions

Research questions must embody the mapping study purpose.

RQ1: which functionalities/features/characteristics of HLLVMs have been most investigated?

RQ2: which are the mainstream HLL VM implementationswithin the academic community?

7 / 24




Search for Primary Studies

Search String → combination of these keywords and acronyms

virtual machine, VM, high-level language virtual machine, and HLLVM.

We used the search string on the following electronic databases:

ACM Digital Library,

EngineeringVillage,

IEEE Xplore,

Springer Lecture Notes in Computer Science (LNCS), and

ScienceDirect.

No limits were placed on date of publication.

8 / 24




Screening: Inclusion Criteria

The inclusion criteria devised and applied are:

if several papers reported similar studies, only the mostrecent was selected;

papers describing more than one study had each studyindividually evaluated;

it has to describe at least a prototypical implementation ofthe proposed improvement, thereby mentioning the HLLVM implementation that was modified.

9 / 24




Screening: Exclusion Criteria (i)

and the following exclusion criteria:

papers that do not present studies pertaining to HLL VMs,e.g., papers describing research on system VMs;

studies describing the introduction of improvements thatconsist in solely modifying the intermediate language of theHLL VM under consideration;

studies whose proposed enhancements do not imply inmaking changes to the underlying HLL VM, e.g., papersdescribing features implemented atop HLL VMs;

10 / 24




Screening: Exclusion Criteria (ii)

studies whose target HLL VM is either a co-designed (e.g.,composed of both software and hardware portions) or anentirely implemented in hardware HLL VM;

technical reports, documents that are available in the formof either abstracts or presentations (i.e., elements of “grey”literature), and secondary literature reviews (i.e., mappingstudies).

11 / 24




Final Set of Selected Primary Studies

Electronic Database Number

ACM Digital Library 1554

EngineeringVillage 1395

IEEE Xplore 309

Springer LNCS 640

ScienceDirect 1123

Total 5021

Candidates 142

Final set 128

Table: Papers retrieved from each electronic database, total of candidatestudies, and the final set.

12 / 24




Keywording

The aim of this step is to devise our own classificationscheme and categories for the selected primary studies.

Certain sections are read for the purpose of findingkeywords and concepts that reflect their contribution.

13 / 24




Resulting Categories

CategoriesOptimizationGarbage Collection (GC)DebuggingMemory Leak Tolerance (MLT)New Language Construct (NLC)ProfilingAspect-Oriented Programming (AOP)Embedded System (ES)SecurityReal-TimeDistributed Computing (DC)Fault Tolerance (FT)Resource Sharing among HLL VMs (RSVM)Testing

14 / 24




Resulting Category Frequencies

Testing

RSVM MLT

Security NLC FT DC

DebuggingAOP

Profiling

Real-Time ES GC

Optimizatio

n

Category

0

5

10

15

20

25

30

35

Fre

qu

en

cy

2 2 23

4 4

78 8

9

13

24

3334

Figure: Frequency of studies in each category*.

∗Certain studies were grouped in more than one category

15 / 24




Most Researched Subjects Evolution

According to our results, these are the “trendy” subjects:

19981999

20002001

20022003

20042005

20062007

20082009

Year

0

1

2

3

4

5

6

7

Fre

qu

en

cy

OptimizationGCES

Figure: Year-wise distribution of publications on the most investigatedcategories.

16 / 24




Distribution of Primary Studies by Electronic Database

EngineeringVillage

30.0%

ACMDigital

Library

48.0%

Springer LNCS

13.0%

IEEE Xplore

9.0%

Electronic Databases

Electronic Database Number

ACM Digital Library 62

EngineeringVillage 38

Springer LNCS 16

IEEE Xplore 12

ScienceDirect 0

17 / 24




Distribution of Primary Studies by Publication Type

Conference

36.0%

Journal24.0%

Symposium

20.0%

Book Chapter

14.0%

Workshop6.0%

Publication Type

Publication Type Number

Conference 46

Journal 31

Symposium 25

Book Chapter 18

Workshop 8

18 / 24




Map: Year-Wise Distribution (detailed)

1999

1998

2002

2003

2000

2001

2006

2007

2004

2005

2008

2009

2010

FT

NLC

DC

Optimization

GC

MLT

Real-Time

Security

ES

Debugging

RSVM

Profiling

Testing

AOP

1

1

1

2

3

1

2

1

1

4

2

1

1

1

1

3

1

1

2

1

1

2

1

2

1

4

5

1

4

3

4

2

7

1

1

1

1

3

3

1

5

2

5

2

1

1

1

3

2

2

6

4

1

3

2

6

1

1

2

3

1

2

1

3

3

4

1

1

1

1

1

1

1

Map

19 / 24




Map: The Most-Widely Used HLL VM Implementations

GForth

VM*

Sabl

eVM

CVM

Har

mon

y

Sim

pleR

TJ

OCV

M

VMKit

Jikes

RVM

IBM

's J2

ME

MONO

Hot

Spot

JESS

ICA2

CACA

O

TclV

M

JeRTy

Stea

mlo

omKaf

feCL

R

Exac

t VM

Jam

aica

Max

ine

JITSJ9

IVM

CEJV

MKVM

ORP

SICS

tus

Jam

VMOVM

FT

NLC

DC

Optimization

GC

MLT

Real-Time

Security

ES

Debugging

RSVM

Profiling

Testing

AOP

12111 6 1 332 3 1

421 1

11 1 52

2 13 1

1 1 1

1 1

31 11 2

123 4 1 2 1 13 11 111

2 1 2 2 13

5 52 3 1 2 1 11 1

2

11 11

1 12

2

Map

20 / 24



Threats to Validity

We cannot rule out threats from a quality assessmentperspective.

(We wanted to be as inclusive as possible) We simplyselected studies without assigning any scores.

Another threat consists in whether we have properlyidentified and selected all relevant publications.

Whether our resulting classification scheme and cate-gories are coherent also represents a threat to validity.

21 / 24



Concluding Remarks

The mapping study results, although not entirely surprising (somemay argue), can be used to support several claims that arefrequently made but not scientifically backed up.

Our mapping study reveals that the majority of researchinto HLL VMs focuses on optimizing these executionenvironments, improving their memory management capa-bilities, and tailoring them to resource-constrained settings.

As for the publication types, the majority of the studies areconference publications.

Another contribution of this paper is the map we havecreated.

22 / 24



References

K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson.Systematic Mapping Studies in Software Engineering.12th International Conference on Evaluation and Assessmentin Software Engineering (EASE), pages 71–80, 2008

J. E. Smith and R. NairThe Architecture of Virtual Machines.Computer 38(5):32–38, 2005.

J. E. Smith and R. NairVirtual Machines: Versatile Platforms for Systems andProcesses.Morgan Kaufmann, 656 pages, 2005.

23 / 24



A Systematic Mapping Study on High-levelLanguage Virtual Machines

Vinicius H. S. Durelli, Katia R. Felizardo, andMarcio E. Delamaro

Computer Systems DepartmentUniversity of Sao Paulo (ICMC-USP)13560-970 – Sao Carlos – SP – Brazil{durelli,katiarf,delamaro}@icmc.usp.br

October 17, 2010

24 / 24

Documents

A Systematic Mapping Study on High-level Language Virtual ...design/vmil/2010/slides/p05-durelli-slide… · Systematic Mapping Threats to Validity Concluding Remarks A Systematic