Programming Exercises Evaluation Systems: An Interoperability Survey

Programming Exercises Evaluation Systems:An Interoperability Survey

Ricardo Queirós and José Paulo LealFaculty of Sciences of University of Porto

4th International Conference on Computer Supported Education (CSEDU’12)Porto, Portugal

April 16, 2012

Introduction Related Work Interoperability Survey Synthesis Conclusion

Outline

1 Introduction

2 Related Work

3 Interoperability Survey

4 Synthesis

5 Conclusion

Queirós & Leal PES: An Interoperability Survey 16-04-2012 2 / 18


xxxContext

Learning programming requires solving programming exercises.Manual assessment of exercises:

Time consuming - teachers need to assess a large number ofexercises (e.g. large classes)Error prone - hinders the consistency and accuracy of assessmentresults

Existent automatic evaluation systems aretoo genericfocused on simple domains (e.g. quizzes)

Appearance of

Programming Exercises Evaluation Systems (PES)



xxxProgramming Exercises Evaluation Systems

Automatic tools tograde students’ programming exercisesgive feedback on the quality of students’ solutions

Used on different learning scenarios:Curricular (e.g. practical classes, assignments and exams)Competitive (e.g. programming contests)

IOI - for secondary school studentsACM-ICPC - for university studentsIEEExtreme - for IEEE student members

Examples:AutoGrader, BOSS, Hustoj, Mooshak, WEB-CAT, etc.



xxxExisting surveys coverage

Several surveys enumerates and compares PES features...how the analysis of the code is madehow the tests are definedhow grades are calculated

...but neglects the interoperability feature. Organized at two levels:contentcommunication



xxxExisting surveys coverage

Content - how a programming exercise is describedas a Learning Objectcomposed by assets such as statement, test cases, feedback, etc.

communication - how it should be shared among systems:

LearningObjectsRepository

LearningManagementSystem

ProgrammingEvaluationSystem

1

2

3

4LO reference, Learner reference,

attemp t

LO reference LO

Evaluatio n repo rt



xxxProject

Work descriptionAn interoperability survey on existing PES

Main goalto gather information on the interoperability features of existent PES

Benefitsfills the gap found in most surveys on PES interoperability featureshelps in the PES selection to integrate in e-Learning environments



FakeTitle1Evolution of assessment tools

Evolution of # systems

Evolution of features

Punched cards

Support for one language (e.g. Algol)

No feedback

No administration facilities

Command-line interfaces (manual operation of scripts)

Support for few languages (e.g. C, JAVA)

Limited feedback

Content management

Static analysis

Student automated testing

Grading-support system

Competitive learning (e.g. contests)

Web based interfaces

Multi-languages (Prolog, SQL, FORTRAN)

Richer and incremental feedback

Course/student administration facilities

Sophisticated testing approaches

Automatic test generation

Plagiarism detection

Service-oriented

Integration with LMS

Early Assessment Syst ems (1960 - 1985) Tool Oriented Systems (1985 - 2000) Web-Oriented Syst ems (2000 - ... )

Figure: Petcha use cases.



FakeTitle1Recent surveys

Five surveys:Douce et al. (2005)

Methodology: details features of PES organized in 3 generationsTrend: evaluation of GUI programs, meta-testing (evaluation of thestudents’ tests), SOA and use of interoperability standards

Kirsti AlaMutka (2005)Methodology: organizes PES in dynamic and static evaluatorsTrend: content and communication standardization.

Liang et al. (2009)Methodology: details dynamic and static analysis methods of PESTrends: security, algorithms for automatic generation of test dataand content standardization

Ihantola et al. (2010)Methodology: discuss PES (2006-2010) features (e.g. testsdefinition, resubmission policies and security)Trends: integration with LMS and assessment of web applications

Romli et al. (2010)Methodology: approaches for test data generationTrends: test data generation techniques, interoperability and security



FakeTitle1Interoperability analysis

Survey on PES interoperability featuresApproach for the selection of

the sample - 15 toolsbased on its effective use (exercises and users management)selected tools: AutoGrader, BOSS2, CourseMaker, CTPracticals,DOMJudge, EduComponents, GAME, HUSTOJ, Moe, Mooshak,Peach3, Submit!, USACO, Verkkoke, Web-CAT

the criteria - 3 facetsbased on surveys trends and our backgroundselected facets: programming exercises, users and assessment results



FakeTitle1Interoperability analysis

These 3 facets are synchronized with PES main objective

to evaluate a user’s attempt to solve a programming exercise andproduce an assessment result.

For each facet each PES is evaluated by 3 interoperability maturity levels:

Level 0 - manual configuration of dataLevel 1 - data import/exportLevel 2 - services invocation



FakeTitle1Programming Exercises

Programming Exercises facetLevel 0 - manual configuration of exercisesLevel 1 - import/export of exercisesLevel 2 - integration with repository services

Systems Level 0 Level 1 Level 2AutoGrader F - -BOSS2 F - -

CourseMaker F - -CTPraticals F - -DOMJudge F - -

EduComponents F - -GAME F - -

HUSTOJ F P -Moe F P -

Mooshak F F FPeach3 F P -Submit! F - -USACO F - -Verkkoke F F -Web-CAT F F P

All systems support the configuration of exercises6 tools export exercises; 3 bidirectional support; few systems useexercises formats2 tools support communication with repositories through SOA



FakeTitle1Users

Users facetLevel 0 - manual configuration of users;Level 1 - import/export of users;Level 2 - integration with user directories services (authentication)and AMS (authorization)

Systems Level 0 Level 1 Level 2AutoGrader F F PBOSS2 F - -

CourseMaker F - -CTPraticals F F PDOMJudge F F P

EduComponents F F PGAME F - -

HUSTOJ F - -Moe F - -

Mooshak F F PPeach3 F - -Submit! F - -USACO F F -Verkkoke F F -Web-CAT F F -

All systems support the manual configuration of users8 tools allow the import/export of users in non-standard formats5 tools communicates with authentication services (LDAP)



FakeTitle1Assessment results

Assessment results facetLevel 0 - visualization of evaluation resultsLevel 1 - export of assessment resultsLevel 2 - integration with LMS

Systems Level 0 Level 1 Level 2AutoGrader F F PBOSS2 F - -

CourseMaker F - -CTPraticals F F PDOMJudge F F -

EduComponents F F PGAME F - -

HUSTOJ F - -Moe F - -

Mooshak F F -Peach3 F F -Submit! F - -USACO F - -Verkkoke F F PWeb-CAT F F -

All systems present the evaluation results to usersThe majority allows its exportation in non-standard formats4 systems support the communication with LMS



FakeTitle1Interoperability facets coverage

Mooshak, Web-CAT, Verkkoke offer the best interoperability levelsHalf of the systems studied did not reach 50% of the maturity rateThere are a lot to do regarding PES interoperability

39%

44%

50%

61%

67%

67%

67%

72%

72%

83%

Moe

USACO

Peach3

DOMJudge

AutoGrader

CTPracticals

EduComponents

Verkkoke

Web-CAT

Mooshak

Sy

ste

ms

33%

33%

33%

33%

39%

39%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

BOSS2

CourseMaker

GAME

Submit!

HUSTOJ

Moe

% of maturity



FakeTitle1Interoperability features coverage

There is no specific trend on interoperability featuresNevertheless, programming exercises facet presents the lower resultsNeed to standardize:

the description of programming exercisesthe communication of PES with other systems (e.g. repositories)

53%43% 44%

30%

40%

50%

60%

70%

80%

90%

100%

% o

f co

ve

rag

e

not covered

covered

46%56% 55%

0%

10%

20%

30%

programming exercises user assessment results

Interoperability facets



FakeTitle1Conclusion

This work presents an interoperability survey on PES15 tools were surveyed based on 3 interoperability trend facets

programming exercisesusersassessment results

Issues detected on most PES regarding interoperabilitythe lack of standards to

describe programming exercises and assessment resultscommunicate with other e-Learning systems (e.g. LMS, AMS, LOR)

Survey benefitsis the first interoperability study on existing PEScan base the

selection of existing PES in e-Learning environmentsdesign of interoperability requirements for the creation of new PES



FakeTitle1Questions

Thanks for your attention

Ricardo Queiró[email protected]


Technology

Programming Exercises Evaluation Systems: An Interoperability Survey