Using Mini-Projects to Teach Empirical Software Engineeringceur-ws.org/Vol-2358/paper-07.pdf · Using Mini-Projects to Teach Empirical Software Engineering Michael Felderer 1 and

Using Mini-Projects to TeachEmpirical Software Engineering

Michael Felderer1 and Marco Kuhrmann2

1University of Innsbruck, [email protected] University of Technology, [email protected]

AbstractEmpirical studies have become a central element ofsoftware engineering research and practice. Yet, teach-ing the instruments of empirical software engineeringis challenging, since students need to understand thetheory of the scientific method and also have to de-velop an understanding of the application of those in-struments and their benefits. In this paper, we presentand evaluate an approach to teach empirical softwareengineering with course-integrated mini-projects. Inmini-projects, students conduct small empirical stud-ies, e.g., surveys, literature reviews, controlled ex-periments, and data mining studies in collaboratingteams. We present the approach through two im-plementations at two universities as a self-containedcourse on empirical software engineering and as partof an advanced software engineering course; with 101graduate students in total. Our evaluation shows apositive learning experience and an improved under-standing of the concepts taught. More than a halfof the students consider empirical studies helpful fortheir later careers. Finally, a qualitative coding anda statistical analysis showed the proposed approachbeneficial, but also revealed challenges of the scien-tific work process, e.g., data collection activities thatwere underestimated by the students.

1 IntroductionEmpirical software engineering aims at making soft-ware engineering claims measurable, i.e., to analyzeand understand phenomena in software engineeringand to evaluate software engineering approaches andsolutions, and to ground decision-making processesin evidence. For this, an extensive portfolio of instru-ments for empirical software engineering has beendeveloped. For instance, Wohlin et al. [27] provide acollection of instruments, e.g., controlled experiments,surveys and case studies, to be used for empirical stud-ies in software engineering. Kitchenham et al. [13]extended these instruments by a detailed guideline forconducting systematic reviews. For most of the basicinstruments used in empirical software engineeringtoday, extended and more detailed (pragmatic) guide-

lines exist, such as for systematic reviews [16, 28],systematic mapping studies [23, 24], multi-vocal re-views [10, 11], or surveys [12, 21]. All these instru-ments are meant to support researchers and practi-tioners alike in conducting empirical studies and toground their work and decisions in evidence.

Conducting empirical studies is challenging andrequires careful preparation and a disciplined workapproach. Quite often, students consider empiricalstudies of little to no help when it comes to softwaredevelopment and project work, since the relation toactual development tasks is not obvious. Yet, manyof today’s applications rely on data, e.g., machinelearning systems like text and speech recognition, IoTdevices, and autonomous cars. Empirical methodsas such are about data analysis and, thus, providea suitable approach to teach data analysis—or dataengineering in general—that is a core competencein data-intensive applications. Furthermore, modernsoftware development paradigms, such as DevOpsincluding continuous integration and deployment, uti-lize data, e.g., to analyze a system’s performance, topredict defects, and to make informed decisions inthe development process as practiced in continuousexperimentation [5]. Therefore, it is necessary forteachers to open the students’ minds for a rigorousand evidence-based work approach.

In this paper, we present and evaluate the concept ofcourse-integrated mini-projects to teach empirical soft-ware engineering instruments. Our approach helpsstudents learn how to conduct empirical studies andunderstand the instruments and challenges comingalong with such studies. Collaborating project teamsconducting small empirical studies form the basis ofour approach. We implemented the approach in twocourses at the University of Southern Denmark (2016,68 students) and University of Innsbruck (2017, 33students). Mini-projects allow students to learn em-pirical instruments by practically applying them. Ourevaluation shows a positive learning experience andan improved understanding of (empirical) softwareengineering concepts. More than half of the studentsperceive empirical studies helpful for their later ca-

V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 75

reers. Our evaluation also shows that notably datacollection activities (e.g., for surveys and experiments)are underestimated by the students. Our findings thuslay the foundation for improving research-orientedcourses that require data and data analysis.

The remainder of the paper is organized as follows:Section 2 gives an overview of background and re-lated work. Section 3 describes the mini-project ap-proach, and Section 4 presents the approach’s evalua-tion based on two implementations at the Universityof Southern Denmark and the University of Innsbruckrespectively. We conclude the paper and discuss futurework in Section 5.

2 Background and Related WorkUsing empirical studies in software engineering ed-ucation is not a new idea [2]. However, empiri-cal studies—notably (controlled) experiments—aremainly used as a tool to support research using stu-dents as subjects, but got little appreciation as a teach-ing tool in software engineering in the first place.That is, students only get in touch with empiricalstudies as subjects in an empirical inquiry, and theyhave to carry out tasks, e.g., in an experiment as forinstance reported in [7, 8, 18, 20]. Yet, teaching em-pirical software engineering as a subject requires asetup in which empirical studies are the main sub-ject or at least provide a significant contribution toa course. In this regard, Wohlin [26] proposes threelevels for integrating empirical studies in softwareengineering courses: (i) integration in software en-gineering courses, (ii) as a separate course, and (iii)as part of a research method course. Wohlin men-tions that introducing empirical software engineeringwill provide more opportunities to conduct empiricalstudies in student settings, but that educational andresearch objectives need to be carefully balanced. Dil-lon [4] comes to the same conclusion and considersa successful observation of a phenomenon as part ofan empirical study not be an end in itself. Studentsneed time to get familiar with ideas and conceptsassociated with the phenomenon under observation.Finally, Parker [22] considers experiments distinctiveand more participative. Students are likely to remem-ber lessons associated with experiments.

In this paper, we present an approach that considersempirical studies major subjects of a course and thatuses such studies as teaching tool. Referring to estab-lished learning models such as Bloom’s Taxonomy ofLearning [1] and Dale’s Cone of Learning [3], we aimto include as many active learning parts as possiblein the courses. Still, we use passive learning methodsto transfer knowledge about theoretical basics, suchas methods and their application contexts. Address-ing the active learning levels, however, is challenging.In “ordinary” software engineering education, projectcourses are used to train software project work. Forempirical studies, it is required that the students carry

out actual research to practice the application of theempirical instruments. In our previous work [17–19],we presented different, self-contained classroom ex-periments and developed a guideline to select the best-fitting study type for a specific context [6]. In [14], weintroduced a teaming model that helps implementingempirical studies in larger project courses.

We contribute a generalized concept grounded in[6,14] that allows for including empirical studies ascourse units. We implemented and evaluated ourapproach in a course on empirical software engineer-ing and an advanced course on software engineeringand demonstrate how to implement and scale course-integrated empirical studies.

3 Course-Integrated Mini-ProjectsWe present the course-integrated mini-projects ap-proach in Section 3.1. The presentation includes thedescription of the team setups, project and task de-scriptions, and examples for which we present detailsin Section 3.2. Section 3.3 demonstrates two inte-gration strategies: the first integration strategy is aself-contained course on empirical software engineer-ing [14] and the second strategy is a topic-specificpart of an advanced software engineering course.

3.1 Mini-Projects and Project TeamsFigure 1 shows the general organization model for themini-project approach. A MiniProject has a Topic, aSchedule, and optional Reference Literature andInput Data. It is always carried out using at least oneMethod, e.g., an experiment [27], a case study [25],and a survey [21]. Finally, every mini-project consistsof a Project Team and an Advisory Team, which wedescribe in more detail in the following.

Service Team

joint research

shared research design

1

1..*

Mini Project

Advisory Team Project Teamprovideadvice

requestadvice1

1..*

I

S

0..*

0..*

1

Student

1

2..*

Teacher

External Advisor

Advisor

1

- Topic [1]- Schedule [1]- Method [1..*]- Reference Literature [*]- Input Data [*]

Practice Team

Method Team

Result1..*1

Figure 1: Overall organization model of mini-projects.

Advisory Teams An advisory team bundles all advi-sors involved in a specific mini-project—usually oneor two persons. An Advisor is either the teacher ofthe course or an external advisor, such as an external

Using Mini-Projects to Teach Empirical Software EngineeringMichael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal


Style Description

Isolated The “normal” way of doing a mini-project isthe isolated way of working. Isolated meansthat a project team has a self-contained taskthat can be worked on without any interactionwith other teams.

Joint This style is applied if project teams collabo-ratively work on a joint (research) project. Acomplex project is broken down into a num-ber of smaller projects. Project teams thushave to be coordinated in terms of task distri-bution, scheduling, and result synthesis.

Shared This style is applied if project teams competi-tively work on the same (research) topic. Twoor more teams are assigned the same task;team-specific methods can be varied and re-sults can be compared. That is, this style helpsconducting controlled experiments or imple-menting independently conducted studies.

Table 1: Overview of the different interaction stylesamong mini-project teams.

project topic sponsor [14]. Besides offering and pro-moting project topics, advisors regularly interact withthe project teams for which they handle individualsupport requests, and they provide general technicaland methodical support.

Project Teams A project is composed of all studentsworking on a specific problem. Project teams caninteract with each other. We distinguish the three in-teraction styles isolated, joint, and shared, which areexplained in Table 1. Furthermore, we distinguishthree types of project teams according to the typeof task they are working on: a Practice Team per-forms an “active” task, e.g., a development task or aresearch task. A Method Team deals with methodologi-cal expertise, i.e., it develops competencies regardingspecific (scientific) methods and offers “consultancyservices” in terms of applying a specific method “right”to practice teams. Finally, a Service Team developsskills in more general topics, such as data analysis orpresentation, and offers respective “services” to otherteams—practice teams and method teams alike.

3.2 Project- and Task DescriptionsEvery mini-project is supposed to produce at leastone Result. In this section, we provide a blueprintfor a 1-page project- and task description, which alsoillustrates the manifestation of the different attributesof the class Mini Project (Figure 1).

For every project, a description that includes tasks,dates, and expected results is necessary. Table 2 pro-vides a summary and a description of task-descriptionitems that we consider relevant. The work descrip-tion requires special attention as it comprises the de-tailed activity list, input material, and the descriptionof expected results. The expected results are speci-

Item Description

Metadata This section contains all information rele-vant to a task, e.g., hand-in date.

Title A project needs a telling title and an IDContext This section briefly describes the context of

the project and provides a short summaryof the basic tasks. Recommendation: thecontext section should be treated as an ab-stract, such that it can be used as a teaserand a small piece of information, e.g., usedin a course management system.

WorkDescription

The detailed work description contains atleast:

1. A detailed task list.

2. A list of input/reference material.

3. A list of deliverables to be shipped.

Note: The level of detail depends on theactual task, i.e., for an “explorative” task,the description needs to be more open whilea specific development task requires a moredetailed task description.

Schedule The basic schedule lists all deadlines andthe respectively expected results.

RelatedProjects

Our concept allows for collaborative andcompetitive work (Figure 1 and Table 1). Ifsuch a collaborative/competitive work styleis implemented, this section provides the in-formation about the other teams. If methodor service teams provide useful services toa project, such teams are referred here too.

Literature This section lists selected reference litera-ture relevant to a project.

Table 2: Structure of a mini-project task description(recommended minimal elements).

fied right here; alternatively, a separate catalog of re-sults has to be provided, e.g., including templates andmandatory/recommended outlines. Relevant typesfor project outcomes are, e.g., (research) data1, es-says or reports, presentations, tutorials, and software.The second important item of the task description isthe list of related projects. For instance, if a task is acollaborative task, this list refers to all related projectsthat contribute to the overall project goal. Further-more, this list also refers those method and serviceteams that provide useful support, e.g., if the projectis concerned with developing and conducting a survey,this list can refer to a method team that is focusedon the theoretical aspects of survey research. Finally,the task description can also be used to develop achecklist, which is used for the final submission. Thischecklist helps students to check if their delivery pack-age is complete, and it helps teachers validating thedelivery and grade its components. Figure 2 shows

1Recommendation: If students conduct a research task, analyzeddata that is necessary for the project documentation must alwaysbe complemented with the original raw data.



Figure 2: Example of a mini-project task description.

a practically used example of a task description asdescribed in Table 2. This task description is takenfrom the course given at University of Southern Den-mark and describes a survey research task, whichwas performed collaboratively with two external re-searchers [9]. This particular project was a collabora-tive project. Specifically, two teams (No. 15 and 16;see the related projects part) were assigned one task,but had to conduct the survey with different targetgroups. Each team submitted its own data and report,but, both teams gave only one joint presentation.

3.3 Course-Integration StrategiesWe describe two implementation strategies for thepresented approach using two graduate courses. Forthese implementations, we provide a short summaryof the respective courses and their learning goals, andwe provide an overview of the projects implementedin the respective courses (Table 3). Furthermore, thissection lays the foundation for the approach’s evalua-tion, which is presented in Section 4.

3.3.1 Implementation as a Self-ContainedEmpirical Software Engineering Course

A course on the Scientific Method (SCM, Universityof Southern Denmark, SDU, 2016) implemented thepresented approach as a self-contained master-levelcourse. Figure 3 illustrates the overall organization ofthe SCM course showing the introduction parts andthe active learning/project parts.

The goal of the SCM course was to teach empiricalsoftware engineering as main subject by letting the stu-dents perform small studies themselves [14]. Specifi-

Study Type SCM ASEGr Top Gr Top

Theory (tutorial) 3 9 9 7

Experiment 3 1 1 3 2 1Survey 3 4 2 3 3 1Systematic Review 3 3 2 3 1 1Mapping Study 3 1 1 7

Simulation 3 2 2 7

Data Mining Study 7 3 7 3

Table 3: Overview of the study types implementedincluding the number of groups for a specific method(Gr) and the number of topics per study type (Top).

Lect

ure

Exe

rcis

e M

odel

Sel

f-di

rect

ed le

arni

ng: w

orki

ng o

n th

e se

lect

ed to

pics

, e.

g., S

LRs

or s

urve

ys (i

ncl.

pres

entin

g an

d w

ritin

g)

Introduction and Fundamentals

theory experts help practice

teams…

Presentation and Scientific Writing

Paper Reviews

Introduction to Empirical Research

Presentations: Theory and Tutorial Topics

Status Control and Guest Lectures

Status Control and Guest Lectures

Presentations: Secondary Studies

Presentations: Experiments and Simulations

Presentations: Survey Research

Group Papers’ Peer Review

Evaluation and Wrap-Up

Figure 3: Overview of the topics and the general orga-nization of the SCM course.

cally, the major learning goals of the SCM course weredefined as follows:

• After the course, students know the basic terminol-ogy and the key concepts of the scientific method.

• After the course, students know and understandthe most important empirical research methods.

• After the course, students have shown their abilityto practically apply one research method, conductand report on a small research project.



In total, 30 research topics were presented to the stu-dents. According to their preferences, students couldapply for up to three topics, which built the founda-tion for the final team setup. Finally, 20 projects werestarted with two to three students each. Besides thetheoretical topics, five research methods were coveredby the projects (Table 3).

The SCM course implements the concept from Fig-ure 1 as follows: method teams became theory teamsfor those students that did not want to carry outa study, but wanted to learn more details about aspecific method or technique, e.g., the systematic re-view method [13]. Service teams became cross-cuttingteams that build up a specific expertise and consultedtheory and practice teams. The 20 teams were con-nected with each other, e.g., a theory team supportedone or many practice teams, and both were supportedby cross-cutting teams. The teacher supervised theindividual teams as well as the groups of collaboratingteams. The teams were formed right in the first ses-sion of the course and, thus, the projects became themain subjects to build the learning experience upon.3.3.2 Integration in an Advanced Software

Engineering CourseA course on Advanced Software Engineering (ASE, Uni-versity of Innsbruck, UI, 2017) implemented the pre-sented approach as part of a master-level course onsoftware engineering in which empirical studies com-plemented the (technical) software engineering topics.These technical software engineering topics were or-ganized around the concept of models in softwareengineering and covered software process models (in-cluding agile process models), modeling languagesincluding UML and DSLs, model transformations aswell as predictive models, e.g., for defect prediction.Figure 4 illustrates the overall organization of the ASEcourse showing the introduction parts and the activeproject parts.

The overall goal of this course was to teach studentsadvanced topics in software engineering and to let stu-dents make the experience of the value that empiricalstudies have to support software engineering activi-ties. Specifically, the major learning goals of the ASEcourse were defined as follows:

• After the course, students know and understanddifferent advanced software engineering concepts.

• After the course, students know the basic empir-ical research methods and know how to utilizeempirical studies in the different software engi-neering activities.

• After the course, students have shown their abil-ity to practically apply one research method to aspecific software engineering activity, conduct andreport on a small research project.

In total, eight topics have been proposed to the stu-dents and students could apply for the topics. Finally,

Tech

nica

l Sof

twar

e E

ngin

eeri

ng

Lect

ures

Sel

f-di

rect

ed le

arni

ng: w

orki

ng o

n th

e se

lect

ed

topi

cs, e

.g.,

SLR

s or

sur

veys

(inc

l. pr

esen

ting

and

wri

ting)

Overview Software Engineering

Empirical Methods in Software Engineering

Software Process Models

Overview: Empirical Methods in Software Engineering (Experiments, Surveys, Data

Mining, Secondary Studies)

Presentation of Project Topics

Complementing Material/Activities:

Topic-specific empirical studies

and tutorials

Topic Selection and Team Building

Project Evaluation

Modeling Languages

Model Transformations

Predictive Models

Initial Project Feedback

Final Project Presentations

Final Project Feedback

Weekly Standup (Project Progress)

Figure 4: Overview of the topics and the general orga-nization of the ASE course.

13 projects were started with two to three studentseach. The projects covered four research methods andsix topics (Table 3).

The concept from Figure 1 was implemented as fol-lows: the 13 project teams were formed as practiceteams. Since the empirical studies were designed tocomplement selected topics of the ASE course, no ex-plicit method teams or service teams have been formed.That is, all practice teams worked on specific topicsand developed the technical skills and parts of therequired methodological skills themselves. Additionalmethodological skills were delivered to the teams bythe teachers and guest lectures, who also acted asadvisors. Teams were formed when the mini-projectswere assigned to the students.

4 Evaluation

In this section, we present the research design andevaluation strategy in Section 4.1, the results and adiscussion in Section 4.2, and threats to validity inSection 4.3.



Research Questions

RQ 1 Do course-integrated empirical studies (mini-projects) help students improving their work ap-proach? We aim to study if course-integratedempirical studies (mini-projects) help studentsbetter understand the value of structured scien-tific work approaches. For this, we investigatedthe following detailed questions:

RQ 1.1 Do mini-projects support a better/more effectivelearning?

RQ 1.2 Do mini-projects support a better understandingof concepts?

RQ 1.3 What are the perceived learnings of mini-projects?

RQ 2 Do course-integrated mini-projects help studentsbetter understand the role of empirical studies?We aim to study the general attitude towardsempirical studies, i.e., do students change theirattitude once they actively conducted an em-pirical study. For this, we investigated twodetailed questions:

RQ 2.1 Do mini-projects change the attitude towardsempirical studies?

RQ 2.2 Do course-integrated empirical studies studieshelp understanding challenges (revealing mis-conceptions)?

RQ 3 What are the perceived pros and cons of themini-project approach? We aim to study dis-/advantages perceived by students that partici-pated in mini-projects.

Table 4: Overview of the research questions studied.

4.1 Research Design and EvaluationStrategy

We evaluated our approach in two master-levelcourses at two universities and by surveying the par-ticipating students. In this section, we present ourresearch questions, outline the survey instrument, anddescribe the data collection strategies.

Research Questions To evaluate the proposed mini-project approach, we aim at answering the researchquestions listed in Table 4. Our three top-level re-search questions address the (general) learning ex-perience, the usefulness of the approach in terms ofimproving the understanding of the role of empiricalstudies, and the perception concerning the pros andcons of the approach presented.

Data Collection To collect the data, we (initially)developed two online questionnaires for the SCMcourse based on Google Forms [14]. The first question-naire was used in a mid-term evaluation; the second(extended) questionnaire was used for the final eval-uation. While preparing the ASE course, we revisedboth questionnaires and conducted the first data col-lection before we started the mini-projects in the ASE

course, and the second data collection, again, in thecourse’s final evaluation when the mini-projects havebeen finished. All four questionnaires including asummary are available online2.

Our questionnaire design allows for a 2-staged datacollection that helps observing changing student per-ceptions and evaluating the courses over time. Thequestionnaires share a number of questions to allowfor comparing courses, the implementations of ourapproach, and to qualitatively analyzing the studentfeedback. As a learning from the SCM data collection,the two ASE questionnaires put more emphasis on thesingle phases of the scientific workflow, e.g., by specif-ically asking for challenges and difficulties regardingthe design of research instruments and conductingthe data collection. Different to the questionnairesused in the SCM course, is the ASE questionnaires,students were asked to provide nicknames (to ensureanonymity), such that tracking individual studentswas possible to evaluate specific ratings and to evalu-ate such ratings over time.

Analysis Procedures All four questionnaires pro-duce quantitative and qualitative data. For the quanti-tative analysis, we primarily use descriptive statisticsto analyze the four measurements individually, overtime per course, and for analyzing both courses. Fur-thermore, due to the questionnaire’s evolution, forthe ASE course we could conduct additional inferen-tial statistical analyses, e.g., hypothesis testing andcorrelation analysis.

To qualitatively analyze the data, we used the free-text answers provided by the students. For the ASEcourse, an analysis of general learning and learningoutcomes was performed using the questions for theexpected learning outcomes, and the questions aboutthe learning regarding the mini-project topics andthe way of conducting empirical research (Appendix;variables MP6–MP8). An overall analysis of the courseas such (in both courses) was performed using thequestions for the courses’ appropriateness, the lecturesand exercises, and the perceived relation to practice(Appendix; variables GC2–GC4; interpreted as schoolgrades). The coding of the feedback into categorieswas jointly performed by the two authors.

Validity Procedures To mitigate threats and to en-sure the validity of the instrument, we reused a ques-tionnaire design that was already applied to othercourses and that received an external quality assur-ance [15,18]. The original questionnaire design wasextended by specific questions to evaluate the suitabil-ity of the mini-project approach.

Demographics In total, 68 students were enrolledin the SCM course of which 39 students participated

2Appendix: https://kuhrmann.files.wordpress.com/2018/11/appendix-draft.pdf



Category SCM, n=38 ASE, n=29Mean SD Mean SD

Course complexity 2.87 0.66 2.62 0.55Course speed 3.05 0.76 2.83 0.38Course volume 2.58 0.91 2.10 0.76

Relation to Practice 2.11 0.99 2.34 1.03

Table 5: Results of the final course evaluation.

in the initial evaluation, and 38 in the final evaluation.In the ASE course, 33 students were enrolled, and 29students participated in both evaluations. From the29 ASE-students, 27 provided a nickname that wasused in the subject-based analyses. Table 5 gives anoverview of the general course evaluation. Studentsof both courses perceived the course complexity, speedand volume as moderate and see a good relation ofthe course to practice.

4.2 Results and DiscussionThis section presents the findings of the evaluation ofour proposed course-integrated mini-project approach.The following sections present the findings accordingto the research questions described in Table 4.4.2.1 RQ 1: Improved Work ApproachWith RQ1, we aim to study if course-integrated mini-projects help students understand the value of a struc-tured scientific work approach. To better structurethe findings, we defined three sub-questions (Table 4),which we discuss in the following.

RQ 1.1: Support for a better learning This sub-question is addressed by the answers to the state-ment: “The mini-projects improve the learning experi-ence”, which was quantitatively analyzed.

17

9

19

8

1

6

1

5 1

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

SCM

ASE

Fully Agree Somewhat Agree Indifferent Somewhat Disagree Fully Disagree

Figure 5: Results for the learning experience fromfinal questionnaires (SCM: n=38, ASE: n=29).

Figure 5 shows the results (taken from the finalevaluation) for both courses and shows that 95% ofthe SCM-students consider the mini-project approachcontributing to an improved perceived learning expe-rience (3% each rate the teaching format neutral orless effective than other teaching formats). For theASE course, 59% consider the mini-project approachmore effective, and 21% each rate it neutral or lesseffective. In summary, mini-projects contribute to animproved perceived learning experience, especially inthe SCM setting, but also in ASE, where mini-projectswere only one part of the course.

RQ 1.2: Support for a better understanding of con-cepts A key to provide value to the students is tomake software engineering concepts better/easier tounderstand. For this, students were asked to rate thestatement: “The mini-projects helped me understandingconcepts better”, i.e., whether or not the understandingof concepts of interest has been improved. In this con-text, an investigation of the role of empirical studies isprovided in Sect. 4.2.2. Again, the majority of the stu-dents (92% for the SCM course and 69% for the ASEcourse; Figure 6) considers the mini-project approachadvantageous for gaining a better understanding ofsoftware engineering concepts.

13

6

22

14

2

5

1

3 1

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

SCM

ASE


Figure 6: Results for the improved understandingfrom final questionnaires (SCM: n=38, ASE: n=29).

RQ 1.3: Perceived Learning The question for theperceived learning is answered using five statements(Appendix, MP2.4–MP2.8). In particular, we were inter-ested to learn about the perceived impact to the latercareer (“The practiced scientific work approach will helpme in my later career.”) and shareable expertise builtin the course (“I built a specific exercise that I couldshare with other teams.”), and a retrospective ratingof the group work in the mini-projects (looking back:“contributed to my learning expericence”, “team work[..] was good” and “collaboration [..] was good”).

Figure 7 shows the aggregated results for the per-ceived learnings. Approximately 63% of the SCMstudents and 59% of the ASE students think that thecourses provide take-aways that will have a positiveimpact on their later careers. Concerning the share-able expertise, 53% of the SCM students state thatthey have obtained knowledge and expertise that theycan share with others; 29% are indifferent. In the ASEcourse, even though the course has more “practical”elements, only 41% of the students think that theybuilt a shareable expertise, but 48% are indifferent.

Concerning the general perceived learning experi-ence and the teamwork within the project team, thevast majority of the students rate the courses as goodand very good. However, the cross-team collaborationshows a different picture—notably in the SCM coursein which interdisciplinary work was enforced by thecourse design. In the SCM course, 29% of the studentsconsidered the cross-team collaboration good to verygood, but 55% rated the cross-team collaboration badto very bad. Analyzing this phenomenon, we foundthe necessity for the different teams to interact witheach other to obtain required knowledge from other



13

10

16

13

3

8

4

2

9

7

21

11

13

10

8

8

16

10

15

10

3

4

7

2

6

7

11

14

9

6

1

2

2

2

14

2

6

2

2

3

2

2

7

4

1

1

3

3

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

SCM

ASE

SCM

ASE

SCM

ASE

SCM

ASE

SCM

ASE

Lear

ning

expe

rienc

eTe

amw

ork

Cro

ss-te

amco

llabo

ratio

nEx

perti

se to

sha

reH

elp

in la

ter

care

er


Figure 7: Results for the perceived leanings from finalquestionnaires (SCM: n=38, ASE: n=29).

teams by also trying to keep their own schedule themost disappointing aspect. On the other hand, wefound an “understanding” for this kind of work, whichreflects reality in interdisciplinary collaboration and,thus, students eventually considered this a significantlearning. Considering the ASE course, we wanted tolearn whether a similar behavior can be observed. AsFigure 7 shows, cross-team collaboration is still con-sidered a problem, even though the heterogeneity ofthe project teams and thus the need to collaboratewas reduced.

An in-depth analysis of the perceived learnings ofmini-projects was performed by qualitatively codingthe responses of the free-form text questions (GC1:“What was your major take-home asset [..]?”, MP7:“What did you learn about the topic of your researchproject?” and MP8: “What did you learn about empir-ical research?”). For GC1, 26 students from the SCMcourse provided feedback. In the ASE course, 27 stu-dents provided feedback for MP7 and MP8. In total,we extracted 38 statements from the SCM course and60 statements from the ASE course (for both ques-tions). The students’ statements were categorizedbased on keywords, and the threshold for building acategory was set to three references.

Table 6 provides the condensed qualitative feed-back on the perceived learnings in eight categories.Summarized, topic specific learnings (e.g., applicationof DSLs or comments in programming languages) aswell as learnings related to the application of empiri-cal methods (e.g., formulation of research questionsor application of specific empirical methods like sur-veys) were frequently mentioned. Also, data manage-ment (i.e., data collection, preparation and analysis

Category Total SCM ASE

Topic specific (i.e., mini-project topics) 16 2 14Empirical methods (e.g. experiments) 16 6 10Reporting findings (from studies) 13 13 0Importance, meaning (emp. research) 12 2 10Data management (in studies) 11 1 10Effort (to plan/conduct a study) 10 0 10

Technical skills 3 2 1Soft skills 17 12 5

Table 6: Qualitative feedback for the perceived learn-ings of mini-projects from final questionnaires.

of data) and reporting results of empirical studies arehighlighted. Students experienced that conducting anempirical study causes effort and might be a complexendeavor (“It is hard to get results, which have a strongmeaning”). On the other hand, students also built anunderstanding of the importance and the meaning ofempirical research in software engineering (“The topicexists and is very useful if done correctly”). Finally, stu-dents hardly report learnings regarding technical skills(e.g., using LATEX or R). However, manifold learningsabout soft skills (e.g., reviewing techniques) are re-ported, notably concerning teamwork and cross-teamcollaboration (see also Figure 7).4.2.2 RQ 2: Improved UnderstandingThis research question aims at investigating if studentsbuilt an understanding of the role of empirical stud-ies. Specifically, if students consider empirical studiesa valuable instrument to complement the technicalsoftware engineering activities in a beneficial way.

RQ 2.1: Changed Attitude towards Empirical Stud-ies To learn about the students’ understanding ofthe value of empirical studies, we asked the studentswhether their view on empirical studies has changedonce they actively conducted an empirical study them-selves (“I like the mini-project part”).

9

5

23

10

3

3

1

9

2

2

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

SCM

ASE


Figure 8: Results for the changed view on sciencefrom final questionnaires (SCM: n=38, ASE: n=29).

Figure 8 shows that 84% of the SCM studentschanged their view on science and the value of em-pirical studies after conducting an empirical studythemselves. The ASE course provides a different pic-ture. Still 52% of the students changed their view, butalmost 38% did not change their view on science andempirical studies.



RQ 2.2: Challenges of Empirical Studies Besidesthe general perception of science and empirical stud-ies, we are also interested in specific challenges. Forthis, we revised the questionnaire (see Appendix) andadded questions that explicitly address the scientificwork process.

Activity Results p-value

Definition of research questions V = 80 0.5257Working with scientific literature V = 55.5 0.4927Design of research instruments V = 68 0.3254Implementation of instrument V = 43 0.7808Data collection V = 135 0.0277Performing data analysis V = 107 0.9529Writing the report V = 106 0.3698

Table 7: Results of the paired Wilcoxon signed-ranktest for the ASE course (n=27).

To study the perceived challenges students facewhen implementing empirical studies, we asked thestudents to rate the different parts of the scientificwork process (Appendix, variables MP5.1–MP5.7; seealso Table 7). Furthermore, 27 students enrolled inthe ASE course provided a nickname, which we usedto investigate challenges over time by evaluating theinitial and the final questionnaires. We performeda Wilcoxon signed-rank test to compare if there aresignificant differences between the initial perceptionof the scientific work process and the final one afterthe study has been performed. Table 7 shows the testresults for the various parts of the work process.

The results show that only for the activity data col-lection there is a significant difference (p < 0.05).This indicates the perceived difficulties and challengesregarding the data collection changed for the partici-pating students. A Spearman rank correlation coeffi-cient for the initial and final feedback on the level ofchallenges regarding the data collection is low (ρ =0.2775), which further indicates that there is only aweak positive correlation between between the datacollection challenges perceived initially and the chal-lenges perceived at the end. Figure 9 shows the uncor-related perceived challenges regarding the differentactivities in the scientific work process (collected inthe ASE course; initial and final questionnaire).

We conclude that the data collection is an activityin empirical studies for which challenges can be easilyover- and underestimated. When teaching empiricalsoftware engineering it is therefore important to putspecial emphasis on the important role of data collec-tion and its challenges to prevent students from over-or underestimating the required effort.4.2.3 RQ 3: Perceived Dis-/AdvantagesThe third research question aims to investigate theperceived advantages (“pros”) and disadvantages(“cons”) of the mini-project approach. For this, wequalitatively analyzed the students’ feedback by cod-

2

2

3

2

2

2

4

2

6

1

1

3

3

6

8

7

4

4

7

5

3

3

6

4

6

3

6

6

6

8

9

6

6

9

13

13

11

10

11

8

7

12

11

11

10

10

12

10

7

8

5

8

8

11

12

1

2

1

4

1

1

1

1

3

3

2

1

2

3

2

3

1

3

1

2

3

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Initial

Final

Initial

Final

Initial

Final

Initial

Final

Initial

Final

Initial

Final

Initial

Final

Def

initi

on o

fR

esea

rch

Que

stio

ns

Wor

king

with

Scie

ntifi

cLi

tera

ture

Des

ign

ofR

esea

rch

Inst

rum

ents

Impl

emen

tatio

nIn

stru

men

tD

ata

Colle

ctio

nPe

rform

ing

Data

Anal

ysis

Writ

ing

the

Rep

ort

St raightforward Fairly easy Indifferent Somewhat diff icult Very difficult Not applicable

Figure 9: Overview of the perceived challenges on thedifferent scientific work activities in the ASE initialand final questionnaire (n=29).

ing the responses of the free-form text questions (GC2:“Up to 5 things that were good” and GC3: “Up to 5things that were bad”).

In the SCM course data for GC2 and GC3 was col-lected in the initial and final questionnaire. For theASE course, data was collected in the final evalua-tion only. Hence, for the data analysis presented inthe paper at hand, we only consider the data col-lected in both final evaluations. In the SCM course, 29students provided comments in the final evaluation.Respectively, 21 ASE students provided feedback onperceived dis-/advantages. In total, we extracted 72pro- and 42 con-statements from the SCM feedback,and we extracted 54 pro- and 23 con-statements fromthe ASE feedback. Both feedback sets were catego-rized and analyzed based on keywords (qualitativecoding), whereas the threshold for a category was setto three mentions. Table 8 provides the aggregatedqualitative feedback on the perceived pros and cons ofthe mini-project approach in nine categories groupedby SCM, ASE, and in total.

General Perception In summary, the mini-projectapproach was seen very positive. For both coursesSCM and ASE, the students identified considerablymore pros than cons on the mini-projects. In both set-tings, i.e., SCM (a self-contained empirical softwareengineering course) and ASE (a part of a software



Category SCM ASE Total- , - , - ,

Structure, organisation 18 14 9 11 27 25Knowledge transfer 18 3 3 2 21 5Mini-projects 14 3 6 2 20 5Research, tech. skills 6 2 8 1 14 3Topics 6 4 7 0 13 4Relevance 3 3 9 0 12 3Guest lectures 5 7 6 0 11 7Group work 2 1 6 1 8 2Effort 0 5 0 6 0 11

Table 8: Qualitative feedback on the perceived prosand cons of mini-projects from final questionnaires.

engineering course), the obtained research-relatedand technical skills were perceived very positive. Thetopics covered in the courses were positively evalu-ated as well; especially in the ASE course in whichmini-projects were integrated as a part of a softwareengineering course and focused on current (hot) top-ics in software engineering. Feedback and knowledgetransfer were especially highlighted and consideredpositive in the SCM course with its different types ofcollaborating teams (dedicated practice, method andservice teams), in-depth introductions to empiricalmethods, and introductions and exercises in scientificreading and writing. Also, group work was generallyconsidered positive, yet, the ASE course with its moreuniform teams was perceived more advantageous. Asalready discussed in Sect. 4.2.2, the setting from theSCM course suffers from the teams’ heterogeneity andthe necessity to establish a cross-team collaboration,which caused more effort for the project teams.

Guest Lectures In both courses, guest lectures weregiven by external researchers. Considering the out-comes from Table 8, guest lectures in the ASE coursewere considered positive, whereas the guest lecturesin the SCM course received a more indifferent rating(with a slight tendency towards a negative evaluation).We argue that this perception is related to the selec-tion of the speakers rather than the course setting, yet,this remains subject to further investigation.

Course Organization From the organizational per-spective, the courses were perceived indifferent and arelatively high number of positive and negative com-ments was provided. On the positive side, studentshighlighted the organization and structure in general(“Organization was good”), and, in particular, alsothe course assessment mode (“Fair grading system”)and the sequence of activities (“The mini-projects werestructured well—good planning of when to do the work,when to make a presentation and when to turn in thepaper”). On the negative side, the overall flow wasmentioned (“Things started to slow down way too muchafter the first 5 lectures”) as well as the speed of the lec-

ture (“A little slow lectures”) and the general schedul-ing and coordination of the lecture with other obli-gations (“During examination time, time overlap withstudying”).

Effort Finally, the effort caused by the courses wasperceived negative in both settings. Feedback for theSCM course says “The volume does not fit well a 5 ECTSpoint course” and, respectively, for the ASE course“Sometimes a single task was too big”. This feedback re-flects a side-effect of project work that typically causesmore effort than closed tasks—or even “listen-only”classes. However, such comments motivate a revisionof the projects, e.g., splitting tasks into smaller unitsto better support continuous work and to reduce theperceived effort in a short time frame, but still keepthe learning effect of performing empirical researchas we received it also from the SCM comments “in myopinion learning the scientific method without workingwith the methods it’s only knowing about the methods,not learning them.”

4.3 Threats to ValidityWe discuss issues, which may have threatened theconstruct, internal and external validity as well asmeasures to mitigate them.

Construct Validity The construct validity might bethreatened by the two different implementations ofthe approach presented and the instrument used forits evaluation. Although the two courses differed withrespect to the applied integration strategy for mini-projects, for both courses, we used the same yet tai-lored questionnaire and combined the responses. Toincrease construct validity, we developed the ques-tionnaire from an external source [15], which bothauthors reviewed and evolved. Furthermore, we col-lected and combined quantitative and qualitative datato answer and discuss the different research questions.

Due to the overall setup, the questionnaires differedbetween the courses. Hence, some analyses like theindividual-based investigation of challenges over time(RQ 2.2) were possible in the ASE course only. Fur-thermore, analyses regarding perceived learnings ofthe students had to be performed using different ques-tions (SCM: GC1, ASE: MP7 and MP8; see Appendix),which was handled using a multi-staged coding pro-cess that resulted in common categories.

Internal Validity The internal validity might bethreatened by the rather low number of participantsand the participants’ self-reporting, which both mightaffect the relationship between course-integrated mini-projects and the investigated effects on learning andunderstanding of concepts and the role of empiricalstudies in software engineering. To mitigate thesethreats, we studied the integration of mini-projects



in two settings with an acceptable number of partici-pants (SCM: 39 and ASE: 29). We also introduced thequestionnaire to the students to minimize the risk ofmisinterpretation.

To triangulate the results of quantitative analysisand to investigate the relationship between course-integrated mini-projects and their effects holistically,we also applied qualitative analysis to analyze theresponses of the participating students.

External Validity The external validity might bethreatened by the issue of the rather low number ofsettings in which the course was performed and eval-uated. However, we implemented and evaluated thecourse for each of the two course integration strategiesproposed in Section 3.3, i.e., self-contained empiri-cal software engineering course and integration insoftware engineering course. Two researchers fromtwo different institutions were involved in preparing,conducting and evaluating the courses. Furthermore,we combined quantitative and qualitative data to geta broader view on teaching empirical software engi-neering with course-integrated mini-projects.

The participants’ self-reporting might also affect thegeneralizability of the results. To mitigate this threat,we introduced the questionnaire to the students tominimize the risk of misinterpretation. Since the pa-per at hand is an initial study, we consider furtherin-depth analyses, e.g., of course artifacts like finalreports, and replications of the courses as well as thetransfer of the approach presented and its evaluationas future work.

5 ConclusionIn this paper we presented and evaluated an approachto teach empirical software engineering with course-integrated mini-projects. In such mini-projects stu-dents conduct small empirical studies in collaborat-ing teams within a software engineering course ora research methods course. We illustrated the ap-proach by presenting two integration strategies: firsta self-contained course on empirical software engi-neering given at the University of Southern Denmarkand second as part of an advanced software engi-neering course given at the University of Innsbruck.Both implementations were complemented by a studythat showed a positive learning experience and animproved understanding of (empirical) software engi-neering concepts.

More than half of the participating students stateas a perceived learning of the course that empiricalstudies are helpful for their later careers (systematicand evidence-driven work). In addition, a statisticalanalysis revealed that especially the data collectionactivities are underestimated by the students, whichallows for future improvements of university courses.We consider this aspect critical not only for empiricalsoftware engineering in particular, but also due to the

increased importance of Data Science, Machine Learn-ing and Artificial Intelligence courses or even programsin general. In all these setting, effective and efficientdata collection and preparation for further analysisis essential. Hence, we encourage other teachers toput special emphasis on all data-related activities, notonly the data analysis.

In summary, students provided an overall positivequalitative feedback on the course-integrated mini-projects as well as on the skills achieved and theirrelevance. This motivates to further explore and dis-seminate the presented approach. However, on thedownside, the students pointed out the overall flow ofthe courses and the perceived high effort to performmini-projects, which requires a further refinement ofthe courses, e.g., by splitting it into smaller tasks ofuniform granularity. In future, we will therefore reviseour approach accordingly and further disseminate it.We also plan to perform further in-depth analyses ofcourse artifacts, e.g., the students’ final reports, aswell as replications of the courses.

Finally, this paper presents an approach that hasbeen implemented twice and it also provides initialdata. To get further insights and to improve the dataavailable, we cordially invite other teachers to adaptand integrate our approach into their courses and toshare their experiences.

References[1] L. W. Anderson and D. R. Krathwohl, editors. A

Taxonomy for Learning, Teaching, and Assessing:A Revision of Bloom’s Taxonomy of EducationalObjectives, Abridged Edition. Pearson, 1st edition,2000.

[2] V. Basili, R. Selby, and D. Hutchens. Experi-mentation in software engineering. Trans. onSoftware Engineering, 12(7):733–743, 1986.

[3] E. Dale. Audiovisual methods in teaching. DrydenPress, 3 edition, 1969.

[4] J. Dillon. A Review of the Research on PracticalWork in School Science. Technical report, King’sCollege, 2008.

[5] F. Fagerholm, A. S. Guinea, H. Mäenpää, andJ. Münch. Building blocks for continuous experi-mentation. In Proceedings of the 1st InternationalWorkshop on Rapid Continuous Software Engi-neering, RCoSE 2014, pages 26–35, New York,NY, USA, 2014. ACM.

[6] F. Fagerholm, M. Kuhrmann, and J. Münch.Guidelines for using empirical studies in soft-ware engineering education. PeerJ ComputerScience, 3(e131), September 2017.

[7] D. Fucci and B. Turhan. A replicated experimenton the effectiveness of test-first development. In



International Symposium on Empirical SoftwareEngineering and Measurement, pages 103–112.IEEE, Oct 2013.

[8] D. Fucci, B. Turhan, and M. Oivo. Impact of pro-cess conformance on the effects of test-drivendevelopment. In International Symposium on Em-pirical Software Engineering and Measurement,pages 10:1–10:10. ACM, 2014.

[9] V. Garousi, M. Felderer, M. Kuhrmann, andK. Herkiloglu. What industry wants fromacademia in software testing?: Hearing prac-titioners’ opinions. In Proceedings of the 21st In-ternational Conference on Evaluation and Assess-ment in Software Engineering, EASE’17, pages65–69, New York, NY, USA, 2017. ACM.

[10] V. Garousi, M. Felderer, and M. V. Mäntylä. Theneed for multivocal literature reviews in soft-ware engineering: complementing systematicliterature reviews with grey literature. In Pro-ceedings of the 20th International Conference onEvaluation and Assessment in Software Engineer-ing, page 26. ACM, 2016.

[11] V. Garousi, M. Felderer, and M. V. Mäntylä.Guidelines for including grey literature and con-ducting multivocal literature reviews in softwareengineering. Information and Software Technol-ogy, 2018.

[12] M. Kasunic. Designing an effective survey. Tech-nical report, Carnegie Mellon University Pitts-burgh Software Engineering Institute, 2005.

[13] B. A. Kitchenham, D. Budgen, and P. Brereton.Evidence-Based Software Engineering and System-atic Reviews. CRC Press, 2015.

[14] M. Kuhrmann. Teaching empirical software en-gineering using expert teams. In SEUH, pages20–31, 2017.

[15] M. Kuhrmann, D. M. Fernández, and J. Münch.Teaching software process modeling. In Interna-tional Conference on Software Engineering, pages1138–1147, 2013.

[16] M. Kuhrmann, D. Mendez Fernández, andM. Daneva. On the pragmatic design of lit-erature studies in software engineering: anexperience-based guideline. Empirical SoftwareEngineering, 22(6):2852–2891, Dec 2017.

[17] M. Kuhrmann and J. Münch. Distributed soft-ware development with one hand tied behindthe back: A course unit to experience the roleof communication in gsd. In 1st Workshop onGlobal Software Engineering Education (in con-junction with ICGSE’2016). IEEE, 2016.

[18] M. Kuhrmann and J. Münch. When teams gocrazy: An environment to experience group dy-namics in software project management courses.In International Conference on Software Engineer-ing, ICSE, pages 412–421. ACM, May 2016.

[19] M. Kuhrmann and J. Münch. Enhancing soft-ware engineering education through experimen-tation: An experience report. In 2018 IEEE Inter-national Conference on Engineering, Technologyand Innovation (ICE/ITMC), pages 1–9, June2018.

[20] K. Labunets, A. Janes, M. Felderer, and F. Mas-sacci. Teaching predictive modeling to juniorsoftware engineers—seminar format and its eval-uation: poster. In Proceedings of the 39th In-ternational Conference on Software EngineeringCompanion, pages 339–340. IEEE Press, 2017.

[21] J. Linåker, S. M. Sulaman, R. M. de Mello, andM. Höst. Guidelines for conducting surveys insoftware engineering. Technical report, LundUniversity, January 2015.

[22] J. Parker. Using laboratory experiments to teachintroductory economics. Working paper, Reed Col-lege, http://academic.reed.edu/economics/parker/ExpBook95.pdf, accessed 2014-10-23.

[23] K. Petersen, R. Feldt, S. Mujtaba, and M. Matt-son. Systematic mapping studies in softwareengineering. In International Conference on Eval-uation and Assessment in Software Engineering,pages 68–77. ACM, 2008.

[24] K. Petersen, S. Vakkalanka, and L. Kuzniarz.Guidelines for conducting systematic mappingstudies in software engineering: An update. Inf.Softw. Technol., 64:1–18, August 2015.

[25] P. Runeson, M. Höst, A. Rainer, and B. Reg-nell. Case Study Research in Software Engineer-ing: Guidelines and Examples. John Wiley &Sons, 2012.

[26] C. Wohlin. Empirical software engineering:Teaching methods and conducting studies. InProceedings of the International Workshop on Em-pirical Software Engineering Issues: Critical As-sessment and Future Directions, volume 4336 ofLNCS, pages 135–142. Springer, 2007.

[27] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson,B. Regnell, and A. Wesslén. Experimentation insoftware engineering. Springer Science & Busi-ness Media, 2012.

[28] H. Zhang, M. A. Babar, and P. Tell. Identifyingrelevant studies in software engineering. Infor-mation and Software Technology, 53(6):625–637,2011.



Documents

Using Mini-Projects to Teach Empirical Software Engineeringceur-ws.org/Vol-2358/paper-07.pdf · Using Mini-Projects to Teach Empirical Software Engineering Michael Felderer 1 and