15
REALL: Rubric for the Evaluation of Apps in Language Learning Elena Martín Monje (UNED), Jorge Arús Hita (UCM), Pilar Rodríguez Arancón (UNED) & Cristina Calle Martínez (UCM) ATLAS Research Group Logroño, 2-3 May 2013

Reall: rubric for the evaluation of apps in language learning

Embed Size (px)

Citation preview

Page 1: Reall: rubric for the evaluation of apps in language learning

REALL: Rubric for the Evaluation of Apps in Language Learning

Elena Martín Monje (UNED), Jorge Arús Hita (UCM),

Pilar Rodríguez Arancón (UNED) & Cristina Calle Martínez (UCM)

ATLAS Research Group

Logroño, 2-3 May 2013

Page 2: Reall: rubric for the evaluation of apps in language learning

Table of contents

Introduction

Pedagogic assessment of mobile learning apps for EFL

◦ Stage 1: Analysis and categorization of EFL apps available

◦ Stage 2: Design of an evaluation rubric for the pedagogic

assessment of EFL apps

◦ Stage 3: Rubric for the evaluation of apps in language

learning (REALL)

Conclusion

References

Page 3: Reall: rubric for the evaluation of apps in language learning

Introduction

Mobile learning

Mobile-Assisted Language Learning (MALL)

SO-CALL-ME project

Stages in this research study:

Stage 1:

Analysis and categorization of EFL apps available

Stage 2:

Design of rubric for pedagogic

assessment of EFL apps

Stage 3:

Creation of REALL (Rubric

for the Evaluation of

Apps in Language Learning

Page 4: Reall: rubric for the evaluation of apps in language learning

Previous rubrics & categorizations

Taken from http://langwitches.org/blog/2012/03/31/ipad-apps-and-blooms-taxonomy/

Page 5: Reall: rubric for the evaluation of apps in language learning

Taken from Tony Vincent’s learninginhand.com

Page 6: Reall: rubric for the evaluation of apps in language learning

Stage 1: Apps available

3 criteria: apps’ cognitive value, similarity/

complementarity with SO-CALL-ME

67 apps assessed

Frequent technical problems downloading/ starting apps

Vast majority for Apple devices

Most expensive: dictionaries, grammar

Many for 2-3€ or with initial free sample pack

Categories:

◦ Games

◦ App versions of printed materials (e.g. coursebooks)

◦ Apps for pronunciation/ grammar/vocabulary

Page 7: Reall: rubric for the evaluation of apps in language learning

Stage 2: Rubric for pedagogic assessment

Quality guide for digital learning objects (Fernández-Pampillón et al., 2011)

• Cognitive value and pedagogic coherence (1)

• Content quality (2)

• Capacity to generate learning (3)

• Interactivity and adaptability (4)

• Motivation (5)

Pedagogical

criteria

•Format and layout (6)

•Usability (7)

•Accessibility (8)

•Visibility (9)

•Compatibility (10)

Technical

criteria

Page 8: Reall: rubric for the evaluation of apps in language learning

Rubric with descriptors and scale 1-5.

E.g. Criterion 3 in rubric for pedagogic assessment:

The 5 top EFL apps of Stage 1 assessed with this rubric:

1 2 3 4 5

3. Capacity to

generate

learning

Contents do not

help to achieve

learning goals

or autonomous

learning

Contents help

autonomous

learning but not

clearly the

achievement of

the initial

learning goals

Contents help

to achieve the

learning goals

but neither

autonomous

learning nor

relating old

knowledge to

new knowledge

Contents help

to achieve the

learning goals

but not

autonomous

learning OR not

relating old

knowledge to

new knowledge

Contents help

to achieve the

learning goals,

autonomous

learning and

relating old

knowledge to

new knowledge

Page 9: Reall: rubric for the evaluation of apps in language learning
Page 10: Reall: rubric for the evaluation of apps in language learning

Stage 3: Rubric for linguistic assessment

Common European Framework of Reference for Languages (Council of Europe, 2001)

Focus on levels A2-B2

Initial rubric for oral

comprehension

Categories: Level

Type of texts

Topics

Delivery

E.g. Delivery in rubric for linguistic assessment:

1 2 3 4 5 N/A

Delivery

Language

difficulty, clarity

and speed mix

different levels.

If adaptive,

delivery is

clearly not well

adapted

Language difficulty,

clarity and speed

rarely belong to the

same level. If

adaptive, delivery

rarely corresponds

to the right level

Language difficulty,

clarity and speed

tend to belong to

the same level. If

adaptive, delivery

more often than not

corresponds to the

right level

Language difficulty,

clarity and speed

usually belong to

the same level. If

adaptive, delivery

usually

corresponds to the

right level

Language

difficulty, clarity

and speed belong

to the same level.

If adaptive,

delivery

corresponds to

the right level

Page 11: Reall: rubric for the evaluation of apps in language learning

Parallel process in stage 2 & stage 3: 1 rubric, 2 evaluators

• Stage 2: 91/100

• Stage 3: 29/40

• Stage 2: 89/100

• Stage 3: 37/40

• Stage 2: 82/100

• Stage 3: 13/40

• Stage 2: 81/100

• Stage 3: 19/40

• Stage 2: 64/100

• Stage 3: 14/40

Page 12: Reall: rubric for the evaluation of apps in language learning

The most comprehensive apps assessed were Speakingpal and Learn English Elementary Podcasts

Page 13: Reall: rubric for the evaluation of apps in language learning

Conclusions

Pedagogic & technical quality ≠ linguistic value &

adequacy for EFL teaching & learning

Importance of all three dimensions:

◦ Pedagogical

◦ Technical

◦ Linguistic

Piloting of stage 2 & stage 3 rubrics enables a future

study on a larger scale

Future theoretical framework for successful,

pedagogically & linguistically sound EFL apps

Page 14: Reall: rubric for the evaluation of apps in language learning

References

Arús-Hita, Jorge; Pilar Rodríguez-Arancón and Cristina Calle-Martínez (in press) “A pedagogic assessment of mobile learning applications”. In Proceedings of ICDE 2013, Mobilizing Distance Education, UNED, Madrid.

Council of Europe (2001). Common European Framework of Reference for Languages. Cambridge: Cambridge University Press.

Fernández-Pampillón Cesteros, Ana; Elena Domínguez Romero and Isabel de Armas Ranero (2011) Herramienta para la revisión de la Calidad de Objetos de Aprendizaje Universitarios (COdA): guía del usuario. v.1.1. Madrid: e-prints Complutense. Retrieved from http://eprints.ucm.es/12533/ (accessed 18/04/2013).

Page 15: Reall: rubric for the evaluation of apps in language learning

THANK YOU

[email protected]