Embed Size (px)
Citation preview
Recommendations on Formative Assessment and Feedback Practices for stronger engagement in MOOCs
ByNikolaos Floratos, [email protected], Teresa Guasch,
[email protected] and Anna Espasa, [email protected] Open University of Catalonia
GO-GN Network
Around 90% the drop rate in MOOCs (Clow, 2013; Lewin, 2013)
Some interesting facts about MOOCs• 8.2 Million participants in Coursera• 700 plus courses by more than 100 universities
in Coursera• Strong MOOC activity around the globe• The numbers are increasing continually!!!!but also drop rates are increasing
Without this• MOOCs do not create impact• Open Education may fail
With this• MOOCs will make a real difference in
education• Open Education will transcend
Student Engagement (Adamopoulos, 2013) depends on non-didactic and didactic factors
Student profiles
Instructors’profiles
Institution Reputations
demographics
Course structure
Training content
Self-paced or not
workload
Course duration
Course topic
Type of examsAssessment
& Feedback types
Interaction with students & instructors
Two contradictory points
The type of assessment and feedback methods as well as interaction with students and instructors is one of the main elements for student engagementHowever, very limited research so far on the proper assessment as well as feedback practices to be applied specifically in MOOCs for enhancing student engagement
Therefore the scope of this study1. to formulate a framework of
requirements/conditions for feedback and formative assessment practices that support student engagement in MOOCs (on literature review)
2. to suggest related recommendations on formative assessment and feedback practices
Literature review Findings
• no research evidence yet on specific feedback and formative assessment practices directly applicable in MOOCs for advancing Student Engagement
• Such research now applicable only in traditional learning environments (mainly in f2f learning)– Nicol and Macfarlane-Dick (2006): 7 principles on feedback
and formative assessment practices for student engagement
– Gibbs and Simpson, 2004: Assessment tasks should satisfy 3 specific conditions and feedback 6 conditions respectively
Student Engagement in MOOCS?
(Hew 2015) defines student engagement in MOOCs as the level of a student’s engagement in a learning activity
How engagement is drivenHew 2015, Deci and Ryan, 1991; Deci and Ryan, 2000; Hardre and Reeve, 2003;(Self Determination Theory SDT)• Need of Autonomy (the need for students to
sense they are non dependent on other peoples' actions
• Need of relatedness (the need for students to connect with other people)
• Need of competence (the need for students to master specific knowledge)
Based on literature review (Nicol and Macfarlane-Dick 2006, Gibbs and Simpson 2004, Hew 2015, Deci and Ryan, 1991; Deci and Ryan, 2000; Hardre and Reeve, 2003;
17 Requirements/conditions for formative assessment and feedback practices to support student engagement in MOOC
17 Requirements/conditions1. Requirement 1: Autonomy e.g. a MOOC student should be
able to assess their own work, assignments and progress2. Requirement 2: Relatedness e.g.a student should be able
to assess the work and assignment of other peers3. Requirement 3: Competence e.g. Assessment to generate
feedback on performance and accelerate learning4. Requirement 4: Students should know in advance what is
a good performance and based on that to improve further5. Requirement 5: Student should know in advance the steps
needed for reaching good performance
17 Requirements/conditions cont.6. Requirement 6: Formative assessment and feedback
practices should help students clarify what good performance is and how much different this is with respect to their current performance
7. Requirement 7: Formative assessment and feedback practices should encourage dialogue around learning between peers and teachers
8. Requirement 8: Formative assessment and feedback practices should advance self-esteem
9. Requirement 9: Formative assessment and feedback practices should create positive motivational beliefs and self-esteem
17 Requirements/conditions cont.10. Requirement 10: Formative assessment and
feedback practices should provide information to teachers for improving their teaching approach
11. Requirement 11: Assessment tasks should be sufficient enough in order to require students to dedicate appropriate study time for addressing them
12. Requirement 12: Assessment tasks should orientate students to allocate appropriate amounts of time and effort to the most important aspects of the course
13. Requirement 13: Assessment tasks should engage students with related productive learning activities
17 Requirements/conditions cont.14. Requirement 14: Feedback should be provided
in sufficient detail and frequent enough15. Requirement 15: Feedback should focus on
students’ performance on their learning16.Requirement 16: Feedback should be delivered
in time to students so that it still matters and it is well received Requirement
17. Requirement 17: Feedback should be aligned with the purpose of the related assignment and advance future learning and use by the student
Approach on formulating related recommendations
• As basis reviews from the special review platform for MOOCs, coursetalk www.coursetalk.com
• Reviews considered only from specific MOOCs with all the following criteria– Offered for free– Offered by Universities– Top-rated– Received more than 100 reviews
• 7 MOOCs have been identified with total 4040 reviews– An introduction to Interactive Programming in Python – offered by Coursera (1863 reviews) – A beginner's Guide to Irrational Behavior offered by Coursera (375 Reviews)– Epidemics - the Dynamics of Infectious Diseases offered by Coursera (315 Reviews)– Design: Creation of Artifacts in Society offered by Coursera (204 Reviews)– Modern and Contemporary American Poetry offered by Coursera (170 Reviews) – An introduction to Operations Management offered by Coursera (104 Reviews)– Think101x: The Science of Everyday Thinking offered by edX (1037 Reviews)
• 5 steps approach
Our 5 Steps Approach
1. Step 1: Filtering and Selection of reviews only related to feedback and formative assessment (418 reviews)
2. Step 2: Additional Filtering for selecting just those reviews from above that contribute with some input to recommendations to be formulated (237 reviews)
3. Step 3: Examine the 237 reviews and formulate 14 recommendations that respect our identified requirements
4. Step 4: Examining again the 237 reviews and checking– how many times each review contributed as input to a formulated
recommendation– Whether there were any new recommendations that may have been missedN.B. Validation: Two additional judges have applied steps 3 and 4 in a sample of reviews in order to ascertain agreement in findings
5. Step 5: Significance Ranking - Based on the number of reviews contributed as input to each of the formulated recommendations, the significance of each recommendation is derived based on the weighted average method
No limitation on number of attempts in quizzes
Feedback to be provided in time before the next assignment
Peer assessment always by the same students (no random allocation)
Students to be able to modify their own related assignment work even after its submission based on feedback received).
Peer assessment can be supplemented with forum discussions for interaction and offer a communication channel for clarifications.
Interaction by human experts say professors’ assistants that intervene and provide comments and views in discussion forums or even in students’ work
Immediate feedback mainly from quizzes is appreciated by students since it confirms their understanding about what they have learnt. The appreciation is even higher in cases that some further feedback on the given answer or on the correct answer is provided. (18 reviews supported this recommendation/finding with 8 as weighted average)
Peer assessment in MOOCs does not always deliver accurate results and in many cases it creates frustrations or negative feelings since it doesn’t provide any feedback on the work assessed or there are concerns on the competencies of the peer-assessor. (26 reviews supported this recommendation/finding with 9 as weighted average)
Peer assessment was more welcome in those MOOCs that there was no mark given but also or alternatively feedback on the quality of the related assignment. (23 reviews supported this recommendation/finding with 10 as weighted average)
Assessment models, such as self assessments (students judge their own work) and peer-assessments (students judge the work of their peers) can be relatively complex in MOOCs and a comprehensive rubric should be provided in all MOOCS that involve peer assessments (29 reviews supported this recommendation/finding with 13 as weighted average)
Discussion forums are appreciated and are a very good way of students to receive support, sympathy, formative feedback and clarification as well as share ideas on their work as long as there is an effective mechanism on managing and accessing the discussion threads. (36 reviews supported this recommendation/finding with 18 as weighted average)
A well structured course syllabus with information on what assignments are required per week/module by each participant, by when as well as its related training content was appreciated (35 reviews supported this recommendation/finding with 21 as weighted average)
Although MOOCs can automatically assess quizzes, self and peer assessments must deal with more comprehensive assignments. However, self- and peer assessment should be formative as well, i.e. specifically intended to generate feedback on performance and improve and accelerate learning rather than just providing a mark. (48 reviews supported this recommendation with 30 as weighted average)
Assignments can be based on practical problems with clear expected output that give the sense of completeness to the students and are not that easy to address but challenging and interesting enough (65 reviews supported this recommendation/finding with 42 as weighted average)
Let’s all move from MOOC to MOOL!
Thank you for your attention
Nikolaos FloratosPhD ResearcherOUC University
GO-GN Memberwww.NikolaosFloratos.com
BibliographyDeci, E. L., & Ryan, R. M. (2000). The" what" and" why" of goal pursuits: Human needs and the self-determination of behavior. Psychological inquiry, 11(4), 227-268. http://goo.gl/Jf7wzV Deci E.L. & Ryan R. M., (1991) “A motivational approach to self: Integration in personality,” in Nebraska symposium on motivation: Perspectives on motivation, R. Dienstbier, Ed., Lincoln, NE: University of Nebraska Press, vol. 38, pp. 237-288Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of educational research, 74(1), 59-109 http://goo.gl/JWf61L Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports students’ learning. Learning and teaching in higher education, 1(1), 3-31 Guasch, T., Espasa, A., Alvarez, I.M. & Kirschner, P.A. (2013). Effects of Teacher and Peer Feedback on Collaborative Writing in an Online Learning Environment. Distance education, 34 (3), 324-338, DOI: 10.1080/01587919.2013.835772Helme, S., & Clarke, D. J. (1998). We really put our minds to it: Cognitive engagement in the mathematics classroom. Teaching mathematics in new times, Brisbane, Qld: Mathematics Education Research Group of Australasia, pp. 250-257. Hew, K.F. (2015). Towards a Model of Engaging Online Students: Lessons from MOOCs and Four Policy Documents. International Journal of Information and Education Technology, 5 (6), 425-431Nicol, N. & Macfarlane, D. (2006) Formative Assessment and self regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31 (2), 199-218 http://goo.gl/aW1Ybs Reeve, J. (2012). A self-determination theory perspective on student engagement. In Handbook of research on student engagement (pp. 149-172). Springer US, http://goo.gl/OyCOKm Skinner, E. A., Kindermann, T. A., Connell, J. P., & Wellborn, J. G. (2009). Engagement and disaffection as organizational constructs in the dynamics of motivational development. Handbook of motivation at school, 223-245. http://goo.gl/bnTOI0 License and CitationThis work is licensed under the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0/. Please cite this work as: Floratos, N., Guasch, T., Espasa, A., (2015). Recommendations on Formative Assessment and Feedback Practices for stronger engagement in MOOCs. In Proceedings of Open Education Global 2015: Innovation and Entrepreneurship.
