Analytics are everywhere

Innovate • Design • Enhance

Analytics are everywhere


“Analytics for Action – enabling in-presentation evidence-based change on Open University modules”

16 Octubre de 2017

• Rafael Hidalgo - Senior TEL Designer

LTI – The Open University


What is Analytics for Action (A4A)?

• A4A is a project to enable evidence-based change in presentation –

Active Presentation

• It started as part of the Learning and Teaching Centre’s Analytics Project

(2014-2016), encompassing:

• Real-time Student Feedback, Student progression reporting, SAS

Visual Analytics, Ethics, Predictive modelling (OU Analyse) and

Analytics 4 Action

• TEL Design provided a natural long-term home for Analytics for Action .

• A4A is run by TEL LTI since 2016/2017- Data is business as usual


“Using the data as prompts, module teams can initiate a range of response actions, including pedagogic change and student support”

- Bart Rienties, IET, The Open University


Analytics for Action Framework


The Active Presentation Toolkit

• Based on the A4A

framework

• Available to all staff

in the Open

University intranet

• Step-by-step

instructions for

running an active

presentation

https://learn3.open.ac.uk/course/view.php?id=300833


The Active Presentation Toolkit

• Staged approach


What support is available?

For all modules

• Toolkit provides for a DIY approach

• Training enables staff to get started with this,

• Surgery support at Learning Systems drop-ins

• Mailbox support: [email protected]

For selected modules:

• Data support meetings: 3 per presentation


Which modules receive full support?

• Agreed with Faculties

• Modules identified as underperforming by the AQR – Z scores

• High student population

• Pedagogic challenges and/or innovations

• Qualifications needs

• Faculty strategic interest


Training

• 2 hours hands-on training sessions run by Senior TEL Designers, using

real and current data.

• Priority places for MTC’s and CM’s of modules selected for formal support

• Main focus: Active Presentation Toolkit and Student Progression

Dashboards

• Small groups to allow enough discussion and specific questions


Data support meetings

• Senior TEL Designers will support the relevant faculty modules

• Academics provide in-depth knowledge of the module,

• STELDs provide wider view as we are exposed to other data,

• Together we agree on the issues to be investigated,

• STELDs suggest actions and help in the implementation (this may

involve other units)

• Comprehensive reports on discussions and agreed actions,

• Regular updates to the faculty stakeholders about progress


What sort of data do we look at support meetings?

Student progression dashboards

• Student profile

• Concurrency

• Assessment submission rates

• Withdrawer profiles

• Online engagement

• Online tools and resource

usage

• Retention

• Pass and retention rates

https://sas-va.open.ac.uk/SASVisualAnalyticsHub/


Student profiles and concurrency

• 267 students registered at FLP25

• Roughly half registered for Q72 and a quarter for Open Degree

• 66% male, most common age group is 30-39 (37%)

• 16.5% have declared a disability

• 73.8% were studying at least another module, mostly T313, with 38 students studying 120 credits

• 79% are employed part or full time

T219

• 187 students registered at FLP25

• Q72 is the most popular qualification (30%)

• 64% male, most common age group 30-39 (37%)

• 11% have declared a disability

• 55% were studying at least one other module with 20 students studying 120 credits

• 87% are employed part or full time

T319


Assessment Submission Rates

93.1

87.0

96.5

83.6

94.9

62.155.2 63.9

65.1 59.6

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

S104 16J SDK100 U116 S111 S10415J

% of TMA Submissions - Level 1 Science

TMA01 TMA02 TMA03 TMA04 TMA05 TMA06


Assessment Submission Rates

87.3

70.9

59.357.1

87.6

76

60.3

0

10

20

30

40

50

60

70

80

90

100

TMA01 TMA02 TMA03 Project

% of TMA Submissions - T219 – by year

13J 14J 15J 16J


Retention

80

82

84

86

88

90

92

94

17 18 19 20 21 22 23 24 25

% Retention

T219 T213 DST206 AVE


Online engagement

VLE use for T219 in 16J


Use of online resources

Use of Block 1 Forum in T319


Completion and Pass rates


Data insight: Conversations and actions

• “Significant numbers of students are studying other modules concurrently. The module team

don’t advise on multiple module study.”

Student Support Teams need to be made aware of significant number of students taking

120 points at once. A faculty discussion is also needed at qualification level.

• Are extensions any good?

• Is the gap between TMA02 and Project too long?

TMA02 weight and timing needs to be assessed

• More support from the tutor at the start of collaborative work

Module Team to consider an intro collaborative activity

• Tutors should be trained in handling the forums, are they?


Data Support Meetings Reports


What can we do about it?

• Based on our findings we could decide:

• A change to the materials

• Some additional support to everyone

• Some targeted support

• Improvements upstream to get the right people onto the module


What can

we do

about it?

http://article.iet.open.ac.uk/Q/QE%20report%20series/2015-QE-

learning-analytics.pdf

http://article.iet.open.ac.uk/Q/QE report series/2015-QE-learning-analytics.pdf


Success examples

KJY113 16B: Improved pass rates

• Registered students dropped 12-13 %

from start to 25FLP. Pass rates under

faculty average for 3 years

• Additional information via dedicated

website and A5 postcard

• AL’s contacted students at risk

• Pass rate up 12%. Completion rate up

10%. TMA submissions improved. Drop

between start and FLP25 was only 3%


Success examples

S350 16J: Managing extensions

• Concurrency levels very high: 73.4%

• 75% of students were employed

• Students unexpectedly taking Sxx390

• AL’s were given list of high workload

students and their TMA dates for

concurrent modules, to manage

extensions.

• No historical data to compare but early

intervention by AL’s was enabled by

data support meeting


Success examples

T192 16J: design to evaluation

• In the design, Module Team addressed

a key mathematical concept –

“rearranging equations”-

• TEL Design helped to set up an

adaptive learning package of iCMAs

with support built-in via the website and

module tutors.

• Monitoring the data allowed to provide

the module team chair with the

information required to decide whether

to take action on these individual

quizzes.

-10

0

10

20

30

40

50

60

70

80

90

100

0 2 4 6 8 10 12 14 16 18 20

Number of students with scores under 40% and 50%

Scores < 40 Scores < 50


Analytics for action: Data is business as usual

In 2016/17

• 29 OU modules fully supported – reaching almost 25,000 students

• Over 100 staff trained in 23 training sessions

• 75 data support meetings held

• Coverage expected to grow in 2017/18


Analytics for action: The future

For 2017/18: growing numbers and quality

• Up to 49 OU modules fully supported

• Improved tools and systems

• More than 120 staff to be trained and over 100 data support meetings to be held

• Increased scholarship

• More/better data from the VLE?


Analytics for action: What are they saying? About Data Training Tool sessions:

“The initial question I had was answered and I found out other useful things also”

“The trainers were able to support individual learners with their unique issues”

“It was informative and with help easy to navigate”

“Hands on appropriate support and guidance using real live data”

About Data Support Meetings:

“Being able to go “off grid” and actually – finally! – get some insights and answers into presentation questions we have had for ages”

“Thank you that gives us something to think about for the future presentations”

“I was updating VB on the great work that has been happening with T219/T319”

“We were enjoying ourselves so much in the world of data that we almost called up for a pizza to keep going!”


The Open UniversityLTI TEL Design teamLevel 7 Perry B

[email protected]

@TEL_Design_OU

Get in touch!

mailto:[email protected]

Documents

Analytics are everywhere