Data Analytics.01. Data selection and capture

Data Analytics process in Learning and Academic

Analytics projects

Day 1: Data selection and capture

Alex Rayón Jerezalex.rayon@deusto.es

DeustoTech Learning – Deusto Institute of Technology – University of DeustoAvda. Universidades 24, 48007 Bilbao, Spain

www.deusto.es

Objectives

How to tackle an AA/LA project

1. Objectives: what do I want to improve?2. Data: automated processes for data discovery

and later processing3. Integration, not substitution4. Technology5. KPIs: define and test

Table of contents

● ETL approach● Data analytics cycle● Architecture principles● Requirements● Components● Data process

○ Questions○ Data model○ Data sources○ Use cases

Table of contents

● ETL approach● Data analytics cycle● Architecture principles● Requirements● Components● Data process

ETL approachDefinition and characteristics

● An ETL tool is a tool that○ Extracts data from various data sources (usually

legacy data)○ Transforms data

■ from → being optimized for transaction■ to → being optimized for reporting and analysis■ synchronizes the data coming from different

databases■ data cleanses to remove errors

○ Loads data into a data warehouse

ETL approachWhy do I need it?

● ETL tools save time and money when developing a data warehouse by removing the need for hand-coding

● It is very difficult for database administrators to connect between different brands of databases without using an external tool

● In the event that databases are altered or new databases need to be integrated, a lot of hand-coded work needs to be completely redone

ETL approachKettle

Project Kettle

Powerful Extraction, Transformation and Loading (ETL) capabilities using an

innovative, metadata-driven approach

ETL approachKettle (II)

● It uses an innovative meta-driven approach● It has a very easy-to-use GUI● Strong community of 13,500 registered

users● It uses a stand-alone Java engine that

process the tasks for moving data between many different databases and files

ETL approachKettle (III)

ETL approachKettle (IV)

Source: http://download.101com.com/tdwi/research_report/2003ETLReport.pdf

ETL approachKettle (V)

Source: Pentaho Corporation

ETL approachKettle (VI)

● Datawarehouse and datamart loads● Data integration● Data cleansing● Data migration● Data export● etc.

ETL approachTransformations

● String and Date Manipulation● Data Validation / Business Rules● Lookup / Join● Calculation, Statistics● Cryptography● Decisions, Flow control● Scripting● etc.

ETL approachWhat is good for?

● Mirroring data from master to slave● Syncing two data sources● Processing data retrieved from multiple

sources and pushed to multiple destinations

● Loading data to RDBMS● Datamart / Datawarehouse○ Dimension lookup/update step

● Graphical manipulation of data

Table of contents

● ETL approach● Data analytics cycle● Architecture principles● Requirements● Components● Data process

Data Analytics cycleChallenges

● Data is everywhere● Data is inconsistent

○ Records are different in each system● Performance issues

○ Running queries to summarize data for stipulated long period takes operating system for task

○ Brings the OS on max load● Data is never all in Data Warehouse

○ Excel sheet, acquisition, new application

Data Analytics cycleChallenges (II)

● Data is incomplete● Certain types of usage data are not logged● Data are not aggregated following a

didactical perspective● Users are afraid that they could draw

unsound inferences from some of the data

[Mazza2012]

Data Analytics cycleAcademic Analytics Model

1) Capture

2) Report5) Refine

4) Act 3) Predict

Academic Analytics [CampbellOblinger2007]

Data Analytics cycleLearning Analytics Model

1) Select

2) Capture

3) Aggregate

4) Process

5) Visualize

On the design of collective applications[DronAnderson2009]

Data Analytics cycleLearning Analytics Model (II)

On the design of collective applications[DronAnderson2009]

1) Select

2) Capture

3) Aggregate

4) Process

5) Visualize

Day 1

Day 2

Day 3

Day 4

Data Analytics cycleLearning Analytics Model (III)

● As [Clow2012] states, it is necessary to close the feedback loop through appropriate interventions unmistakable

● It also draws on the wider educational literature, seeking to place learning analytics on an established theoretical base, and develops a number of insights for learning analytics practice

Data Analytics cycleLearning Analytics Model (IV)

Data Analytics cycleLearning Analytics Model (III)

Table of contents

● ETL approach● Data analytics cycle● Architecture principles● Requirements● Components● Data process

Architecture principlesA model for adoption, use and improvement of analytics

A framework of characteristics for AnalyticsAdam Cooper, 2012 [Cooper2012]

Architecture principlesDevelopment of common language for data exchange

The IEEE defines interoperability to be:

“The ability of two or more systems or components to exchange information and

to use the information that has been exchanged”

Architecture principlesDevelopment of common language for data exchange (II)

Architecture principlesDevelopment of common language for data exchange (III)

● The most difficult challenges with achieving interoperability are typically found in establishing common meanings to the data

● Sometimes this is a matter of technical precision○ But culture – regional, sector-specific, and

institutional – and habitual practices also affect meaning

Architecture principlesDevelopment of common language for data exchange (IV)

● Potential benefits○ Efficiency and timeliness

■ No need for a persona to intervene to re-enter, re-format or transform data

○ Independence■ Resilience

○ Adaptability■ Faster, cheaper and less disruptive to change

○ Innovation and market growth

■ Interoperability combined with modularity makes

it easier to build IT systems that are better

matched to local culture without needing to create and maintain numerous whole systems

Architecture principlesDevelopment of common language for data exchange (V)

● Potential benefits○ Durability of data

■ Structures and formats change over time■ The changes are rarely properly documented

○ Aggregation

■ Data joining might be supported by a common set

of definitions around course structure, combined with a unified identification scheme

○ Sharing■ Specially when there are multiple parties involved

Architecture principlesDevelopment of common language for data exchange (VI)

[LACE2013]

Architecture principlesDevelopment of common language for data exchange (VII)

[LACE2013]

In our case?

Architecture principlesDevelopment of common language for data exchange (VIII)

[LACE2013]

In our case?

Table of contents

● ETL approach● Data analytics cycle● Architecture principles● Requirements● Components● Data process

Requirements

● Usability: prepare an understandable user interface (UI), appropriate methods for data visualization, and guide the user through the analytics process.

● Usefulness: provide relevant, meaningful indicators that help teachers to gain insight in the learning behavior of their students and support them in reflecting on their teaching.

● Interoperability: ensure compatibility for any kind of VLE by allowing for integration of different data sources.

[Dyckhoff2010]

Requirements (II)

● Extensibility: allow for incremental extension of analytics functionality after the system has been deployed without rewriting code.

● Reusability: target for a building-block approach to make sure that re-using simpler ones can implement more complex functions.

● Real-time operation: make sure that the toolkit can return answers within microseconds to allow for an exploratory user experience

● Data Privacy: preserve confidential user information and protect the identities of the users at all times

[Dyckhoff2010]

Table of contents

● ETL approach● Data analytics cycle● Architecture principles● Requirements● Components● Data process

Components

● Process○ A systematic process of educational data analysis

● Model○ The definition of a suitable model to represent the

knowledge domain● Tool/platform

○ The design and implementation of a monitoring and presentation tool based on the Process and Model

[Mazza2012]

Table of contents

● ETL approach● Data analytics cycle● Architecture principles● Requirements● Data process

Data processIntroduction

“Measurement, collection, analysis and reporting of data about learners and

their contexts, for purposes of understanding and optimising learning

and the environments in which it occurs”

First international conference on Learning Analytics and Knowledge, Alberta, 2011 [LAK2011]

Data processIntroduction (II)

However, the challenge is to determine which data are of interest

We are now in an era where gaining access to data is not the problem;

the challenge lies in determining which data are significant and why

Data processIntroduction (III)

“The basic question is not what can we measure? The basic

question is what does a good

education look like? Big questions”

Data processIntroduction (IV)

“More data does not mean more knowledge” [Jenkins2013]

Searching for the evidence in a mass of data requires knowing what kind of evidence is

neededKnowledge of the domain and understanding

and interpretation of the patterns we see

Data processIntroduction (V)

Data processIntroduction (VI)

Source: http://www.learningfrontiers.eu/?q=story/will-analytics-transform-education

Data processIntroduction (VII)

A brief comparison of the two fields (George Siemens and Ryan Baker [SiemensBaker2012])

Data processIntroduction (VIII)

First of all, education is a highly collaborative space and it represents a social good. Keeping a valuable secret that might help students succeed is antithetical to the nature of education. Second, education is a complex

ecosystem of people, processes, policies, content, etc. I would have strong doubts about anyone who claimed to have a formula that worked for a wide variety of institutions.

Mike Sharkey, 2014

Data processQuestions

Source: http://www.slideshare.net/sbs/learning-analytics-uts-2013

Data processQuestions (II)

The question depends on who is making it ;)

Data processQuestions (III)

Horizon Report 2012 [HR2012]

Data processQuestions (IV)

1) Adaptive testing, tracking and reporting

● Progress summary, daily activity report, class goals report, progress report, student activity report, student focus report, etc [Khan2012]

● By using various analytics tools, students can review their learning progress and teachers are also supported in how to personalise learning for students in need for more help in specific areas

Data processQuestions (V)

2) Analytics tools for early alert, intervention and collaboration

● Integrating their data collected from a variety of information management systems○ Allowing educators to assess the risk, initiate early

interventions and support collaborative learning

Data processQuestions (VI)

2) Analytics tools for early alert, intervention and collaboration

● For example, the Signals project at Purdue University utilizes the data collected from student information systems, learning management systems, and the grade book for a specific course to track students’ performances and identify at-risk students in real time

Data processQuestions (VII)

2) Analytics tools for early alert, intervention and collaboration

● The LOCO-Analyst provides teachers with charts, graphs, and other data representations that help them see how their students are performing and how students interact with one another in web-based learning environments to help the teacher determine how to engage their students online

Data processQuestions (VIII)

2) Analytics tools for early alert, intervention and collaboration

● Social Networks Adapting Pedagogical Practice (SNAPP), a network visualization tool developed by researchers at the University of Wollongong, can analyse students’ interactions in a forum and display it in a visualised diagram which help teachers to identify the key connections and disconnected students and support collaborative learning in a web-based learning environment

Data processQuestions (IX)

3) Analytics projects for institutional efficiency and effectiveness

● There are a number of institutional analytics initiatives which enable institutions to improve the effectiveness of operations, including admission management and drop-out prevention, resource management, financial planning, etc○ Student Experience Traffic Lighting (SETL)

○ The Enhancing Student Centred Administration for Placement Experience (ESCAPES)

Data processQuestions (X)

Learning Analytics are not neutral

Data processQuestions (XI)

“Accounting tools… do not simply aid the measurement of economic

activity, they shape the reality they measure”

[GayPryke2002]

Data processQuestions (XII)

Fuente: http://mfeldstein.com/harvard-mit-learn-university-phoenix-analytics/

Data processQuestions (XIII)

● The Harvard and MIT data ignores student goals or any information giving a clue on whether students desired to complete the course, get a good grade, get a certificate, or just sample some material

● Without this information, the actual aggregate behavior is missing context ○ We don’t know if a certain student intended to just

audit a course, sample it, or attempt to complete it.

○ We don’t know if students started the course intended to complete but became frustrated

Data processQuestions (XIV)

● The value of learner behavior patterns, which can only be learned by viewing data patterns over time

● If you want to “share best practices to improve teaching and learning”, then you need data organized around the learner○ With transactions captured over time – not just in

aggregate

○ What we have now is an honest start, but a very limited data set

Data processData model

So, which is our data model to answer to

our questions?

Data processData model (II)

The data model, or the concept map, describes the concepts and their

relationships used by the organization in its daily work, expressed in its own

language

It enables the whole organization to participate in the maintenance of it

Data processData model (III)

Source: http://www.economist.com/news/finance-and-economics/21578041-containers-have-been-more-important-globalisation-freer-trade-humble

Source: http://www.economist.com/blogs/economist-explains/2013/05/economist-explains-14

Data processData model (IV)

The best approach that we have found for this task is constituted by the theory of eLearning functions

Reinmann [Reinmann2006]

Data processData model (V)

[Reinmann2006]

Data processData model (VI)

Example

[Mazza2012]

Data processData model (VII)

Example

This model answers the monitoring questions:

● Which way of eLearning enables to reach the given objectives?

● By which means (functions, tools) does the LMS enable these ways of learning?

● How is the use of these means traced in the log files (activity log codes)?

[Mazza2012]

Data processData model (VIII)

[Mazza2012]

Data processData sources

Today we have so much data that come in an unstructured or semi-structured form that may nonetheless be of value in understanding more about our

learners

Data processData sources (II)

“Learning is a complex social activity” [Siemens2012]

Lots of dataLots of tools

Humans to make sense

Data processData sources (III)

Traditional data sources:● Student data: demographics,

qualification aim, modules taken, results, etc.

● Student feedback data: end of module survey and others

● Student activity data: delivery data, completion, pass rates, etc.

Data processData sources (IV)

● The world of technology has changed [Eaton2012]○ 80% of the world’s information is unstructured

○ Unstructured data are growing at 15 times the rate of structured information

○ Raw computational power is growing at such an

enormous rate that we almost have a supercomputer in our hands

○ Access to information is available to all

Data processData sources (V)

Source: http://www.bigdata-startups.com/BigData-startup/understanding-sources-big-data-infographic/

Data processData sources (VI)

● RDBMS (SQL Server, DB2, Oracle, MySQL, PostgreSQL, Sybase IQ, etc.)

● NoSQL Data: HBase, Cassandra, MongoDB● OLAP (Mondrian, Palo, XML/A)● Web (REST, SOAP, XML, JSON)● Files (CSV, Fixed, Excel, etc.)● ERP (SAP, Salesforce, OpenERP)● Hadoop Data: HDFS, Hive● Web Data: Twitter, Facebook, Log Files, Web Logs● Others: LDAP/Active Directory, Google Analytics,

etc.

Data processUse cases

1) Student data: XML

Data processUse cases

1) Student data: XML

Data processUse cases

1) Student data: XML

Data processUse cases (II)

2) Moodle: MySQL database

mdl_forum

- id- course- name

mdl_user

- id- username- firstname- lastname

mdl_forum_discussions- id- name- userid- timemodified- usermodified

mdl_forum_posts- id- userid- discussion- message- modified- created

Data processUse cases (III)

3) MediaWiki: MySQL database

user

- user_real_name- user_editcount recentchanges

- rc_old_len- rc_new_len

revision

- rev_timestamp page

- page_counter- page_len

rev_user = user_id

rev_page = page_id

user_id = rc_user

Data processUse cases (IV)

4) Google Doc: Google API

Data processUse cases (IV)

4) Google Doc: Google API

Data processUse cases (IV)

4) Google Doc: Google API

Data processUse cases (V)

4) Google Doc: Google API

Data processUse cases (VI)

4) Google Doc: Google API

Data processUse cases (VII)

4) Google Doc: Google API

Data processUse cases (VIII)

4) Google Doc: Google API

Data processUse cases (IX)

4) Google Doc: Google API

Data processUse cases (X)

4) Google Doc: Google API

Data processUse cases (XI)

4) Google Doc: Google API

Data processUse cases (VI)

5) Twitter: API

References[CampbellOblinger2007] Campbell, John P., Peter B. DeBlois, and Diana G. Oblinger. "Academic analytics: A new tool for a new era." Educause Review 42.4 (2007): 40.

[Clow2012] Clow, Doug. "The learning analytics cycle: closing the loop effectively." Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. ACM, 2012.

[Cooper2012] Cooper, Adam. "What is analytics? Definition and essential characteristics." CETIS Analytics Series 1.5 (2012): 1-10.

[DronAnderson2009] Dron, J., & Anderson, T. (2009). On the design of collective applications. In Proceedings of the 2009 International Conference on Computational Science and Engineering, 4, 368–374.

[Dyckhoff2010] Dyckhoff, Anna Lea, et al. "Design and Implementation of a Learning Analytics Toolkit for Teachers." Educational Technology & Society 15.3 (2012): 58-76.

[Eaton2012] Chris Eaton, Dirk Deroos, Tom Deutsch, George Lapis & Paul Zikopoulos, “Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data”, p.XV. McGraw-Hill, 2012.

[GayPryke2002] Cultural Economy: Cultural Analysis and Commercial Life (Culture, Representation and Identity series) Paul du Gay (Editor), Michael Pryke. 2002.

[HR2012] NMC Horizon Report 2012 http://www.nmc.org/publications/horizon-report-2012-higher-ed-edition

[Jenkins2013] BBC Radio 4, Start the Week, Big Data and Analytics, first broadcast 11 February 2013 http://www.bbc.co.uk/programmes/b01qhqfv

[Khan2012] http://www.emergingedtech.com/2012/04/exploring-the-khan-academys-use-of-learning-data-and-learning-analytics/

[LACE2013] Learning Analytics Community Exchange http://www.laceproject.eu/

[LAK2011] 1st International Conference on Learning Analytics and Knowledge, 27 February - 1 March 2011, Banff, Alberta, Canada https://tekri.athabascau.ca/analytics/

[Mazza2006] Mazza, Riccardo, et al. "MOCLog–Monitoring Online Courses with log data." Proceedings of the 1st Moodle Research Conference. 2012.

[Reinmann2006] Reinmann, G. (2006). Understanding e-learning: an opportunity for Europe? European Journal of Vocational Training, 38, 27-42.

[SiemensBaker2012] Siemens & Baker (2012). Learning Analytics and Educational Data Mining: Towards Communication and Collaboration. Learning Analytics and Knowledge 2012. Available in .pdf format at http://users.wpi.edu/~rsbaker/LAKs%20reformatting%20v2.pdf

Data Analytics process in Learning and Academic

Analytics projects

Day 1: Data selection and capture

Alex Rayón Jerezalex.rayon@deusto.es

DeustoTech Learning – Deusto Institute of Technology – University of DeustoAvda. Universidades 24, 48007 Bilbao, Spain

www.deusto.es