Agile Technologies for Personalizing Instruction

Faisal Ahmad, Sebastian de la Chica, Qianyi Gu, Shaw Ketels, Ifi OkoyeTammy Sumner, Jim Martin, Alice Healy, Kirsten Butcher, Michael Wright

Digital Learning SciencesUniversity of Colorado at Boulder

University Corporation for Atmospheric Research

This work is supported in part by an ICS Generalization Grant, and NSF awards #0537194 and #0734875

2

Central Challenge

Enable personalized learning, while still supporting recognized learning goals

Do it at scale

How People Learn (NRC)

Extreme Diversity (KnowledgeWorks)

Disrupting Class (Christiansen)

N=1, R=G (Prahalad)

www.DLESE.org

Strandmaps.NSDL.org

Curriculum Customization

CLICK Personalization Service

4

CLICK Personalization Service

Automatically identify potential learner misconceptions by analyzing student work

Customize the selection and presentation of learning resources based on identified misconceptions

High school plate tectonics

5

Guiding Principles Personal and intentional

Build on learner understanding Learner control Learning goals organize and guide

Agile technologies Domain independent: knowledge maps for

human cognition and machine reasoning Automatic: NLP and ML Embeddable: web services, not applications Open: leverage existing web content

6

7DEMO

8

Major CLICK Components What should students know?

Domain knowledge map

What do they already understand? Compare student and domain maps

What learning activities would be useful? Select resources to address misconceptions and gaps

How to embed in learning environments? Provide web service to application and portal developers

9

Detecting Potential Knowledge Gaps

Alignment andComparison

Student Essay

(1) Student Knowledge Model (2) Domain Competency Model

Digital Library Resources

(3) Knowledge Trace

Human-Centered Methodology Expert studies to inform algorithms

(Ahmad et al 2007)

Domain knowledge map creation Student essay to student knowledge map Knowledge gap diagnosis Personal instruction plan generation

Expert scoring of intermediate results

Mixed-method learning study10

Algorithms Concept extraction (de la Chica 2008)

MEAD: multi-document summarization toolkit (Radev et al 2004)

Custom sentence scoring features: standards, gazetteer, hypertext, content word density

Eliminate redundancy, rank and choose top 5% Student essays – lexical chains (de la Chica 2008)

Knowledge gaps – NLP and graph structure comparisons (Ahmad 2008)

Personalized information retrieval – concept matrix (Gu 2008)

11

12

CLICK Personalization Web Service

Misconception diagnoses and knowledge map generation exposed via request types (Ahmad 2008)

Submit or remove a concept map Construct student map from essay Construct domain map from URLs Get student misconceptions Get important concepts Get related concepts

32 undergraduates 16 – CLICK to revise essays on Earthquakes

and Plate Tectonics 16 – control Digital Library environment

Data collected original essays, revised essays, detailed screen

capture “movies”, reflective questions, factual knowledge tests

13

Mixed-Method Learning Study

14

Essay Content Revisions

Shallow. CLICK< Control: F (1, 27) = 3.602, p = .068 (TREND)

Deep. CLICK > Control: F (1, 27) = 5.222, p = .030 (SIG EFFECT)

15

Shallow revisions Copying out of resource,

Paraphrasing, Integrated copying, Integrated paraphrasing, Concept deletion

Deep revisions Integrated sentence

paraphrasing to create new sentence, Integrated resource paraphrasing to create new sentence, Inferencing, Generation

Codes based on Wiley and Voss 1999, Constructing arguments from multiple sources

Types of Content Revisions

16

CLICK > Control: F (1, 27) = 6.490. P = 0.17 (SIG EFFECT)

Omissions Gaps in student content

knowledge such as missing details and missing concepts

Incorrect Statements Coding still underway

Process Data

17Exploration. CLICK>Control: F (1, 27) = 6.076, p = .02 (SIG EFFECT)Essay. CLICK>Control: F (1, 27) = 6.815, p = .015 (SIG EFFECT)Switches. CLICK>Control: F (1, 27) = 6.447, p = .017 (SIG EFFECT)

Exploration Episodes Exploring learning

resources and personalized feedback

Essay Episodes Revising or working with

essay

Switches Moving between essay and

exploration Integration of content

resources and developing essay

Recognizing need for outside knowledge source

Conclusions Learning - Initial CLICK results promising

Encourages deep content revisions Promotes integration between information

seeking and knowledge transformation Students more likely to recognize that they

need new knowledge, a critical element of self-directed learning

Algorithm Generalization: Promising results for “near” domain

Misconception prioritization and link generation need further work

18

Further Reading Ahmad, F., S. de la Chica, K. Butcher, T. Sumner, and J. Martin. (2007). Towards

automatic conceptual personalization tools. In Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2007): Vancouver, Canada (June 18-23), pp. 452-461.

Butcher, K. and S. de la Chica. (in press). Supporting student learning with adaptive technology: Personalized conceptual assessment and remediation. In M. Banich and D. Caccamise (Eds.), Generalization of Knowledge: Multidisciplinary Perspectives. London, England: Taylor and Francis.

de la Chica, S., F. Ahmad, J. Martin, and T Sumner. (2008). Pedagogically useful extractive summaries for science education. 22nd Meeting of the International Committee for Computational Linguistics (COLING 2008).

de la Chica, S., F. Ahmad, T. Sumner, J. Martin, and K. Butcher. (2008). Computational foundations for personalizing instruction with digital libraries. International Journal of Digital Libraries. To appear in the Special Issue on Digital Libraries and Education.

Gu, Q., de la Chica, S., Ahmad, F., Khan, H., Sumner, T., Martin, J., Butcher, K. (2008). Personalizing the Selection of Digital Library Resources to Support Intentional Learning. Research and Advanced Technology for Digital Libraries, 12th European Conference, ECDL 2008, Aarhus, Denmark, September 14-19. Lecture Notes in Computer Science, pp. 244-255.

19

Examples of “Good” Concepts

20

  Plate Tectonics Weather and Climate

Good standalone concept

A gradual build-up of mechanical stress in the crust, primarily the result of tectonic forces, provides the source of energy for earthquakes; sudden motion along a fault releases it in the form of seismic waves.

The shape and position of waves in the polar jet stream determine the location and the intensity of the mid-latitude cyclones.

Good concept in context

Many places near this plate boundary are at high risk for earthquakes, including the San Francisco area, the Pacific Northwest, and Alaska, yet fully half the nation's earthquake hazard is in Southern California.

This energy is used to heat the Earth's surface and lower atmosphere, melt and evaporate water, and run photosynthesis in plants.

21

22

Detecting Potential Knowledge Gaps

Alignment andComparison

Student Essay

(1) Student Knowledge Model (2) Domain Competency Model

Digital Library Resources

(3) Knowledge Trace