Team Difficulty, UI, and NQ

Team Difficulty, UI, and NQ

Team Difficulty, UI, and NQ

Mi Seon Park

Caitlyn Seim

David Fleischhauer

Charles Ethan Hayes

Weiqing Li

2

Introduction and MotivationIntroduction and Motivation

• Wanted to make improvements to the system that would make it more useful for students as a learning tool

• Mined data from previous semester was not being fully utilized

• Improvements could be made to give the students and instructors more information, more options and an overall smarter system

3

ObjectivesObjectives

• Previous system lacked a concise, customizable presentation of question difficulty, skips, etc. for instructor view

• Previous system only presented random questions• Instructors and students could only judge difficulty

based on subjective rating• Using the mined difficulty and the student’s success at

answering questions, the system could be automated to give the student a specific question, not just a random one

4

DifficultyDifficulty

• Compiled data from Fall 2011 semester onward• This data was stored in the MinedData and

MinedDataMC tables

5

DifficultyDifficulty

• From data the key factors for difficulty are extracted:– Average Duration (default 40%)– Average Score (default 25%)– Number of Skips (default 25%)– Average Rating (default 10%)

• The computed difficulty information is then stored in the questions_difficulty table.

6

Questions_DifficultyQuestions_Difficulty

7

Difficulty ImprovementsDifficulty Improvements

• Three ways are used to alternate the difficulty ratings to get a better spread:– Using the tags to select the max and min value– Capping the average skips and duration at one

standard deviation above and below– Capping the average skips and duration at the Nth

percent.

8

Regular Difficulty Regular Difficulty

9

Difficulty 2Difficulty 2

10

Difficulty STDDifficulty STD

11

Difficulty Drop N (N=5%)Difficulty Drop N (N=5%)

12

The Overall Plot of difficulty ordered by the original The Overall Plot of difficulty ordered by the original

13

Tools to analyze the difficulty Tools to analyze the difficulty

• The stat php files that are stored in VIP/Fall2012 directory where there is a stat file for each type of difficulty that fetches the overview of that difficulty category.

• Manually go into the sql database and copy the data into excel and plot.

• The MATLAB file that is stored in the same directory which allows one to plot all the difficulty using either data from each of the chapters or using all the data points.

14

Improvements to UIImprovements to UI

• Add different elements to UI that uses the mined data from previous semester and difficulty calculated this semester.– Upgrade student view to display difficulty information

to the student– Upgrade instructor view to display different

information about the question

15

Improvements to UIImprovements to UI

• Student View:– classes/ITS_screen2.php– classes/ITS_rating.php– css/ITS.css

• Instructor View:– Course.php– classes/ITS_statistics.php– js/ITS_course_jquery.php– ajax/ITS_admin.php

16

Improvements to UIImprovements to UI

17

Improvements to UIImprovements to UI

18

Improvements to UIImprovements to UI

19

Old ITS SystemOld ITS System

Submit

Random Question

Score

Skip

20

Created a new ITS_NQ class to handle the Next Question Selection

21

Sample of a function in ITS_NQFunctions are commented and clean to make additions in the future easier

22

Screen shot of the new proposed ITS system

23

Database EntriesDatabase Entries

• Each button has a unique database entry– Useful for future debugging/datamining

• ITS_NQ operates based off of the button’s database entry for the most recent question

24

Each database event correlates to a case in ITS_NQ

25

Submit Button functionalitySubmit Button functionality

• Regular Submit– User receives a harder or easier question based

on whether their answer was correct (>50%) or incorrect

• User-selected submits/skips– User receives a harder or easier question based

on which type of question they requested

26

Skip Button functionalitySkip Button functionality

• Forward/Back– The system keeps track of which questions you

have seen and will let you go back and forth to previous questions

– A question can only be inserted into the list once, so it must be restarted if you want to “smart skip” through the questions again

• Easier/Harder– User receives a harder or easier question based

on which type of question they requested

27

New ITS System General ViewNew ITS System General View

Score

Skip Easier Skip Harder

Question

Score>50Score<=50

Question

Harder Question

Easier Question

A Submit Button

Submit Easier Submit Harder

Smart Submit

28

DirectionFind last question

BACK Go forward 1 question

FORWARD

Question

Difficulty

Randomize question from array

Get array of all questions that are

easier

Get array of all questions that are

harder

Get array from “Smart Skip”

EASIERHARDER

empty

Question

Skip button functionalities

29

Bucket/Increment SystemBucket/Increment System

CenterSpan

Center=difflast

Span=.25

QueryDiffMin=Center-SpanLDiffMax=Center+Span

H

User receives Question

Result not empty

Result Empty, Min>0 or Max<10

Span=Span+Span

Assignment Finished

Result Empty, Min=0 and Max=10

Submit or skip

30

Array is empty

Array of questions Diff=Center+-span

Restart list (forward/back) to let all of the unanswered questions be available

Output next question

NO

Get last event info

Smart SkipCenter=round(difflast)

Span=.5

Center = round(difflast)+1

Span=.5

Center = round(difflast)-1

Span=.5

Answered question with submit

Score<=50

Answered question with submit

Score>50

Span = span + .5Until) center+-span

=(0,10)

First empty event

Randomize question from array

Yes

YesNO

No eventCenter = 5Span=.5

“Smart” submit and skip functionality

31

ExamplesExamples

• User submits (harder) question with diff=6.2– Next question comes from difficulty range 6.1-

6.45, but this bucket is empty– Bucket expands, question should come from

range 6.0-6.70. Bucket has at least one question, user receives a question

• User smart submits a matching question (2 of 5 correct) with diff=4.8– Next question comes from range 4.55-4.90

32

Automating Question SelectionAutomating Question Selection

• Using the mined difficulty and the student’s success at answering questions, the system is automated to give the student a custom-selected question, not just a random one

• Created 2 schemes:– Scheme of One– History Scheme

33

Groundwork:New Variables in Screen2.phpGroundwork:New Variables in Screen2.php

$last_id= query("SELECT question_id FROM stats_' . $this->id . ' ORDER BY id DESC LIMIT 1;");

- id is an auto-increment index of questions taken- DESC makes the last in on top- LIMIT 1 gives that value

$last_dif=query("SELECT difficulty FROM questions_difficulty WHERE q_id = $last_id;");

$last_event = query("SELECT score FROM stats_' . $this->id . ' WHERE question_id = $last_id;"); -NULL for skips

34

Scheme of OneGetting the next questionScheme of OneGetting the next question

• For any Null scores (skips)– Default to 0

• Changed from default query to 2 query statements depending on conditionals (if’s):• if they scored >=50%

on the last answered question (this accounts for Matching types etc.)

• if they scored <50% on the last question

If (is_null($last_event)){$last_event=”0”;

}

35

Scheme of One Running out of Acceptable QuestionsScheme of One Running out of Acceptable Questions

• When user has answered all questions in the desired difficulty range• Query is made again without difficulty restrictions• Could have future applications with query in

broader difficulty range (but History Scheme is more complex)

36

37

38

Scheme of One ExpandingScheme of One Expanding

• Expanded the queried history for the user• To include history of past three answers• This requires that at least 3 entries be in the

database• Future work may work around this DB

requirement• And grow to use a scheme of many past answers

39

Scheme of One and History SchemeLogical FlowScheme of One and History SchemeLogical Flow

The student’s last score: any skip>=50% or <50%

Harder Easier or the same

Next question is:

The student’s last 3 scores were NULL, NULL, NULL (all skips): Yes No

Harder

The average of the last 3 scores: >=50% <50%

The average of the last 2 scores: >=50% <50%

Last diff. -1/+2 Easier

40

History Scheme:CasesHistory Scheme:Cases

• NULL, NULL, NULL—When there are three skips (any kind) in a row—This prevents skipping until a suitably easy question

appears—Also alright if the user is skipping harder, this will give

harder• Average of 3 is >=50

• Doing well• Harder questions will challenge the student

• Average of 3 is <50• Consider average of most recent questions

41

History Scheme:CasesHistory Scheme:Cases

• Average of 2 is >=50— Improving—Questions are harder, but not too hard (-1/+2

range)—Could be implemented as if (Average of most

recent 2 > Average of 3 ~or~ Average of earliest 2)• Average of 2 is <50

• Not improving• Not only skipping, also trying• Easier questions will help the student

42

Final ResultsFinal Results

• General UI improvements for Instructor view• A variety of difficulty scores for existing questions and

the ability to dynamically update difficulties• Next Question selection was automated using scripts

based upon difficulty values and student answers• Next Question selection can be automated or user-

selected• The NQ algorithms and NQ/skip buttons enable a more

intelligent tutoring system

43

Future WorkFuture Work

• More complex selection algorithms can be developed and tested based on a longer history of student usage

• Student rating/grading can be implemented based on patterns of use

• A practical plan can be put in place to limit repetitive skipping/misuse of the NQ buttons and selection

• More advancements can be made to the UI to display information better for the students and professor

• Weighting the score of the questions based on their difficulty

• More complex algorithm/testing more percentage choices to get the best difficulty spread.

44

Bonus MaterialBonus Material

45

Groundwork:New Variables in Screen2.phpGroundwork:New Variables in Screen2.php

• Tested variable assignment queries – All returned results of one value– Values were correct (desired table location was

read)– NOTE: calls currently use difficulty field in

questions_difficulty

46

GroundworkVariable Assignments in _screen2GroundworkVariable Assignments in _screen2

• Revised assignment statements to be the following:

47

Groundwork$last_event fix for multiple entriesGroundwork$last_event fix for multiple entries

$last_event = & $mdb2->query("SELECT score FROM stats_' . $this->id . ' WHERE question_id = $last_id;");

$last_event =& $mdb2->query('SELECT score FROM stats_' . $this->id . ' ORDER BY id DESC LIMIT 1;');

• Score variable was revised to return the most recent incidence of that question in the user’s table (rather than NULL for any times skipped before answering)

48

Scheme of One Question SelectionScheme of One Question Selection

• When correct (>=50%)– Makes sure that the eligible questions

for the next question are only of greater or equal difficulty.

• When last question answered was incorrect- Eligible next questions are only of lesser difficulty.

49

Scheme of One Other StepsScheme of One Other Steps

• Compared and cleaned queries

• Set to display query for demos and testing

• Discovered occurrence of running out of <harder/easier> questions

50

Scheme of One ExpandingScheme of One Expanding

• Expanded the queried history for the user• To include history

of past three answers

• This requires that at least 3 entries be in the database

• Future work may work around this DB requirement

• And grow to use a scheme of many past answers