LOG O Development of a diagnostic system using a testing-based approach for strengthening student prior knowledge Computers & Education (September 2011)

LOGO

Development of a diagnostic system using a testing-based approach for strengthening student prior knowledge

Computers & Education (September 2011)

Yi-Chun Lin, Yen-Ting Lin, Yueh-Min Huang*Department of Engineering Science, National Cheng Kung University

Introduction

Assist instructors in diagnosing and strengthening students’ prior knowledge before new instructions and to enable students to attain greater learning motivation and improved learning performance

A testing-based diagnosis system is proposed in this study to cope with these problems

Methodology

To measure the strength of understanding of prior knowledge

Prior knowledge diagnosis (PKD) model is proposed

Two data sources: Testing information assigned by teachers

Testing information derived by students

Represents a relationship between each concept and test item in a test, and the relationships among the concepts

Represents a relationship between student’s answers and the test items

Methodology

A course specifies n concepts C1, C2, C3,…, Ci,… Cm,…, Cn

Prior knowledge of the subject for r participating students S1, S2, S3,…, Sl,…, Sr

Teacher select k test items from the test item bank to form the pre-test T1, T2, T3,…, Tj,…, Tk

Methodology

Xmj indicates the degree of relevance between the m-th concept and the j-th test item represent the degree of relevance between

each concept and test itemZim indicates the relationship between

the ith and the mth concepts(ranged from 0 to 1) represent the relationship between the

concepts

Methodology - Strength of concept

The strength of concept Ci in the pre-test

Zim represents the relationship between the i-th and the m-th concepts, 0 ≤ Zim ≤ 1

Xmj indicates the degree of relevance between the m-th concept and the j-th test item, 0 ≤ Xmj ≤ 1

0 ≤ S(Ci) ≤ nk

Methodology - Importance ratio of concept

The importance ratio of concept Ci in the pre-test

Zim represents the relationship between the i-th and m-th concepts, 0 ≤ Zim ≤ 1

Xmj indicates the relevance degree between the m-th concept and the j-th test item, 0 ≤ Xmj ≤ 1

0 ≤ IRP(Ci) ≤ 1

Methodology - Understanding strength of the lth student

The understanding strength of the lth student on the ith concept

Rlj indicates the answer of the l-th student on the j-th test item• If the student answers the test item correctly,

Rlj is 1; otherwise Rlj is 0

Zim represents the relationship between the i-th and the m-th concepts

Xmj indicates the degree of relevance between the m-th concept and the j-th test item

Methodology - Understanding strength of the concept

Translate the importance ratio of the concept into the understanding strength of the concept

To t(Ci) represents the threshold value of the i-th concept, 0 ≤ t(Ci) ≤ 1

m indicates the gradient of the function, m = 1 IRP(Ci) represents the importance ratio of

concept Ci in the pre-test, 0 ≤ IRP(Ci) ≤ 1

b is the point at which the line crosses the y-axis, b = 0

Illustrative example

Test item Concept

C1 C2 C3 C4 C5

T1 1 0 0 0 0

T2 0.2 0 0.6 0 0.2

T3 0 1 0 0 0

T4 0 0.4 0 0 0.4

T5 0 0 0 0.2 0

Illustrative example of relationship between test items and concepts


Concept Concept

C1 C2 C3 C4 C5

C1 1 0 0.4 0 0

C2 0 1 0.6 0 0.2

C3 0.4 0.6 1 0.2 0.6

C4 0 0 0.2 1 0.4

C5 0 0.2 0.6 0.4 1

Illustrative example of relationship between concepts

Strength of Concept

The strength of the first concept (C1): S(C1) = Z11 *X11 + Z11 * X12 + Z13 * X32

= 1.0 × 1.0 + 1.0 × 0.2 + 0.4 × 0.6 = 1.44

S(C2) = Z21 *X23 + Z21 * X24 + Z23 * X32 + Z25 * X52 + Z25 * X54

= 1.0*1.0 + 1.0*0.4 + 0.6*0.6 + 0.2*0.2 + 0.2*0.4 = 1.88

Test item

Concept

C1 C2 C3 C4 C5

T1 1 0 0 0 0

T2 0.2 0 0.6 0 0.2

T3 0 1 0 0 0

T4 0 0.4 0 0 0.4

T5 0 0 0 0.2 0

ConceptConcept

C1 C2 C3 C4 C5

C1 1 0 0.4 0 0

C2 0 1 0.6 0 0.2

C3 0.4 0.6 1 0.2 0.6

C4 0 0 0.2 1 0.4

C5 0 0.2 0.6 0.4 1

Importance Ratio of Concept

Importance ratio of concept

Concept

C1 C2 C3 C4 C5

IPR 0.19 0.25 0.31 0.07 0.18

The importance ratio of second concept (C2)

IRP(C2) = 1.88 / 7.52 = 0.25


Test itemStudent

S1 S2 S3 S4 S5

T1 1 1 0 1 1

T2 1 0 1 1 1

T3 0 0 0 0 1

T4 0 1 0 1 1

T5 0 0 1 0 0

Illustrative example of the relationship between students’ answers and test items

Relationship between students’ answers and

test items (R)


StudentConcept

C1 C2 C3 C4 C5

S1 1 0.21 0.52 0.36 0.42

S2 0.69 0.26 0.38 0.29 0.36

S3 0.31 0.21 0.36 0.71 0.48

S4 1 0.47 0.72 0.64 0.79

S5 1 1 0.98 0.64 0.94

Relationship between students’ understanding

strength and concepts (USS)

Test item

Student

S1 S2 S3 S4 S5

I1 1 1 0 1 1

I2 1 0 1 1 1

I3 0 0 0 0 1

I4 0 1 0 1 1

I5 0 0 1 0 0

USS(S4,C5) = = = = 0.79

Test item

Concept

C1 C2 C3 C4 C5

I1 1 0 0 0 0

I2 0.2 0 0.6 0 0.2

I3 0 1 0 0 0

I4 0 0.4 0 0 0.4

I5 0 0 0 0.2 0

ConceptConcept

C1 C2 C3 C4 C5

C1 1 0 0.4 0 0

C2 0 1 0.6 0 0.2

C3 0.4 0.6 1 0.2 0.6

C4 0 0 0.2 1 0.4

C5 0 0.2 0.6 0.4 1

Relationship between concepts (Z)Relationship between test items and

concepts (X)

R41Z51X11 R41Z52X21 R41Z53X31 R41Z54X41 R41Z55X51





1.0 x 0.0 x 1.0 1.0 x 0.2 x 0.0 1.0 x 0.6 x 0.0 1.0 x 0.4 x 0.0 1.0 x 1.0 x 0.0

1.0 x 0.0 x 0.2 1.0 x 0.2 x 0.0 1.0 x 0.6 x 0.6 1.0 x 0.4 x 0.0 1.0 x 1.0 x 0.2

1.0 x 0.0 x 0.0 1.0 x 0.2 x 1.0 1.0 x 0.6 x 0.0 1.0 x 0.4 x 0.0 1.0 x 1.0 x 0.0

1.0 x 0.0 x 0.0 1.0 x 0.2 x 0.4 1.0 x 0.6 x 0.0 1.0 x 0.4 x 0.0 1.0 x 1.0 x 0.4

1.0 x 0.0 x 0.0 1.0 x 0.2 x 0.0 1.0 x 0.6 x 0.0 1.0 x 0.4 x 0.2 1.0 x 1.0 x 0.0

1.0 x 0.0 x 1.0 1.0 x 0.2 x 0.0 1.0 x 0.6 x 0.0 1.0 x 0.4 x 0.0 1.0 x 1.0 x 0.0

1.0 x 0.0 x 0.2 1.0 x 0.2 x 0.0 1.0 x 0.6 x 0.6 1.0 x 0.4 x 0.0 1.0 x 1.0 x 0.2

0.0 x 0.0 x 0.0 0.0 x 0.2 x 1.0 0.0 x 0.6 x 0.0 0.0 x 0.4 x 0.0 0.0 x 1.0 x 0.0

1.0 x 0.0 x 0.0 1.0 x 0.2 x 0.4 1.0 x 0.6 x 0.0 1.0 x 0.4 x 0.0 1.0 x 1.0 x 0.4

0.0 x 0.0 x 0.0 0.0 x 0.2 x 0.0 0.0 x 0.6 x 0.0 0.0 x 0.4 x 0.2 0.0 x 1.0 x 0.0


Threshold value of concept Concept

C1 C2 C3 C4 C5

IPR 0.19 0.25 0.31 0.07 0.18

*m = 1, b = 0 for threshold function in this case

PKT&D System Architecture

Experiment

Participants : A course instructor 80 university students

Course: bioinformaticsGroup:

Control group: 40 students (used the PKT&D system)

Experiment group: 40 students (did not use the PKT&D system)

Experiment

Subject: sequence analysis approaches and tools

Concepts in prior knowledge: sequence characteristics and structures, statistical hypothesis testing, and formula expression format

Unit Instruction activities Time (min)

Understanding the importance of similarity

1. A series of guided questions (5)

2. Slide presentation (15)3. Discussions (10)

30

Introduction to the most popular data-mining tool: BLAST


2. Slide presentation (15)3. Practice (10)

30

BLASTing protein sequences


2. Slide presentation (10)3. Practice (15)

30

Understanding BLAST output1. Slide presentation (15)2. Discussions (15)

30

BLASTing DNA sequences


2. Slide presentation (10)3. Practice(15)

30

The BLAST way of doing things1. Slide presentation (15)2. Practice (15)

30

Experiment Process

The learning motivation post-test score between the two groups

GroupNumber of students

Mean S.D.Adjusted

meanF(1, 77) P-value

Experimental group

40 46.45 3.42 46.411 16.340 .00*

Control group

40 41.05 7.59 41.086

Total number of students

80 43.75 6.38

To measure the students’ learning motivation, Motivated Strategies for Learning Questionnaire was adopted in this study. using nine questionnaire items and a seven-point

Likert scale

The paired t-test results of learning motivation for the two groups of students

Group Tests N Mean S.D. t(39)

Experimental group

Pre-test 40 34.85 4.451 -9.688*

Post-test 40 46.45 3.425

Control group

Pre-test 40 33.1 9.072 -3.074*

Post-test 40 41.05 7.591

Students’ attitude towards bioinformatics learning

# ItemExperiment

Group (Mean, S.D.)

Control Group (Mean, S.D.)

t-value

1 I like learning bioinformatics 5.45/0.87 5.05/1.08 1.82

2The bioinformatics learning activities are helpful

5.68/0.76 5.35/0.83 1.82

3I like to practice using the software in the bioinformatics learning

5.75/0.87 5.53/0.88 1.15

4I had enough ability to learn the bioinformatics material

5.28/0.99 4.38/1.44 3.25*

5I can meet the instructor’s requirements during the bioinformatics learning process

5.38/0.90 4.40/1.52 3.50*

6I can understand the bioinformatics material taught by the instructor

5.45/0.85 4.55/1.55 3.22*

Experiment group students’ perceptions of using the PKT&D system

# Question EU(%) QU(%) SU(%) Neither(%) SL(%) QL(%) EL(%) Mean

1

Using the PKT&D system in learning bioinformatics would enable me to diagnoseand strengthen prior knowledge more effectively

0 5 7.5 5 27.5 32.5 22.5 5.43

2Using the PKT&D system would improve my bioinformatics learning performance

0 0 5 20 30 30 15 5.30

3

Using the PKT&D system in learning bioinformatics would increase my learningcomprehension productivity

0 0 2.5 12.5 42.5 30 12.5 5.38

4Using the PKT&D system would make it easier to learn bioinformatics

0 2.5 7.5 10 37.5 37.5 5 5.15

5I would find the PKT&D system useful in the bioinformatics class

0 0 0 10 42.5 35 12.5 5.50

Note: EU: Extremely Unlikely; QU: Quite Unlikely; SU: Slightly Unlikely; SL: Slightly Likely; QL: Quite Likely; EL: Extremely Likely.

Independent pre-test on knowledge of bioinformatics of the two groups

Variable

Pre-test

t-valueN Mean S.D.

Experiment group 40 50.25 14.32 -0.385

Control group 40 51.50 14.77

The paired t-test results of the learning improvement for the two groups

Group Tests N Mean S.D. t-value

Experimental group

Pre-test 40 50.25 14.23 -8.460*

Post-test 40 69.50 11.54

Control group

Pre-test 40 51.50 14.77 -9.595*

Post-test 40 63.50 13.12

*p<0.05.

Independent Post-test on knowledge of bioinformatics of the two groups

Variable

Post-test

t-valueNumber of students

Mean S.D.

Experimental group

40 69.50 11.54 2.172*

Control group 40 63.50 13.12

Example interview comments about the three topics

Inductivetopics Perspectives Interviewees Transcript sample of interview comments

InstructionRealization of

studentsIN

I felt that the students in the experiment group better met my requirements during the course.

Achievement of students

INI felt that the students in the experiment group demonstrated high performance in each learning activity.

Progress of instruction

INI had much more time to teach the concepts in more detail and interact with the students of experiment group.

Learning situation INOnly one third of the students from the control group could fully follow the activities and understand my instructions.

SC & SELearning about bioinformatics software through practicing using it was interesting. I would have preferred more direct instruction from the course instructor.

SCI felt some of the concepts were difficult to grasp, which led to obstacles when I used the software during the course.

InteractionDiscussion willingness

INThe students in the experiment group had better discussions than the control group.

Responsiveness of students

INThe students in the experiment group often gave feedback and asked questions.

TechnologySystem

usefulnessSE

I felt the user interface of the PKT&D system was clear, straightforward, and convenient to use. .I clearly saw the diagnostic results and learning suggestions.

Auxiliary components

IN & SEI felt that the PKT&D system served as a guide that helps the students to diagnose the weakness of their concepts. I can learn more prior knowledge using the PKT&D system.

Evaluation of correctness rate results for the three concepts diagnosis


represents the correctness rate of the diagnoses derived from the PKT&D system,

𝐶𝑅=𝑛−(𝑛−𝑚)

𝑛×100%


Student ID 1 2 3 4 5 6 7 8

Correctness rate of

diagnoses

Expert 1 100% 100% 100% 66.60% 100% 100% 100% 100%

Expert 2 100% 66.60% 100% 66.60% 100% 100% 100% 100%

Student ID 9 10 11 12 13 14 15 16

Correctness rate of

diagnoses

Expert 1 66.60% 100% 66.60% 100% 100% 100% 100% 100%

Expert 2 100% 100% 33.30% 100% 100% 100% 100% 100%

Student ID 17 18 19 20 21 22 23 24

Correctness rate of

diagnoses

Expert 1 100% 100% 100% 100% 100% 100% 100% 100%

Expert 2 100% 100% 100% 100% 66.60% 100% 100% 100%

Student ID 25 26 27 28 29 30 31 32

Correctness rate of

diagnoses

Expert 1 100% 66.60% 33.30% 100% 100% 100% 100% 66.60%

Expert 2 100% 100% 100% 100% 66.60% 100% 100% 100%

Student ID 33 34 35 36 37 38 39 40

Correctness rate of

diagnoses

Expert 1 100% 100% 100% 33.30% 100% 33.30% 100% 100%

Expert 2 100% 100% 100% 100% 100% 33.30% 100% 100%

Evaluation of correctness rate results for the five concepts diagnosis

Student ID 1 2 3 4 5 6 7 8

Correctness rate of

diagnoses

Expert 1 80% 100% 100% 80% 100% 100% 80% 100%

Expert 2 100% 100% 100% 100% 100% 100% 80% 100%

Student ID 9 10 11 12 13 14 15 16

Correctness rate of

diagnoses

Expert 1 100% 100% 100% 80% 100% 80% 100% 60%

Expert 2 100% 100% 100% 100% 100% 80% 100% 40%

Student ID 17 18 19 20 21 22 23 24

Correctness rate of

diagnoses

Expert 1 100% 60% 100% 60% 100% 100% 100% 100%

Expert 2 60% 40% 80% 60% 100% 80% 100% 100%

Student ID 25 26 27 28 29 30 31 32

Correctness rate of

diagnoses

Expert 1 40% 80% 100% 100% 100% 100% 100% 100%

Expert 2 100% 80% 100% 100% 80% 100% 100% 100%

Note: Correctness rate obtained by comparing the diagnoses of the experts with those obtained using the proposed approach to artificial intelligence course. The average correctness rates are 90.625% and 90% for the students.

Conclusion & Discussion

Propose a testing-based approach to diagnose the strength of individual students’ prior knowledge of concepts, and then provide them with appropriate materials to strengthen this

Provide instructors can undertake their teaching more smoothly

Educators can use the proposed system in different educational contexts

Limitation

Two variables of the linear function have to be adjusted based on instructors’ expertise in different educational contexts

Future Work

The number of test items in the item bank should be continually increased to address various subject objectives and the instructors’ needs

Students’ learning portfolio can be integrated into the proposed system to develop more appropriate diagnosis mechanisms

Documents

LOG O Development of a diagnostic system using a testing-based approach for strengthening student prior knowledge Computers & Education (September 2011)