Wu et al., NJIT ©2004 1 Constructivist Learning with Participatory Examinations Dezhi Wu, Michael Bieber, S. Roxanne Hiltz Information Systems Department

1Wu et al., NJIT ©2004

Constructivist Learning

with Participatory Examinations

Dezhi Wu, Michael Bieber, S. Roxanne HiltzInformation Systems Department

College of Computing Sciences

New Jersey Institute of Technology

Hyo-Joo HanInformation Systems Department

College of Information Technology

Georgia Southern University


Outline

• Motivation

• Participatory Exam approach

• A bit of theory

• Experimental results

• Interesting issues


Motivation

• To increase learning of course content

• Learning through active engagement– involve students as active participants– with the full exam life-cycle– through peer evaluation

• Minimize overhead for instructors


Outline

• Motivation


• A bit of theory




PE Process• Each student creates 2 exam problems• Instructor edits the problems if necessary• Each student solves 2 problems• Students evaluate (grade) the solutions to the problems

they authored, writing detailed justifications• Other students evaluate each problem a second time• Instructor gives a final grade• optional: Students can dispute their solution’s grade, by

evaluating it themselves and writing detailed justifications

• Instructor resolves the dispute

All entriesposted on-line


Screen ShotWebBoard System


Exam Process ControlAssign ID

Edit questionsAssign who answers questions

Assign level-2 graders

Course Design

Determine Final Grades

Set up on-line environment

Dispute final grade

Level-1 and Level-2 gradersgrade solutions

Make up problems

Read- other problems- other solutions

- grade justifications- disputes

Solveproblems

Instructor Control Process Student Learning Process

Resolve Disputes

Process Flow:Learning from doing the PE activities

additional learning from reading everything peers write


Exam Process ControlAssign ID

Edit problemsAssign who solves problems

Assign level-2 graders

Course Design

Determine Final Grades

Set up on-line environment

Dispute final grade

Level-1 and Level-2 gradersgrade solutions

Make up problems

ConfirmationID, understand process

Read- other problems- other solutions

- grade justifications- disputes

Solveproblems

Instructor Control Process Student Learning Process

Resolve Disputes


Evaluation (grading)

• Evaluation includes:

– Written critique or “justification” (positive or negative)

– Optional: separate sub-criteria to critique

• Solution result is correct and complete (40%)

• Solution was well explained (30%)

• Solution demonstrated class materials well (10%)

• Solution cited appropriate references (20%)

– Grade

• Evaluation may be disputed (optional)

– Student must re-evaluate own solution when disputing

example of four sub-criteria(totals to 100%)


Instructor should provide…

• Detailed instructions and timetable

• Solution: what is expected

• Critiquing and grading guidelines


Outline

• Motivation


• A bit of theory




Constructivism(Learning Theory)

• The central idea is that human learning is constructed, that learners build new knowledge upon the foundation of previous learning{learning throughout the exam process}

• Two classic categorizations– Cognitive Constructivism (Piaget’s theory)– Social Constructivism (Vygotsky’s theory)


Learning

• People learn as they navigate to solve problems (Koschmann et al, 1996) and design representations of their understanding (Suthers 1999)

• Learning requires cognitive flexibility (Spiro et al.

1991), and results from interaction with people having different experiences and perspectives (Goldman-Segall et al. 1998)


Where is Knowledge Constructed in PE?

• In all PE stages:constructing problems, solutions, grade justifications, dispute justifications

• When reading everything their peers write– Students also are motivated to learn more when

peers will read their work (McConnell, 1999).


Outline

• Motivation


• A bit of theory




Course Information

NJIT CIS677: Information System Principles• Graduate level introductory survey core course (Masters/Ph.D.)• Aim: study how IS/IT can be used effectively• Both on-campus and distance-learning sections• software: WebBoard• Traditional Exam:

– Three-hour, in class, 3-4 essay questions, 6 pages of notes

• Used PE 5 times between Fall 1999 and Summer 2002• We compared control groups without PE and treatment groups with

PE

• Also, we used with shorter essay questions in CIS 365, undergraduate course on file structures in Fall 2002, with similar survey results.


Enjoyability

Questions SA A N D SD Mean S.D. #

I enjoyed the flexibility in organizing my resources

26.2% 48.9% 16.7% 3.6% 4.6% 3.88 1.00 221

I was motivated to do my best work 23.5% 42.9% 28.2% 3.4% 2.1% 3.82 .92 238

I enjoyed the examination process

17.2% 42.3% 22.6% 10.5% 7.4% 3.51 1.13 239

SA - strongly agree (5 points); A - agree (4); N - neutral (3); D - disagree (2); SD - strongly disagree (1); the mean is out of 5 points; S.D. - standard deviation

Cronbach’s Alpha=0.68


Perceived Learning

Questions SA A N D SD Mean S.D. #

I learned from making up questions

17.9% 42.5% 21.3% 13.8% 4.5% 3.55 1.08 240

I learned from grading other students answers

17.7% 48.1% 19.4% 9.3% 5.5% 3.63 1.06 237

I learned from reading other people’s answers

15.8% 45.0% 22.1% 11.3% 5.8% 3.54 1.07 240

I demonstrated what I learned in class 13.6% 50.2% 22.6% 10.9% 2.7% 3.61 .95 221

My ability to integrate facts and develop generalizations improved

21.8% 49.2% 25.6% 2.1% 1.3% 3.88 .83 238

I learned to value other points of view

17.6% 51.9% 27.6% 1.3% 1.6% 3.82 .81 239

I mastered the course materials

7.4% 51.6% 31.4% 6.9% 2.7% 3.54 .84 188

Cronbach’s Alpha=0.88


Recommendation: Do Again!

Question SA A N D SD Mean S.D. #

Would you recommend in the future that this exam process used?

20.7% 40.1% 24.5% 8.9% 5.8% 3.60 1.10 237

Similar results for CIS365: undergraduate file structures course using short essay questions (Fall 2002)


Outline

• Motivation


• A bit of theory




Trade-offs

• Trade-offs for students (traditional vs. PE)– Participation: solutions only vs. entire exam life-cycle– Timing: concentrated vs. drawn-out (2.5 weeks)– Access to information: limited vs. the Internet

• Trade-offs for professors– Fewer solutions to evaluate, but each is different– Timing: concentrated vs. drawn-out process– Much more administration


What students liked best

• Active involvement in the exam process

• Flexibility

• Reduction in tension


• Degree of Evaluation– Currently: students only evaluate solutions– What about evaluating:

• quality of problems (how good was the problem?)

• quality of evaluations/grades (how good was the grading?)

– All could be disputed

Extending the PE Approach


Full Collaboration

• Groups for:– Problems, solutions, evaluation, dispute arbitration

• Requires group process support– Group roles: leader, scheduler, etc.– Process: work on each activity together or separately,

internal review– Grading of individual group members– Process Tools: brainstorming, voting, etc.


Support Software

• We plan to develop support software– Guide students (what to do next)– GSS tools for collaboration– Manage administration for instructor– Minimize overhead for students– Minimize overhead for instructors


PE: Contributions

• Systematic technique to increase learning– Constructivist approach, actively engaging students

in the entire problem life-cycle– Minimize overhead for students and instructors

• Experimental evaluation

• Supporting software

• Looking for collaborators to try this out with us!

Thank you! Questions, please?

Documents

Wu et al., NJIT ©2004 1 Constructivist Learning with Participatory Examinations Dezhi Wu, Michael Bieber, S. Roxanne Hiltz Information Systems Department