Labrador: A Tool for Automated Grading Support in Multi-Section Courses

Labrador: A Tool for Automated Grading Support in Multi-Section Courses

Drexel University ProgrammingLearning Experience (DUPLEX)http://duplex.mcs.drexel.edu

Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, Paul Zoski

Drexel UniversityMathematics and Computer Science

2

Roadmap

Introduction

Problems and Solution Goals

Labrador

Discussion

3

The Group

Bruce Char– Professor, Computer Science

Nira Herrmann– Professor and Head, Mathematics

Jeffrey L. Popyack– Associate Professor, Computer Science

Paul Zoski– Instructor, Math and Computer Science

Christopher D. Cera– Computer Science Graduate Student

Robert N. Lass– Computer Science Undergraduate Student

4

Who Am I

First TA in MCS to experiment with WebCT in December 2000

Courses I have TA’ed using WebCT– 3 x Introductory Programming I, II– 1 x Object Oriented Programming

Migrated course content to WebCT from previous course website– HTML assignments and labs into question database– Gradebook maintainer

Wrote demo’s and documentation to train other TAs

Developer of software supplements to WebCT to support course administration

5

The Course

Intro Programming I, II

Variety of Majors – Computer Science– Computer

Engineering– Information Systems– Mathematics– Digital Media

Class– 1 x 1 hour lecture– 2 x 1 hour labs

People– 250-300 Students– 2-3 Professors– 10-12 Teaching

Assistants

6

The Duplex Project: An Overview

Take advantage of advances in Information Technology to improve instruction and reduce costs for computer programming courses

– Modular Structure Multiple Entry Points Multiple Audiences Multiple levels of knowledge (Bloom’s Taxonomy)

– Computer Supported Cooperative Work (CSCW) in student labs

– Online Services – Today’s Topic

7

Roadmap

Introduction

Problems and Solution Goals

Labrador

Discussion

8

Before WebCT

Hundreds of people involved in paper exchanges of handwritten assignments and quizzes

Testing programs required floppy exchanges No Chat No newsgroup-style threads Feedback and grades are not online

9

Before WebCT

Large Classes

– Managing several hundred students is difficult due to the logistics involved with organizing paperwork and people.

10

WebCT

Service from Information Resources & Technology (IRT) department (v3.1)

MCS started Dec. 2000 (v3.5)

Upgraded to v3.7 recently

Transition to v3.8 planned in Fall

11

Noteworthy Features

General Course Website

Labs: Online Quizzes with Automated Grading

Homework: Online Assignments

Joint Staff--Student Chat

Discussion Threads

13

Software Design Goals

Bulk Assignment Downloading (prior to 3.7)

Bulk Quiz Downloading

Section Sorting for bulk downloads, or only one section

14

Software Design Goals

Post-Processing student files– Archive Extraction (tar, zip)– Decompress (gz, zip)– Decode (uue)

Minimal staff intervention when transferring submissions between systems, eg. Plagiarism Detection Systems (PDS)

Automatically collate source code

Generate electronic documents to facilitate grading and archiving

Remote Execution– downloading to computer x (on

campus) while operating at computer y (off campus)

15

The Bigger Picture

Course-Specific Tasks– Not all processing should be done on the server

Select files have to be transferred to a different system for further processing and analysis by staff

Client-side support is needed to perform this, preferably automated and not necessarily using a web browser.

16

Roadmap

Introduction

Problems and Solution Goals

Labrador

Discussion

17

Labrador: Our Solution

Client-side WebCT Supplement

Implemented in Perl

Works for users with TA and Designer access to WebCT

18

Perl

Practical Extraction and Reporting Language

Created in 1987 to replicate Unix shell functionality

Powerful text manipulation

High-level: rapid prototyping and development

Large library of modules and active development community

Perl is used by many applications, including WebCT

19

A Taste of Labrador Perl Programming

20

Labrador works on Windows, Mac, Linux, Other Unix Variants, etc.

Perl has been ported to all major operating systems

Perl programs can be compiled:

– No need for the end user to install Perl

– Perl2exe for Windows support (www.perl2exe.com)

21

User Interface

Different users / Different UI preferences

Command-line

Interactive

Configuration File

GUI (in development)

22

Required Info

http://webct.drexel.edu/SCRIPT/CS164_Fall2002/scripts/designer/dropbox_edit.pl?DROPBOX_ASSN_VIEW+_side_nav++1006285909

Username/Password

Server URL webct.drexel.edu

Course ID CS164_Fall2001

Submission Name or ID 1006285909

Optional Username List

23

Components

SubmissionDownloader

SectionSorting

PostProcessing

Moss

JPlag

PDFGenerator

24

Bulk Downloader

Web-Crawler: simulates clicks of actual staff member

– Downloads actual web content (HTML)

– Parses HTML to find desired text or URLs to crawl to next

– Uses HTTP operations

Works on assignments and quizzes

Only component in Labrador which interacts with WebCT

25

Organizing Submissions by Section

Not supported in original Labrador prototype, added later on.

Organizes files into directories

3 possible input methods:– Creating a Section column in

gradebook– “Username, Section” CSV File– Username file

26

PDF Generation for Electronic Mark-up

Adobe Portable Document Format can be viewed on all major platforms

With Adobe Acrobat, PDFs can be annotated by graders

Sony VAIO Slimtop PC (PCV-LX920)

Wacom Pen Tablet.

27

VAIO / Wacom

28

PDF Markup Example

29

Redistribution

How can we return annotated PDF’s back to students using WebCT?

WebCT Mail

WebCT Groups

Version 3.8 now addresses this issue

30

Heterogeneous Systems

Different systems want different formats

Plagiarism Detection Systems– Moss has been used most extensively– Began experimenting with JPlag more recently

Future work: Automatic program compiling and testing

31

Automated Plagiarism Detection

Digital formats make "borrowing" easy

– Browsing similar works needs a simple and quick user interface.

– Careful review by faculty to assess results and present to students

32

Moss

Processes C, C++, Java, ML, Lisp, Scheme, Pascal, and Ada programs.

Common code feature reduces false positives

33

Moss Interface

34

JPlag

Processes C, C++, Scheme, and Java programs

For plain text files, it matches a user specified number of words appearing in succession

Could be used for any course grading written (text) documents

35

Command Line Invocation

36

Interactive

37

Quiz Download with Config File

38

Roadmap

Introduction

Problems and Solution Goals

Labrador

Discussion

39

Recent WebCT Enhancements

Relevant to this talk:

– 3.7 Addressed bulk download issue for assignments

– 3.8 Attaching documents to an assignment

40

Future WebCT Enhancements

Power users will need functionality not yet supported

Every domain will also require additional functionality

Not feasible for all domain-specific functionality to run on the WebCT server

41

Wanted

API for non-administrators

Stateful protocol so clients can be built by 3rd parties

42

HTTP: Insufficient Protocol

Inconvenient content interchange Heavy client interaction

A HTTP based approach will be sensitive to the exact location of web pages, and format of text within them.

43

Labrador Availability

Contact Us

http://duplex.mcs.drexel.edu

44

Project Support

The Pew Learning and Technology Program at the Center for Academic Transformation

National Science Foundation, Division of Undergraduate Education, DUE-#0089009

The Ramsey-McCluskey Family Foundation, Margaret Ramsey, '84

Drexel University

45

References

[1] Alex Aiken. Moss: A system for detecting software plagiarism (unpublished), http://www.cs.berkeley.edu/~aiken/moss.html.

[2] Jeffrey L Popyack, Bruce Char, Paul Zoski, Nira Herrmann, and Christopher D. Cera. Managing course management systems. In Proceedings of the thirty-third SIGCSE technical symposium on Computer Science Education, Birds-of-a-Feather Sessions, page 423, 2002.

[3] L. Prechelt, G. Malpohl, and M. Philippsen. Jplag: Finding plagiarisms among a set of programs. Technical Report 2000-1, Fakultat fur Informatik, Universitat Karlsruhe, Germany, March 2000.

[4] Larry Wall, Tom Christiansen, and Jon Orwant. Programming with Perl. O’Reilly and Associates, 3rd edition, 2000.

[5] Michael J. Wise. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the twenty-seventh SIGCSE technical symposium on Computer Science Education, pages 130–134. ACM Press, 1996.

46

Roadmap

Introduction

Problems and Solution Goals

Labrador

Discussion

Bonus Slides

47

Blooms Taxonomy

Knowledge– remembering of previously learned material; recall (facts or whole theories); bringing to mind.

Comprehension– grasping the meaning of material; interpreting (explaining or summarizing); predicting outcome and

effects (estimating future trends). Application

– ability to use learned material in a new situation; apply rules, laws, methods, theories.   Analysis

– breaking down into parts; understanding organization, clarifying, concluding. Synthesis

– ability to put parts together to form a new whole; unique communication; set of abstract relations.  Evaluation

– Ability to judge value for purpose; base on criteria; support judgment with reason. (No guessing).