Ready:Set

A Cloud-based Self-assessment Tool for Measuring Information LiteracyWesley LeonardSenior Application ProgrammerHealth Technology Group @ Central Michigan University

IntroductionAn online self-assessment tool, Ready:Set, was developed to measure information literacy and provide immediate feedback to the user. This tool made heavy use of visual analog scales or, as they are commonly known, sliders. Adaptive testing was added to the latest version of the assessment. This allowed the difficulty of questions to fluctuate based upon the users perceived (or actual) skill level. Ready:Set was deployed on the Amazon Elastic Computing Cloud. Since its original incarnation in 2003, the assessment has been completed by students from Central Michigan University and partner institutions around the world.

About meStudied computer engineering and computer science at Western Michigan UniversityParticipated in the Computer Club @ WMU and the Kalamazoo Linux Users GroupWorked for a failed dotcom company, 2000 2001Worked as a freelance programmerStarted as a student at Central Michigan University in 2003Worked as a grad assistant and was hired full-time in the new college of health professionsWas introduced to Dr. Lana Ivanitskaya and RRSACompleted M.S. in computer science in 2008

History of Ready:SetDevelopment started in 2003Original idea from 1998Perl, MySQL, IBM AIX serverDeveloped by Lana V. Ivanitskaya, Ph.D. and Anne Marie Casey, A.M.L.S. with assistance from student programmersOriginal version called RRSAResearch Readiness Self AssessmentQuestions selected for individual subject areas and designed to accurately gauge users abilitiesOriginal topic was students ability to utilize library resources and effectively find information

Subject AreasMajor topics: Library, HealthInformation LiteracyLibrary resourcesSearch enginesPeer-reviewed journals versus less trustworthy contentPlagiarismEvaluating quality of web sitesOnline information (commercial sites disguised as research)Online pharmaciesDietary supplementsMajor Versions30 different versions have been created for various groups and institutions from around the world (England, China, Brazil, Taiwan, Texas, Calgary to name a few)Questions, instructions, and feedback translated (as necessary) by volunteersQuestions customized for each institution

Feedback systemAfter completing the assessment, the user receives feedback based upon their resultsUser may also opt to receive an email with additional resources to help them improve their skillsUser may email results to themselves at any time after completing assessmentUser may not retake assessment using same PINVisual analog scalesVisual Analog Scales (VAS) or Sliders Replaced discrete questions400 distinct pointsCustomizable

Slider ConfigurationsWith/without feedbackWith/without labels (midpoint, endpoints)Custom widthsReverse scale (less is better)Custom background images and custom granularity

Slider Results

Distribution of decimal valuesUsers like Whole Numbers

Minor (cool) featuresPIN Password used to access users version of the assessmentSelf-enrollment User is given a group code which can be used to generate a unique PINFake web sites Dietary supplement, online pharmacy, and several other fake web sites were created for users to evaluate while taking the assessmentReporting Completion, placement, performance reports were created so instructors from each institution can see what their students are doingExport All user information can be exported to CSV format for analysisAdaptive testingOriginal RRSA was long (30 40 minutes)Adaptive testing was added to shorten test timeCurrently, using simple up or down algorithmQuestion difficulties determined by how often they were answered incorrectly in the pastA set of shadow questions determines initial starting difficultyIn our case, shadow questions are actually questions about the users perceived skills (the hypothesis being that perceived skills and actual skills are inversely-related)

Adaptive ExampleoutcomesSubjects showed significant improvement on all measures from pre to post when time lag is greatShort-term effect: strong change in self-perceived skills and beliefsResearch with or based on RRSA has generated 20 publicationsThis work has been cited 21 times(See http://rrsa.cmich.edu/twiki/bin/view/RRSA/Publications for a complete list)Collaboration with groups of students and faculty from around the world a truly global perspective on information literacy in the digital ageAdditional research topics: Rogue online pharmaciesRevenue generation

Part 2:The CloudHow we brought Ready:Set to the Amazon Elastic Computing CloudHow the Amazon EC2 WorksSign up for accountLaunch instancePreconfigured or create your ownConnect to instanceCreate security keys, connect as root/administrator with keyInstall and configure software as neededExecute applicationsTerminate instance when completedOur configurationSmall Linux instance (EC2)Small storage space (Volume)Simple email serviceCustom script to install necessary software and bring up Ready:Set serverGet SPECS for small Linux instance18AWS Console

AWS Console

Learn AL of these things20AWS Console

AWS Console

RedesignIn late 2009, the RRSA was to be redesigned and rebrandedPython frameworkPostgreSQL databaseRe-implement existing features but designed for the futureDecided to bring it to the clouddjangoA high-level Python Web framework that encourages rapid development and clean, pragmatic designFollows MVC paradigmDesigned to handle two challenges: the intensive deadlines of a newsroom and the stringent requirements of the experienced Web developers who wrote it. Lets you build high-performing, elegant Web applications quicklyFocuses on automating as much as possible and adhering to the DRY principlehttp://www.djangoproject.com

Startup Script# Script to start up RRSA server on Amazon AWS### Install necessary packages:apt-get install postgresql-8.3pg_createcluster 8.3 main --startapt-get install python-pygresqlapt-get install python-djangoapt-get install python-psycopg2

### Point Postgres to static volume:echo 'Now changing postgres directory... (hit ENTER to continue)'/etc/init.d/postgresql-8.3 stopreadmv /var/lib/postgresql /var/lib/postgresql.BACKUP

echo '(hit ENTER to continue)...'read

ln -s /app/postgresql /var/lib/postgresqlecho '(hit ENTER to continue)...'read/etc/init.d/postgresql-8.3 start

### Start the server in the background:python rrsa/manage.py runserver 0.0.0.0:80 2>/dev/null 1>/dev/null &

echo 'DONE'Advantages of the cloudScale quicklyHigh securityFailoverReliabilityCan dedicate resources to certain institutions if necessaryNo need to purchase or maintain infrastructureCost is comparable to utilizing CMUs infrastructureOnly pay for resources used

Amazon pricing

Other cloud providersRackspace.comGoogleVmwareMicrosoft+ many, many more!AcknowledgementsLana Ivanitskaya, Ph.D. CMU, DHA ProgramAnn Marie Casey, A.M.L.S. Formerly with CMUs Park LibraryProgrammers: Sherzod Ruzmetov, Ryan Laus, Xinxin Wu, Jeff Pipas, Weber SchulzLawrence Fulton, Ph.D. Texas State University, Computer Information SystemsPaul Albee, Ph.D. CMU, Computer Science department

Additional infoFor more information, see our wiki:http://rrsa.cmich.edu/twiki/bin/view/RRSA

We are always seeking research partners, so contact us to try out Ready:Set

Currently beta testing both cloud and adaptive versionsDiscuss details of RRSA ApplicationSegue to cloud discussionWill touch on question selection and test design in the initial (non-adaptive) version of the assessment. Technical aspects of implementing sliders and the resulting differences when asking the same question as a multiple-choice versus a slider.Technical aspects of the complete reimplementation of the original assessment with the Django framework on the Amazon EC2.Technical aspects of converting to an adaptive testing platform from a traditional static assessment. Will use screenshots from the actual assessment, graphics representing various analyses of data collected over the years, and detailed descriptions of the technology used in the assessment. Ideally, a live demo will be performed. A video demo will be created ahead of time in case the live demo isnt working.I hope the audience will appreciate the unique combination of technology, innovative testing tools, and carefully selected questions tailored to specific subject areas.

30