Standardized Information Literacy Assessment

A Standardized Approach to Assessment:

Results from a Pilot Study of Library Instruction Sessions at San José State University

Purpose of Study

To Develop and Test an assessment tool that:

• Facilitates the development of pre and post achievement tests

• Links survey questions to learning outcomes and ACRL standards

• Delivers real time reports and statistical analysis

Timeline:An Overview

• Review of the Literature – Summer 2008

• Create assessment tool – Fall 2008

• Garner input from library faculty / Refine tool - 12/08

• Develop multiple choice questions – 12/08

• Standardize questions – 1/09

• Pilot test assessment tool – 1/09

• Enact actual study – Spring 2009

Methodology:Develop Multiple-Choice Questions

• Investigated questions from already existing tutorials / survey instruments for ideas

• Created 20 multiple-choice questions based on social sciences 100W learning outcomes

• Each question presented four answers, with only one correct answer

• Each question had an added option for “not sure”

Methodology:Standardization of Survey Multiple Choice Questions

• Multiple choice questions were emphasized to allow for immediate computation of results

• The co-occurrence of LibQual during Spring 2009 provides for complimentary qualitative data

• Questions were reviewed by library faculty to ensure they reflected learning outcomes / ACRL standards

• Questions and learning outcomes were emailed to all teaching faculty in the Psychology department for review

• Questions were reviewed by experts at the Center for Assessment under the Office of Institutional Research

• Cognitive interviews with 5 students further addressed the clarity of survey questions

Methodology:Pilot Study

Conducted with students in a PSYC 139 course to

determine:

• Average length of time to complete surveys

• Overall impact on what could be covered in a single instruction session

• Any technical glitches with survey interface, login, usability etc.

• If survey questions were clear and comprehensible to students


Conducted with students in PSYC 139 to

determine:

• Average length of time to complete surveys - 5 minutes (10 minutes total for pre and post survey completion)

• Overall impact on what could be covered in a single instruction session




Conducted with students in PSYC 139 todetermine:


• Overall impact on what could be covered in a single instruction session - no major impact with tighter planning







• Any technical glitches with survey interface, login, usability etc. - no major problems. Logins for the post test could not contain typos






• Any technical glitches with survey interface, login, usability etc. - no major problems. Logins for the post test could not contain typos

• If survey questions were clear and comprehensible to students – no confusion was reported

Methodology:Planning and Instruction

• Coordinated with instructor beforehand in encouraging students to show up early

• Scripted instructions were read to students prior to taking the pre test survey

• Lecture with embedded exercises; last 15 minutes reserved for getting started on course assignments

• Course outline and handouts were distributed to students after completion of their post test surveys

Methodology:Actual Study – Spring 2009

• Conducted with students in 5 PSYC 100W courses (83 students total)

• Survey participation was voluntary

• Participants who completed the pre-test survey but failed to complete the post-test survey were automatically eliminated from the data pool

• 11 multiple-choice survey questions covering 3 ACRL standards and 8 learning objectives

• 6 background questions on the pre-test survey – they were not repeated on the post-test

Methodology:Pre-Test Survey Background Questions

Methodology:PSYC 100W Multiple-Choice Survey Questions

Data AnalysisA Bird’s Eye View



Data AnalysisStatistical Significance

• Paired TTEST of student mean scores

• Excel formula computes statistical significance in the report generation feature of the assessment tool

Data AnalysisAssessment Tool Report Options

Data AnalysisAssessment Tool Sample Report

Data AnalysisFuture Enhancements

• Master statistical spreadsheet that incorporates all reports

• TTEST analysis of individual questions

• Multivariate statistical analysis of how background factors (class level, research experience, instruction experience) influence overall scores

Benefits

• Assessment data will inform pedagogical directions and enhance the educational impact on SJSU students

• Data can be shared with department faculty and campus administrators to demonstrate instructional efficacy

• Data can also be used to bolster documentation during the accreditation process

• Librarians can publish findings from their own sessions, particularly with a more rigorous statistical model

• Librarians can include assessment reports in their

dossiers

How Can I Get Involved

• Attend the InfoLit meeting in July regarding use of the Information Literacy Assessment Tool

• Prepare to brainstorm a process for widespread implementation

• Work with Shannon to set up an individual account for using the tool

• Start working with departmental faculty in developing learning objectives within ACRL standards

• Begin to develop survey questions that can be vetted through expert review and student feedback

Documents

Standardized Information Literacy Assessment