Lessons Learned from the Assessment Practices of a Sample of Libraries

P R E S E N T E D BY: L a u r a G i l -Tr e j o , M P H & M A

S o c i a l S c i e n c e Re s e a r c h C e n t e r

P R E S E N T E D T O : T h e S t a t e w i d e C a l i f o r n i a E l e c t r o n i c L i b r a r y C o n s o r t i u m

Demonstrating the Value of Academic Libraries: Challenges and

Opportunities

Culture of Assessment

Growing recognition of its importance in higher educationWhat is the value of education?

What is the library’s place in contributing to that value?WASC AccreditationCreation of external push for assessment observed across all campusesVariation exists in the extent to which libraries are being pushed to the same extent that academic departments are 

Culture of Assessment Within Libraries

There are diverse attitudes toward assessment within and across campusesAmong those surveyed, there is a belief that the culture of assessment has become more positive as administrators have become convinced of its importance.

Culture of Assessment

Some believe it is important because: They have to do it: it has become critical for survival or

someone is asking for the information directly They genuinely see assessment as part of ensuring

student success “…we’re very student focused. We want to ensure that

[they] are getting the educational resources they need, and that’s the only way you can do that is through some kind of assessment.”

Culture of Assessment

Those who are less enthusiastic or hesitant because they:

fear change fear what the evaluation will tell them

“…but also the people who you say the word and immediately you can see the strain on their face and the stress because they think it’s going to be some sort of negative reaction to them personally. They don’t want to be evaluated personally.”

think it is unnecessary don’t have the resource, skill, and/or time to do it

Culture of Assessment

Factors that shape attitude toward assessment Perceived meaningfulness of assessment activities Consequences for not doing assessment activities Not knowing what is meant by assessment exactly

“I don’t think anyone in the library questions the importance of [assessment]. Resistance and challenge comes from ignorance, and we don’t have a magic bullet to close the loop. It is never an easy solution—it is always complex and iterative.”

Approaches to Assessment

There is also diversity as to how assessment is approached.Those who:

Get it done Get it done, and continue look for better ways to get

it done Want to get it done, but don’t know how Are not sure it can be done rigorously so avoid it all

together.

Definitions of Assessment

Main Themes Lack of a formal definition shared by libraries Use outside entity standards to inform definitions Difference between what should be and what is

“…we want to know that the services we offer in the form of collections and our reference services and our interlibrary loan…are helping students to learn and ultimately to graduate. But that’s always been really difficult to make those connections. We end up sticking with satisfaction…we really don’t have a good assessment program.”

Discrepancy/misunderstanding between and within libraries as to what is considered assessment “Within the library there is not agreement as to what is assessment, with one

librarian saying all data collection efforts are assessment and another indicating that assessment is more confined to ‘outcomes.’”

Definition of Assessment

A tool to fit various needs Encompassing a variety of activities

“I wouldn’t tie to any one form of assessment. Our definition is broad on purpose because we do all kinds of assessment in the building, so it’s any feedback we can get for a service we’ve provided…to both meet the students’ needs and do all the stuff…as well as assessing our services. But we see assessment fulfilling both of those needs.”

Approaches to Assessment

Assessment is happening at multiple layers University Library

Some as part of university-efforts Some coordinated library-wide Some individual

The majority of libraries do not have repositories or keep track of their assessment activities on one hand, while others are publishing the results of their efforts on the other

Potential List of Services Being Assessed

The following services were identified:ServicesCollectionsE-Resources/ Digital ResourcesInformation Literacy Instruction (unit-bearing, one shot, multi-session, online tutorials, individualized)Study RoomsReference DeskInterlibrary Loan (ILL)/ Get it NowIntegrated Course DesignLaptop Loan ProgramComputers/Scanners/Printing/Wireless AccessPhysical SpaceFaculty/Instructor ServicesArchives & Special CollectionsExhibitsCommunity OutreachWebsiteAffordable Learning Initiative

Assessment Activities

Service/Resource Use

Did Service/Reso

urce Get Delivered as

Intended

Did Student Receive the Service as Intended?

What Impact Did the

Service/Resource Have on the Student Immediately

Or in the Short Term?

What Impact Did it Have

on the Student in the Long

Term?

Does Student Demonstrate

SLO

Assessment as Output

Most common Usage statistics (quantitative)

Question: To what extent are our services being used? Circulation, E-Resources/digital resources, ILL, Reference, Laptop

loan program, Study room reservation, Information literacy instruction

Mandated for several service points Critical to assessment

If done over time using the same definition and data collection systems, it can be used to establish change over time or measure the impact of an outreach program*

Introduction to rapid ILL Change in the way data is collected/use defined

Assessment as Process

Very common Question: Are our services being delivered/received as

intended? Services: IL instruction, study room, circulation, ILL, etc.

Question: Is the library as a whole or its various components giving the student what he/she achieve SLO.

Most common is the satisfaction survey (LibQual, NSSE, and other independent paper/pencil and web surveys)

Comment boxes where students can provide feedbackFocused group interviewsOther qualitative methods Samples of varying degree of representativeness from

convenience to attempts at randomly samplingBoth faculty and students included in samples

Very often non-users are not used as sample

Assessment as Outcome/Impact

Least common , but viewed as most critical Some have made steps in that direction with varying

degrees of rigor Question: Does Student X demonstrate some skill at the

end of an IL session/workshop/course? Posttest only design (without comparison group)

Rubrics applied to student samples (without control group). Typically students sampled from X number of sessions out of Y

total number of sessions Or, pre- an and posttest after online IL modules

Looking at percent correct on some measure Some efforts to incorporate comparison groups have been

made

Assessment as Outcome/Impact

Question: What impact has this service had on students? How did it impact you? If you didn’t have the library, how would your academic

experience have been altered? For the better? For the worse?o Surveyo Anecdotallyo Focused group interviews (student and faculty samples)o One on one interviews

Assessment as Outcome/Impact

Challenges experienced: Difficult in obtaining control groups Control and condition groups are typically not randomly

assigned Some tutorials require students to pass before producing a

certificate of completion Rubrics are difficult to develop and time consuming to apply to

student products Challenges finding instructors to participate For surveys, relying of self reports Does not incorporate all library services

Assessment as Impact/Output: A Simplified Model of Student Learning

Precollege Student Performance

Support for College Outside of Campus

Quality in Instruction

Engagement in High-Impact

Practices

Student Success

Where does the library fit

in?

Where Does the Library Fit In?

Student Success

Library Services and Resources

Mechanisms

Assessment as Outcome/Impact

Other potential methodology: Merging student outcome data with use data

Can factor out confounding factors Can examine moderating factors Becomes stronger when you start with a cohort and add a

longitudinal component to it (tracking students over time) Allows you to estimate the value of the library on student

outcomes. Permits for large sample sizes Avoid costs of doing surveys The down side?

Start-up is potentially TIME-CONSUMING and EXPENSIVE Relies on statistical expertise that many librarians do not

possess Safeguards must be taken to protect student confidentiality

Merging Student Outcome and Usage Statistics

Other downsides:Data collected may not gets at actually usageSometimes usage gets missed

Group study rooms Accessing databases inside the library Checking out a book

Doesn’t tell us how use is linked to student successEven if a correlation is present, it doesn’t tell us

anythning about the mechanisms.

Merging Student Outcome and Usage Statistics

Being piloted at two universities within the CSU.Already done at University of Minnesota Twin Towers

Potential problem lies with the accuracy of the data and assumptions that are made

Should validate student-use data as collected through library systems with self-reported data to make sure that it approximates reality; otherwise these data are useless

Relies on knowing which student demographic data to include

Relies on being able to bring non-users into the data file Should conduct follow up qualitative studies to focus on

mechanisms Also relies heavily on statistical expertise

Good Assessment Plans:

Leveraging what standards we do possess as a starting point (WASC, ACRL)

Leverage existing campuses resources Almost all campus respondents indicating having an office of

institutional research IT departments are always helpful Each other

Publications Best Practices/Good models (e.g., assessment team)

Must be meaningfulMust not add to existing workloadsMethodological rigor must match assessment question

Do not let a method be your master

A Tool Box Approach?

Starts with assessment question driving the effort:Who is using our services?Who is not using our services and why?How are our services being used?Are users satisfied with the services they receive? How do we best deliver services to users? Does this

differ by user?Do students have basic competency in some skill that is

required of them at the end of IL Instruction? Does exposure to our services/resources have a

measurable impact on users?

A Tool Box Approach

Selected: Does exposure to our services/resources have a measurable impact on students? Quantitative data Qualitative data Both

A Tool Box Approach

Selected: Does exposure to our services/resources have a measurable impact on students? Selected: Both quantitative and qualitative

What impact/outcome would you like to document? GPA Persistence Time to graduation Faculty scholarship Instruction of faculty Self efficacy Academic achievement (other) Etc….

A Tool Box Approach

Selected: Does exposure to our services/resources have a measurable impact on students? Type of Data Selected: Both quantitative and qualitative

Impact/outcome selected: One Year Retention Potential methods

• Quantitative: Survey : Two time points vs. one time pointCorrelating usage statistics with student outcome statistics

• Samples: First time freshmen • Qualitative

Focused group interviewsKey informant/face-to-face interviews

• Samples: First time freshmen users and non users• Other samples: Student advisors or other educators who work

directly with students.

A Tool Box Approach

For Every Method… Limitations of method How to set up your assessment

Sample survey/interview guides Different sampling approaches Examples of what data files should like Sample data analysis How to report your data

Highlighting new methods of assessing similar questions on library campuses both nationally and locally

Publications using similar methods Places to submit newly discovered methods that address similar

questions. Potentially: Online workshops or training for particular topics Potentially: a moderated community chat room Potentially: access to technical assistance

Reactions, Questions, or Ideas?

Contact Information

Laura Gil-TrejoDirector, Social Science Research CenterCalifornia State University, Fullertonlgil-trejo@exchange.fullerton.edu