37
Elise Y. Wong, Sharon Radcliff*, Gina Kessler Lee, and Suellen Cox Librarians at Saint Mary’s College of California *SMC AiA team leader April – August 2013 Measuring the effectiveness of library information literacy instruction Assessment in Action:

Measuring the effectiveness of library information literacy instruction

Embed Size (px)

DESCRIPTION

Measuring the effectiveness of library information literacy instruction / Elise Y. Wong, Sharon Radcliff, Gina Kessler Lee, and Suellen Cox, Librarians at Saint Mary’s College of California. Our project aimed to strengthen the connection between English Composition courses and library instruction sessions to achieve information literacy learning goals. To this end, SMC librarians compared two versions of library instruction in ENGL5 “Argument & Research” sections to measure the effectiveness of embedding the “information evaluation and research practices” and “critical thinking” learning outcomes into our library instruction.

Citation preview

Page 1: Measuring the effectiveness of library information literacy instruction

Elise Y. Wong, Sharon Radcliff*, Gina Kessler Lee, and Suellen Cox

Librarians at Saint Mary’s College of California

*SMC AiA team leader April – August 2013

Measuring the effectiveness of library information literacy instruction

Assessment in Action:

Page 2: Measuring the effectiveness of library information literacy instruction

Project overview

• Seeks to strengthen the connection between English Composition courses and library instruction sessions to achieve information literacy learning goals and positively impact student success

• Compares and assesses two versions of library instruction in teaching information literacy in ENGL5 “Argument & Research” sections

Page 3: Measuring the effectiveness of library information literacy instruction

Research questions

• How effective are our traditional library sessions in helping students meet their ENGL5 Information Evaluation and Research Practices (IERP) learning objectives?

• Does our “treatment” in experimental library sessions have any effect building student Critical Thinking (CT) skills?

Page 4: Measuring the effectiveness of library information literacy instruction

Essential terms defined

• Critical thinking (CT)• Information Evaluation and Research Practices

(IERP)• Toulmin’s model of argument analysis• Confirmation or myside bias• Flipped teaching/classroom• Threshold concepts in ACRL’s new framework

Page 5: Measuring the effectiveness of library information literacy instruction

Flipping the classroom

• “A form of blended learning where students learn new content, usually at home, and what used to be homework is now done in class with teachers offering guidance and interacting with students” (Branch, Dahlen, and Ransom, 2014 CCLI panel)

• “Flipped classroom” was coined by Bergmann and Sams of Colorado in 2007

• Lage, Platt, and Treglia. (2000). Inverting the Classroom: A Gateway to Creating an Inclusive Learning Environment. Journal of Economic Education, 31(1), 30-43

Page 6: Measuring the effectiveness of library information literacy instruction

Benefits and challenges of flipped teaching

• Facilitators work with students in an inquiry-based and interactive environment

• More responsive to student learning needs• Encourage independent thinking and

cultivate a deeper active learning experience• Planning for homework materials can be

time-consuming and technologically challenging

• Students don’t always come to classroom prepared

Page 7: Measuring the effectiveness of library information literacy instruction

We Learned from literature that…

Students who are taught argument schema using the Toulmin model construct better arguments with more alternative viewpoints and rebuttals.

Students who learn to critically analyze arguments and alternative viewpoints in dialogue can better avoid “myside” or confirmation bias.

Students who are also taught to generate critical thinking questions based on argument schema create better arguments.

(Nussbaum, 2007) (Wolfe, 2009)

(Song, 2012)

Page 8: Measuring the effectiveness of library information literacy instruction

Project design (Spring 2014)

Faculty survey (all ENGL 5)

3 faculty volunteers (6 ENGL 5 sections)

Project SAILS cohort test

Traditional & Experimental

library sessions

Qualitative & Quantitative assessments

Data report & discussion

Page 9: Measuring the effectiveness of library information literacy instruction

COMPOSITION learning outcomesStudents will:

•Develop search strategies and use library catalogs and databases to find relevant materials for research

IERP 1

•Practice evaluating sources critically

IERP 2

•Seek and identify confirming and opposing evidence relevant to own hypothesis

CT 2a

•Evaluate and synthesize evidence for the purpose of drawing valid conclusions

IERP 3

•Demonstrate academic honesty and safeguard the intellectual property of others by properly integrating and citing sources

IERP 4

Page 10: Measuring the effectiveness of library information literacy instruction

Traditional vs. Experimental

Focus on IERP Focus on IERP + CTLittle pre-session work More pre-session work • Developing a topic• Formulating search strategies• Using databases• Evaluating sources: credibility, reliability, accuracy, & bias• Ethical use of information; citing in MLA

• Developing a topic• Considering alt. viewpoints• Evaluating scholarly articles: claim, evidence, reasoning/ assumptions, alt. viewpoints, rebuttal, credibility, & reliability• Citing in MLA

Page 11: Measuring the effectiveness of library information literacy instruction

Poll: did flip teaching work?

• Did you do the Considering Viewpoints worksheet?

Yes (23) No (16) Not sure (7)

• Did you read over the PowerPoint? Yes, carefully (7) Yes, skimmed (22) No, not at all (17)

• Did you read the annotated article? Yes, skimmed the article and read the annotations (8) Yes, skimmed the article (23) No, not at all (14)

• 46 responses from 3 experimental sessions

Page 12: Measuring the effectiveness of library information literacy instruction

Faculty survey: extended research essay

• 15/17 responses 88%• 11 request 2-3 drafts (scaffolding) prior to

final version• 14 include student peer review sessions• 11 require students to consult and cite

specific type of sources (books, magazines, peer-reviewed articles etc.)

• All require students to demonstrate the use of multiple or competing viewpoints

Page 13: Measuring the effectiveness of library information literacy instruction

Library sessions and learning outcomes

How important is the library session on covering the following learning outcomes?• IERP 1: important to

very important (3.6)• IERP 2: important (3.4)• IERP 3: important

(3.07)• IERP 4: important

(3.07)

How effective is the library session on covering the following learning outcomes?• IERP 1: effective to

very effective (2.67)• IERP 2: effective (2.2)• IERP 3: not effective

to effective (1.8)• IERP 4: effective

(2.13)

Page 14: Measuring the effectiveness of library information literacy instruction

What is lacking and should be covered in the library session?

• More coverage of databases and relevant search techniques

• More attention on evaluating sources critically, and why we emphasize peer-reviewed academic journals

• At present, the library sessions do not cover the evaluation, synthesis, and ethical use of sources

• Add another library session if necessary• Prefer for students to have more time during

the session to explore what was covered on their own

Page 15: Measuring the effectiveness of library information literacy instruction

Project SAILS: SMC student performance

Better than the institution-type

benchmark

• Developing a research strategy• Using finding tool features

About the same as

the institution-type benchmark

• Searching• Retrieving sources• Evaluating sources• Documenting sources

Worse than the institution-type

benchmark

• Selecting finding tools• Understanding economic, legal, and social issues

Page 16: Measuring the effectiveness of library information literacy instruction

Student performance in the order from Best to Worst

• Documenting sources• Retrieving sources• Understanding economic, legal, and social issues• Using finding tool features• Searching• Evaluating sources• Selecting finding tools• Developing a research strategy

Page 17: Measuring the effectiveness of library information literacy instruction

Pre-library session

Devel

opin

g a

rese

arch

stra

tegy

Retrie

ving

sour

ces

Sele

ctin

g fin

ding

tool

s

Eval

uatin

g so

urce

s

Sear

chin

g

Under

stan

ding

eco

nom

ic, le

gal,

and

social

issu

es

Using

findi

ng to

ol fe

atur

es

Docum

entin

g so

urce

s0

100200300400500600700800

FE vs. FT

FEFT

Page 18: Measuring the effectiveness of library information literacy instruction

Pre-library session

Using

findi

ng to

ol fe

atur

es

Retrie

ving

sour

ces

Devel

opin

g a

rese

arch

stra

tegy

Sear

chin

g

Under

stan

ding

eco

nom

ic, le

gal,

and

social

issu

es

Sele

ctin

g fin

ding

tool

s

Docum

entin

g so

urce

s

Eval

uatin

g so

urce

s0

100200300400500600700

GE vs. GT

GEGT

Page 19: Measuring the effectiveness of library information literacy instruction

Pre-library session

Using

findi

ng to

ol fe

atur

es

Devel

opin

g a

rese

arch

stra

tegy

Docum

entin

g so

urce

s

Retrie

ving

sour

ces

Sear

chin

g

Sele

ctin

g fin

ding

tool

s

Under

stan

ding

eco

nom

ic, le

gal,

and

social

issu

es

Eval

uatin

g so

urce

s0

100200300400500600700

ST vs. SE

SEST

Page 20: Measuring the effectiveness of library information literacy instruction

Sources used in research papers

Popu

lar a

rticle

s fro

m th

e W

eb

Scho

larly

arti

cles f

rom

the

libra

ry

Popu

lar a

rticle

s fro

m lib

rary

Books

Autho

ritat

ive W

ebsit

es

Gener

al W

ebsit

es

Scho

larly

arti

cles f

rom

the

Web

Film

s/med

ia

Other

s fro

m th

e lib

rary

Ency

clope

dia

artic

les f

rom

the

libra

ry

Inte

rviews/s

urve

ys

Ency

clope

dia

artic

les f

rom

the

Web

0.00%5.00%

10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%

EXPTRAD

Page 21: Measuring the effectiveness of library information literacy instruction

Paraphrasing vs. Quoting

EXP TRAD0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

35.64%43.43%

64.36%(IQA

51.79%)

56.57%(IQA

55.87%)

QUOTESPARAPHRASES

Page 22: Measuring the effectiveness of library information literacy instruction

Overall learning outcomes

IERP1

IERP2

CT2a

IERP3

IERP4

2.20 2.30 2.40 2.50 2.60 2.70 2.80 2.90 3.00

2.65

2.88

2.50

2.79

2.64

2.83

2.96

2.68

2.94

2.74

TRADEXP

1: Beginning 2: Developing 3: Accomplished 4: Exemplary

Page 23: Measuring the effectiveness of library information literacy instruction

Learning outcomes by group

IERP1 IERP2 CT2a IERP3 IERP4FE 3.14 3.46 2.86 3.11 2.96FT 3.18 3.14 2.89 3.11 2.86           GE 2.23 2.45 2.14 2.36 2.27GT 2.06 2.17 1.89 2.17 2.22           SE 2.45 2.55 2.41 2.82 2.59ST 2.97 3.25 2.94 3.22 2.94

Page 24: Measuring the effectiveness of library information literacy instruction

Faculty feedback on library sessions

• “Most useful component was showing the students that research is a process and not just a quick glance at Wikipedia.”

• “What's best about the pre-session materials is that it gets their brains kicked into gear and the process begins right there. Essential!”

• “The topic development worksheet was too easy. Students still needed significant help in determining and narrowing down their research topic. I think the alternative viewpoints worksheet was helpful but compliance was poor.”

• “It is essential that students be introduced to scholarly journals and learn how to navigate them because they can be off-putting to students unused to academic research.”

• “[Students] were asked to think more critically with the counter-argument paper, and I like that and it follows the core curriculum more closely.”

Page 25: Measuring the effectiveness of library information literacy instruction

Faculty feedback on library sessions

• “In the [experimental] session, the contextualization was helpful. The students seemed more engaged…and I think they understood more of the theoretical whole of citation from that experience.”

• “I like the more extensive session—it felt more interactive and thorough. It showed them an argument in "real life“ and once they saw that I think it helped them relax a bit and not be intimidated by what up to that point might even seem abstract.”

• “The new [experimental] session is my preference, with a few tweaks to the presentation and the pre-materials.”

• “I liked both [sessions] for the emphasis on critical thinking in terms of finding key words to search, examining some of the many databases…and in finding and evaluating articles. The teach back sessions are great.”

Page 26: Measuring the effectiveness of library information literacy instruction

Faculty feedback on student papers

• “I saw measurable improvements. Both classes really did great jobs establishing the credibility of their sources as well as backing up their own arguments with the evidence they found. I do think they could have used more diverse viewpoints--that seemed to be a risk that could have been taken more.”

• “Overall they did an effective job of using their sources to support an argument they made in their own words (rather than having the sources make the argument for them).”

• “I did not see a distinguishable difference as yet. One major factor is that the two classes were not equal controls. The one place where I think the ‘new’ class did better was choosing sources.”

• “Overall I was a little disappointed with the topics chosen by students in both classes, but I think they did a good job of identifying and using sources that related to their topics and they attempted to have a variety of opinions.”

Page 27: Measuring the effectiveness of library information literacy instruction

Student reflection on research process

• “The research portion is the hardest part & once that aspect is out of the way the paper comes easy.”

• “I learned that you can't just type in every little word into every database, like you can with Google.”

• “Using library databases was more useful than Google if I want to find credible sources.”

• “I searched for the information the same way I learned last semester (in the library).”

• “Nothing new, I used the same techniques as I've always done.”

Page 28: Measuring the effectiveness of library information literacy instruction

Student reflection on finding myside bias & alternative

viewpoints• “I found good counter arguments & I discovered that I have

a bias toward my argument as well.” • “I found competing arguments, which helped me create a

stronger argument!”• “During the research I did find some articles that do

compare against my article which is good because I have other things to write about such as a rebuttal.”

• “While working on this paper I realized some of the competing viewpoints and I actually changed my topic to side with a competing view.”

• “I didn't have biases. I found possible viewpoints but I didn't specifically use any of them for resources.”

• “I realize that I haven't thought about this subject in depth, but now I am interested.”

Page 29: Measuring the effectiveness of library information literacy instruction

Student reflection on critical thinking exercises

• “The critical thinking questions were indeed helpful in looking for arguments that were being made, & alternative viewpoints in the paper itself. However, the alternative viewpoints were always more difficult.”

• “The topic development worksheet was very useful in teaching me how to find counterarguments and rebuttal paragraphs in long articles.”

• “They were somewhat useful, but I already knew some of the information before the presentation.”

• “I've been taught through several other library presentations to ask these questions of the articles/sources. It was a review.”

Page 30: Measuring the effectiveness of library information literacy instruction

Further reflection

• Did flip teaching work?• Did the Toulmin method work?• What were the challenges and what did we

learn?

Page 31: Measuring the effectiveness of library information literacy instruction

Challenges

• Lack of faculty volunteers• Lack of student incentives• # of library sessions vary• Too little time/too much materials• Some materials maybe overly complex

Page 32: Measuring the effectiveness of library information literacy instruction

Discussion

• The differentiation of library session for English 5 from that of English 4 is not all clear.

• The library session needs to align more specifically with the curriculum of English 5, and should be more integrated with the teaching of English 5

• Librarians should collaborate with faculty on assessing student skill level and any “pre-requisites” that should be covered prior to attending the library session

• How to avoid students getting repeat content in their library sessions

Page 33: Measuring the effectiveness of library information literacy instruction

Critical thinking and argument analysis

• Does the teaching of critical thinking and argument analysis have a positive impact on student performance in research writing?

• What should librarians do to ensure that the teaching of critical thinking and argument are sustainable for use in a variety of information literacy instructional settings?

Page 34: Measuring the effectiveness of library information literacy instruction

Future directions for research

• Explore collaborative opportunities available for librarians to work with faculty in other departments on assessment

• Design library instructions aligned with the threshold concepts in the new ACRL framework

• Consider using hybrid learning model to make one-shot instruction more effective

Page 35: Measuring the effectiveness of library information literacy instruction

References

• Nussbaum, E. M, & Schraw, G. (2007). Promoting argument-counterargument integration in students’ writing. The Journal of Experimental Education, 76 (1), 59-92.

• Song, Y. & Ferretti, R. (2012). Teaching critical questions about argumentation through the revising process: effects of strategy instruction on college students’ argumentative essays. Reading and Writing Quarterly, 26 (1), 67-90.

• Wolfe, C., Britt, M.A, & Butler, J. A. (2009). Argumentation schema and the myside bias in written argumentation. Written Communication, 26 (2), 183-209.

Page 36: Measuring the effectiveness of library information literacy instruction

Questions and comments?

Elise Y. Wong, team leader [email protected]

SMC AiA campus team

• Katherine Field, Composition faculty• Kathryn Koo, Composition director• Hilda Ma, Composition faculty• Gabrielle Myers, Composition faculty• Chris Procello, Director of Educational Effectiveness and

Associate Director of Institutional Research

Page 37: Measuring the effectiveness of library information literacy instruction

Acknowledgement

This project is part of the program “Assessment in Action: Academic Libraries and Student Success” which is undertaken by the Association of College and Research Libraries (ACRL) in partnership with the Association for Institutional Research and the Association of Public and Land-grant Universities. The program, a cornerstone of ACRL's Value of Academic Libraries initiative, is made possible by the Institute of Museum and Library Services.