Magia G. KrausePh.D. Candidate

School of InformationUniversity of Michigan

Society of American Archivists Annual MeetingAugust 14, 2009

Undergraduates in the Archives:

Measuring the Impact of Archival Instruction Through Rubric-based Assessment

Instruction and Assessment in Academic Archives

Krause, Magia G.(2008). Learning in the archives: A report on instructional practices, Journal of Archival Organization, 6(4) 233-268.

Overview of Teaching ExperimentResearch Questions

Do undergraduate students learn from archival instruction?

What do undergraduate students learn from archival instruction?

Goals of studyIntroduction of a reliable assessment tool to

measure student learning Introduction of archival literacy scale that

might compliment a curriculum of undergraduate research education in archives

MethodologyQuasi-experiment in the fieldBackground

US history course in a large state university82 students total

Control and treatment groupsTo compare students that did not receive

archival instruction with students that didPre- and Post- Document Analysis Exercise

To provide a baseline comparison of the groups and look for an improvement in treatment group

Archival InstructionDescription of treatment (aka archival

instruction)Lecture/presentation to entire class by

archivist Hour-long session in the archives with 4

stationsBibliographic InstructionCritical ThinkingPhotograph AnalysisFootnote/Citation Analysis

Photographs from the Instruction Session

Document Analysis ExerciseDivided into 3 sections

Adapted from NARA’s Teaching with DocumentsWritten Document AnalysisPhotograph AnalysisFinding Aid Analysis

Difference between Pre- and Post- TestsAlternative primary sources and finding aid

Assessing Student LearningRubric

Basic definitionScoring guide that includes criteria and levels of

performanceTypes of rubrics

Holistic and analyticSteps in creating a rubric

Criteria : Archival Literacy SkillsObservationInterpretation/Historical ContextEvaluation/Critical ThinkingResearch Skills

Rubric for this study

Using the Rubric3 raters

Two professional archivists with at least five years of experience teaching undergraduates and me

Important for increasing the reliability of the scores

Achieving kappa

Final resultsOur Fleiss Kappa for the pretest was .80218 and .788 for

the post-test, both of which are considered excellent strengths of agreement.

Statistical TestsNon-parametric tests

Useful for analyzing ordinal and nominal dataGreat for small samplesTests used:

Mann Whitney U Test for Independent SamplesResults

The tests confirmed that the null hypothesis could be rejected with 95% confidence

Students in the treatment group did better on the post-test after participating in archival instruction. In fact, there is a statistically significant difference between the two groups’ scores for each of the criteria.

DiscussionThe Research Skills Problem

Disappointing resultsPossible factors

Students not actively engaged during instructionFollow-up assignment involved pre-selected

documents – no chance to practice searching for and identifying documents

Literature on undergraduate information-seeking behavior

Archival Literacy ScaleElements

4 categories of the rubric:ObservationInterpretationEvaluationResearch Skills

Testing for reliabilityResults

ConclusionsLimitations to the generalizability of results

– Need to replicate the study to increase the

reliability of the instrumentsWish for better instructional design for

instruction componentNeed for more communication and

collaboration between archivists to develop and share curricula and standardized tools to assist in instruction and assessment

Thank YouFor more information about the study

described in this presentation, please contact: Magia Krause mghetu@umich.edu.