The Price of Experience: evidence for the value of the experienced health librarian above that of the novice Andrew Booth, Reader in Evidence Based Information Practice, ScHARR, University of Sheffield.

Me and My Big Mouth!

• ABo: “As you know, Paul, I am interested in Evidence Based Library and Information Practice, “To what extent do you think that the finding that performance gets worse with duration of practice applies equally to Librarians?””

• PG: “I am not aware of any studies that look at this issue in other professions. I would imagine that someone like yourself would be best placed to look into that particular issue”

• [ABo: {Sotto voce} “Assuming, of course, that the job wouldn’t be better done by a librarian with less experience!”]

The Trade-off!

• Limitations of trading “Precise answers to the wrong questions at the expense of approximate answers to the right questions” John Tukey (1915-2000)

• What is experience?

What is experience?

• Age?

• Number of years in practice?

• Number of years post-qualification?

• Number of years in generic context (e.g. health librarianship)?

• Number of years in specific context (e.g. online searching)?

• Domain versus System

• Relationship to volume of experience (how much/how often)?

• Problem with “expertise” – comparing “experts” versus “novices” can be self-fulfilling!

• Objective evaluation versus self-assessed?

• Outcome measures – skill, knowledge etcetera?

From the Wider Literature - 1• Repertoire of 10 years experience generally

accepted to qualify person as an expert (Klein & Calderwood, 1986)

• Not only length of experience important; also quality of experience. • For fire-fighters, 10 years on-the-job experience

not as valuable for skill development as one year in decaying inner city (Klein et al. 1986)

• Mental structure of novices more chaotic and disorganized. Experts have acquired great deal of prior knowledge; more able to impose structure on content (Spires and Donley, 1998).

From the Wider Literature - 2

Lipshitz and Shaul (1997) found that:• Experts collect more information on situation before they

make a decision because they know what questions to ask.

• Experts engage in more efficient information search because they know what information to keep track of.

• Experts read situation more accurately because they are able to identify the difference between relevant and surplus information.

• Experts make fewer bad decisions related to reading situation correctly.

• Experts communicate more frequently and elaborately with friendly units.

• Experts more likely to consider other people’s perspectives when making their decisions.

From the Wider Literature - 3

• Glaser and Chi (1988) extracted list of themes commonly associated with expert performance.• Expertise is domain specific.• Experts see patterns.• Experts are faster and make fewer errors.• Experts have superior memory in their domain

(environmental cues are an aid to recall).• Experts see and represent problem at a deeper level.• Experts spend more time trying to understand

problem but have more efficient path to solving it.• Experts have strong self-monitoring skills.• Experts have refined perceptual abilities.

Focused Question

• Population: Librarians (health librarians) or non-librarians engaging in “typical” library-related activities

• Exposure: Extensive exposure (hours, years) OR arbitrary identification as “expert”. Ideally duration of exposure is principal measure (not one of many variables); other confounders controlled for.

• Comparison: Less Extensive exposure (hours, years) OR arbitrary identification as “novice”.

• Outcomes: Performance (not Attitudes) • In English, Published 1980-2010 [Only one

study excluded]

Indicative Search results• LISA (285), LISTA (370), Google Scholar

(16,900*), ERIC (76), Pubmed (17), Web of Knowledge (77)

• Started with Very Sensitive Subject Searches• (Experience* OR Expert* OR Advanced) AND

(Novice* OR Beginner* OR Newbie*) {AND Librar*}

• Follow-up of References• Email to Discussion Lists• Citation Searching

Number of Studies Retrieved• In Health Librarians 10• In Librarians (non-health) 13• Skills Commonly Practised 56

by Librarians

Sample comparisons• Libns vs Libns; Libns vs Students, Libns vs

Clinicians; Students vs Students etc

Findings – 1

• 96 studies for review• Review Articles ( 3)

• 79 studies included• Information Retrieval (67)• Teaching & Learning ( 6)• Reference/Enquiry Work ( 3)• Knowledge Management ( 2)• Marketing ( 1)

More Experience > Less Experience

Web of Science: • Experienced females best for success score/error

rates; novice males worst (Ahmed et al, 2004)

Database Searching: • Experts significantly faster than novices (Dillon &

Song, 1997). Novice librarians twice as long as experienced librarians (Fenichel, 1981)

• Librarians significantly better precision. Novice clinicians lower recall and precision than librarians/expert clinicians (Haynes et al, 1990) Clinicians recall slightly lower than for librarians, precision significantly less (Haynes et al, 1990).

• Experienced librarians statistically significant advantage in recall over physicians (Hersh et al, 1994)

More Experience > Less Experience• Clinical Experts retrieved significantly more references

than Librarians or Novices; Librarians retrieved significantly more relevant references. (McKibbon et al, 1988)

• Librarians significantly better for both recall and precision than novices (McKibbon et al, 1990). Undergraduates poorer relevance than librarians (Penhale & Taylor 1986)

• Librarians better precision than experienced end-users. (McKibbon et al, 1990).

• Use of Boolean operators [OPAC] (Dinet et al, 2004); Use of thesaurus terms (Fenichel, 1981) Greater Number of terms, Less errors in use of nonsupported operators (Lucas & Topi 2002). Experts used six times as many ORs (Rudner, 2000) 1.3 non-typographical errors/search for experienced subjects and about twice as many for novices (Fenichel, 1981) .

• Cost effectiveness: moderately ERIC-experienced searchers performed briefest, most cost effective searches (Fenichel, 1981)

More Experience > Less Experience

Web Searching:

• Task Completion Speed (Aula et al, 2006) • Information experts capture higher quality

web sites compared with doctors (Groot et al, 2001)

• Experts faster, completed more tasks, more efficient and effective. Significantly more proficient in locating Web sites (Lazonder et al 2000)

• Experts used more keywords and evaluated sites based on established criteria (wider knowledge base). Novices used “Back” key more often, used fewer numbers of search engines, and missed some highly relevant sites (Tabatabai& Luconi, 1998).

More Experience = Less Experience

Searching: • Precision of novice/experienced librarians

nearly same (Fenichel, 1981); Experienced clinicians/librarians achieved comparable recall (Haynes et al, 1990; McKibbon et al, 1990)

• For locating information on specific Web sites, performance of experienced/novices equivalent (Lazonder et al 2000)

• No significant performance differences between experience levels on any of four Web search tasks (Thatcher 2008 )

• Experts outperform novice users when system interface organized semantically, not when organized randomly (Salmeron et al 2005)

More Experience = Less Experience• Enquiries: NS Difference: Accuracy of

executives (experts)/frontline employees (novices). Executives demonstrate overconfidence in judgements; frontliners demonstrate underconfidence. (Hallin et al, 2009)

• Reference work: Students’ answers had same level of quality as responses supplied by professional, experienced librarians working on Ask The Librarian service (Sveum 2007).

• Bibliographic Instruction: NS difference in undergraduate learning gains (methods more important). (Stec 2006)

More Experience < Less Experience• Novice clinicians much higher recall

than expert searchers, paid price in precision (Hersh & Hickam, 1994)

• Skilled students more errors than novices when using physically inconsistent e-learning system (Rhee et al 2006)

• More experience results in more passive methods of curriculum implementation; less experienced librarians display more active methods (Pittman 2003)

Findings – 2• Knowledge Management

• Older, more experienced librarians higher level of collaboration, more familiar with organizations, more secure in workplace, understand importance and essence of KM and ready to share knowledge with colleagues (Aharony, nd).

• Computer Learning• Novices best with offline manual, and worst

with visual online. Experienced best with online training and worst with visual online.

• Experienced users reasonably good perceptions of what kinds of training best contribute to own learning. Novices largely erroneous in their perceptions (Hsu & Turoff 2007)

Findings – 3

How Knowledge is Organised• Experienced archivists' knowledge of institution

significant compared with novices'. Experienced archivists directly access primary resources whereas novices often needed to perform meta-searches.

• Experienced archivists’ decisions based on evaluations of quality of primary resources for form and format, comprehensiveness and accuracy of content to be as efficient as possible while still providing 'good' information. Evaluation extended to use of access tools.

• Experienced archivists used cues in environment and episodic memory of prior events to locate relevant information without referring to labels, texts, etc.: ability to intellectually and physically access information was intertwined Yakel & Antony (2006)

How good are we?• Experienced reference librarians typically

retrieve about twice as many citations as less experienced users (Hersh & Hickam, 1993).

• With minimal training, physicians could improve their performance to level of experienced searchers by their fourth online search (Haynes et al, 1993).

• “After about eight searches they [clinicians] got as many relevant citations from the literature as the experienced librarians.“ (Computers a cornerstone of evidence-based care, conference told. F Lowry - CMAJ: Canadian Medical Association Journal, 1995)

Other elements in the mix

Domain & System Interplay (Jenkins et al, 2003)

Web Novice Web Expert

Domain Expert

Searched breadth-first but evaluated information more thoroughly using subject knowledge.

Depth-first searches, following deep trails of information. Evaluated information based on most varied and sophisticated criteria.

Domain Novice

Searched breadth-first little/no evaluation of results

Mixed, breadth-first/depth-first pattern attempted to evaluate information using general criteria.

Discussion

• Performance relies on mix of Technical and Communication Skills (E.g. Reference Interview plus Online Searching)

• Perform better when we use interfaces that play to our strengths (semantic and hierarchical)

• It ain’t what you do it’s the way that you do it (e.g. instruction/curriculum development)

Interim Conclusions

• Overplaying Technical Skills (easily acquired)• Should stress superior mental models (see

“whole picture”) and communication abilities• Self awareness, evaluation• Structure can reduce differences

(checklists/metaphors/maps/models)• Experience is red herring cp.

• “Poor searchers either gave up too quickly, employed few search terms, used only simple queries, or used wrong search terms. Good searchers persisted longer, used larger, richer set of terms, constructed more complex queries, more diligent in evaluating retrieved results” (Performance!). (Sutcliffe et al, 2000)

The Way Forward

• Categorise Outcome Tools and Measures (Subjective/Objective; Surrogate Outcomes/Final Outcomes)

• Analyse Qualitative Data (The How, not just the What)

• Article in Evidence Based Library and Information Practice or HILJ?

• [References available on request]

Acknowledgements

• Dr Alison Brettle, University of Salford

• Lori Kloda, McGill University

• Kate Misso, Information Specialist, Kleijnen Systematic Reviews Ltd.

• Professor Dianne Oberg, University of Alberta