Qualitative Methods in Qualitative Methods in Assessing Libraries, Assessing Libraries,

Users and ICT Users and ICT ApplicationsApplications

Dr. R. SevukanDr. R. SevukanAssociate Professor & HeadAssociate Professor & Head

Dept. of Library & Information ScienceDept. of Library & Information SciencePondicherry UniversityPondicherry University

Puducherry, IndiaPuducherry, Indiasevukan2002@yahoo.comsevukan2002@yahoo.com

NACLIN, 9-11 December, 2014NACLIN, 9-11 December, 2014

What is an Assessment?What is an Assessment?

It is the process of gathering It is the process of gathering information from the stakeholders information from the stakeholders using various methods to using various methods to systematically gauge the systematically gauge the effectiveness of an institution.effectiveness of an institution.

Why is assessment Why is assessment important?important?

Are we delivering what we think we Are we delivering what we think we are delivering?are delivering?

Are customers getting what they are Are customers getting what they are supposed to be getting?supposed to be getting?

Are customers satisfied with what Are customers satisfied with what they are provided?they are provided?

Is there a way to deliver the Is there a way to deliver the information better, thereby information better, thereby promoting satisfaction?promoting satisfaction?

Answers to these questionsAnswers to these questions Information sources needed for research, Information sources needed for research,

teaching, and learning; teaching, and learning; Reasons and frequency of library use; Reasons and frequency of library use; Campus computer network connectivity; Campus computer network connectivity; Use of electronic resources; Use of electronic resources; Instructional needs and effectiveness; Instructional needs and effectiveness; Satisfaction on the services available; andSatisfaction on the services available; and Enhance the quality of services rendered.Enhance the quality of services rendered.

Methods of AssessmentMethods of Assessment

(Source: Niglas, 1999)(Source: Niglas, 1999)

Qualitative Vs. Qualitative Vs. QuantitativeQuantitative

Qualitative Research is primarily exploratory is primarily exploratory research used to gain an understanding of research used to gain an understanding of underlying reasons, opinions, and motivations. It underlying reasons, opinions, and motivations. It provides insights into the problem or helps to provides insights into the problem or helps to develop ideas or hypotheses for potential develop ideas or hypotheses for potential quantitative research.quantitative research.

Quantitative Research is used to quantify the is used to quantify the problem by way of generating numerical data or problem by way of generating numerical data or data that can be transformed into useable data that can be transformed into useable statistics. It is used to quantify attitudes, statistics. It is used to quantify attitudes, opinions, behaviors, and other defined variables – opinions, behaviors, and other defined variables – and generalize results from a larger sample and generalize results from a larger sample populationpopulation

Qualitative Methods: Qualitative Methods: Characteristics Characteristics

Naturalistic – Naturalistic – Investigate social processes as they unfold Investigate social processes as they unfold naturally rather than reasoning from either the conditions naturally rather than reasoning from either the conditions under which they occur or the outcomes that correlate with under which they occur or the outcomes that correlate with themthem

Emergent – Emergent – Willingness to adapt methods of analysis as Willingness to adapt methods of analysis as knowledge and understanding deepensknowledge and understanding deepens

Purposeful – Purposeful – Cases are selected because they are Cases are selected because they are “information rich”“information rich”

Source: Patton, Michael Q. Source: Patton, Michael Q. Qualitative Research and Evaluation MethodsQualitative Research and Evaluation Methods . 3. 3rdrd ed. (Thousand Oaks, CA: Sage, 2002), pp. 40- ed. (Thousand Oaks, CA: Sage, 2002), pp. 40-41. 41.

Data Collection Characteristics of Data Collection Characteristics of Qualitative MethodQualitative Method

Thick description – Thick description – The goal is deep understanding The goal is deep understanding about phenomena and experiences.about phenomena and experiences.

Personal experience and engagement – Personal experience and engagement – The The researcher is the primary instrument of data collection and researcher is the primary instrument of data collection and analysis.analysis.

Empathic stance – Empathic stance – Sensitivity, respect, awareness, Sensitivity, respect, awareness, responsiveness, and openness.responsiveness, and openness.

Attention to process – Attention to process – Situational and contextual Situational and contextual changes occur throughout the life of the study.changes occur throughout the life of the study.

Analytical Characteristics of Analytical Characteristics of Qualitative MethodQualitative Method

Unique case orientation – Unique case orientation – Every case is special and Every case is special and uniqueunique

Inductive and creative – Inductive and creative – Research process guided by Research process guided by analytical principles rather than rulesanalytical principles rather than rules

Holistic perspective – Holistic perspective – Phenomena under study are Phenomena under study are complex and cannot be reduced to a few variables complex and cannot be reduced to a few variables

Context sensitivity – Context sensitivity – Avoid broad generalizations; Avoid broad generalizations; investigation does not rest in isolation from the larger investigation does not rest in isolation from the larger social/ cultural/ economic/political environmentsocial/ cultural/ economic/political environment

Reflexivity – Reflexivity – Critical self-awareness of one’s own Critical self-awareness of one’s own presence in the research processpresence in the research process

Selective Applications inSelective Applications inLibrary SettingsLibrary Settings

To assess individualized outcomesTo assess individualized outcomes To explore internal dynamics of programs and servicesTo explore internal dynamics of programs and services To evaluate critical cases of program or service utilizationTo evaluate critical cases of program or service utilization To acquire details about program implementationTo acquire details about program implementation To obtain information about the nuances of program or service To obtain information about the nuances of program or service

qualityquality To personalize the evaluation process and/or outcomes To personalize the evaluation process and/or outcomes

assessmentassessment To discover effects of specific programs or servicesTo discover effects of specific programs or services To add depth, detail, and meaning to statistical dataTo add depth, detail, and meaning to statistical data To provide privileged access to processes, causes, and effects.To provide privileged access to processes, causes, and effects.

Source: Patton, Michael Q. Source: Patton, Michael Q. How to Use Qualitative Methods in EvaluationHow to Use Qualitative Methods in Evaluation. 1. 1stst ed. (Thousand Oaks, CA: Sage, 1987), pp. 41-42. ed. (Thousand Oaks, CA: Sage, 1987), pp. 41-42.

Some Techniques for Collecting Some Techniques for Collecting Assessment DataAssessment Data

In-Depth InterviewsIn-Depth Interviews – – speaking with and listening to speaking with and listening to our usersour users

ObservationObservation – – witness what is going on in a particular witness what is going on in a particular setting and explore the experiences of otherssetting and explore the experiences of others

Qualitative Case StudiesQualitative Case Studies – – holistic understanding of holistic understanding of social phenomena in library settingssocial phenomena in library settings

In-Depth Interviews In-Depth Interviews Outcomes for AssessmentOutcomes for Assessment

What is meaningful or important in the respondent’s What is meaningful or important in the respondent’s own voice?own voice?

Researcher’s knowledge, expertise, and interpersonal Researcher’s knowledge, expertise, and interpersonal skills can be used to explore interesting or unexpected skills can be used to explore interesting or unexpected concepts or issues raised by respondentsconcepts or issues raised by respondents

Opportunity for deep probing of issuesOpportunity for deep probing of issues

Source: Patton, Michael Q. Source: Patton, Michael Q. Qualitative Research and Evaluation MethodsQualitative Research and Evaluation Methods . 3. 3rdrd ed. (Thousand Oaks, CA: ed. (Thousand Oaks, CA: Sage, 2002)Sage, 2002)

Observation: Outcomes for Observation: Outcomes for AssessmentAssessment

Can facilitate multidimensional and long-term Can facilitate multidimensional and long-term imbeddedness in the “natural setting”imbeddedness in the “natural setting”

Melds together aspects of looking and listening Melds together aspects of looking and listening and watching and asking (connect what people and watching and asking (connect what people are saying to what they are actually doing)are saying to what they are actually doing)

Unmasks everyday interaction between people Unmasks everyday interaction between people and organizations; reveals the mundaneand organizations; reveals the mundane

Cases can be revisited repeatedlyCases can be revisited repeatedly

Case Studies Design ConsiderationsCase Studies Design Considerations IndividualsIndividuals – comparative sample of individual experiences – comparative sample of individual experiences

Groups – Groups – sample of individuals who share a common identitysample of individuals who share a common identity

Communities – Communities – sample of individuals with ethnic, cultural, sample of individuals with ethnic, cultural, spiritual or other types of characteristics in commonspiritual or other types of characteristics in common

Programs – Programs – comparative sample of projects or services comparative sample of projects or services intended to meet a public needintended to meet a public need

Policies – Policies – comparative sample of plans of actioncomparative sample of plans of action

Organizations – Organizations – comparative sample of a formal group of comparative sample of a formal group of people with one or more shared goalspeople with one or more shared goals

Qualitative Case StudiesQualitative Case StudiesApplications for AssessmentApplications for Assessment

IllustrativeIllustrative: Descriptive and intended to add “realism” and in-depth : Descriptive and intended to add “realism” and in-depth examples to other information about a program or policy.examples to other information about a program or policy.

ExploratoryExploratory: Also descriptive, but aimed at generating hypotheses for : Also descriptive, but aimed at generating hypotheses for later investigation rather than for purposes of illustrating.later investigation rather than for purposes of illustrating.

Deviant/Critical InstanceDeviant/Critical Instance: Examines a single instance of unique interest : Examines a single instance of unique interest or serves as a critical test of an assertion about a program, problem, or or serves as a critical test of an assertion about a program, problem, or strategy.strategy.

Program Implementation:Program Implementation: Investigates operations, often at several sites, Investigates operations, often at several sites, and with a subjective, value laden, and/or affective emphasis.and with a subjective, value laden, and/or affective emphasis.

Program Effects:Program Effects: This application uses the case study to examine This application uses the case study to examine causality and usually involves multi-site, multi-method assessments.causality and usually involves multi-site, multi-method assessments.

Cumulative:Cumulative: This brings together findings from many case studies to This brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause-answer an evaluation question, whether descriptive, normative, or cause-and-effect.and-effect.

Assessment Tools: A Assessment Tools: A GlanceGlance

The Challenge ofThe Challenge ofAssessment in LibrariesAssessment in Libraries

Traditional statistics emphasize inputs, Traditional statistics emphasize inputs, expenditures, acquisitions, holdings, etc.expenditures, acquisitions, holdings, etc.

Helping funding agencies understand success of Helping funding agencies understand success of investmentinvestment

No demonstrable relationship between No demonstrable relationship between expenditures and service qualityexpenditures and service quality

Lack of metrics describing outcomes: success Lack of metrics describing outcomes: success from the user’s point of viewfrom the user’s point of view

Need to redesign library services to better meet Need to redesign library services to better meet changing patterns of usechanging patterns of use

Building the climate, tools, and skill set for library Building the climate, tools, and skill set for library assessmentassessment

No Single Method!!!No Single Method!!!

““The difficulty lies in trying to find a single model or The difficulty lies in trying to find a single model or set of simple indicators that can be used by set of simple indicators that can be used by different institutions, and that will compare different institutions, and that will compare something across large groups that is by definition something across large groups that is by definition only locally applicable – i.e., how well a library only locally applicable – i.e., how well a library meets the needs of its institution. Librarians have meets the needs of its institution. Librarians have either made do with oversimplified national data or either made do with oversimplified national data or have undertaken customized local evaluations of have undertaken customized local evaluations of effectiveness, but there has not been devised an effectiveness, but there has not been devised an effective way to link the two.” effective way to link the two.”

Sarah Pritchard, Library Trends, 1996 Sarah Pritchard, Library Trends, 1996

Successful Model

“Different is not always better but better is

always different”

““Let us replicate but not duplicate Let us replicate but not duplicate research”research”

--- H.K.Kaul--- H.K.Kaul