IS 4800 Empirical Research Methods for Information Science Class Notes March 21 and 23 Instructor: Prof. Carole Hafner, 446 WVH [email protected] Tel:

IS 4800 Empirical Research Methods for Information Science

Class Notes March 21 and 23

Instructor: Prof. Carole Hafner, 446 [email protected] Tel: 617-373-5116

Course Web site: www.ccs.neu.edu/course/is4800sp12/

2

Qualitative Research : General characteristics

•Purpose: to study behavior patterns not well understoodand/or too complex for quantitative methods

•Research questions can be non-specific (“panoramic view”)•Research hypotheses and procedures can evolve

•Collection of data •Only partially structured (use of open-ended observations and/or questions)•Who: individuals, groups, cultures•Where: natural setting

3

•Analysis is interpretive:• importance of experimenter’s background and possible bias• coding of data defined or enhanced during analysis stage

•Reporting of results in the form of an essay or narrative• use of quotes to illustrate and back up claims• use of little “stories” of what happened for the same reason

4

Qualitative Research: Specific qualitative methods

Ethnography – observe communities/cultureCase study – observe groups/goal orientationPhenomenology – observe individualsGrounded theory – observe groups or cultures

Relevance to IT field

5

Strategy Example(s) or categories of IT-related research

Ethnography Ethnography – Observe behavior patterns (introduce email or other apps) Ex: instant messaging among HS students. Technique: field observations/ artifacts, interviews to get background and motivations. Ex: room scheduling

Case study Case study – usually IS development/deployment. Telemedicine study: introduction of IT, how it changed the patterns of service provision. Technique: interviews (facts, motivations, explanations), artifacts.

Phenomenology

How does a new technology change employee’s work experience: study secretaries, middle managers, executives. Example: email, problem reporting, cust support. Specifics: your tasks, required skills, communication (with whom, how much, method) your perception of your job (autonomy, influence, collegiality). Technique: interviews (attitudes)

Grounded theory

Grounded theory: Create a model that categorizes college students in terms of studying/learning behavior. The procrastinator. The super-prepared. Etc. and use it to explain observed impact of distance learning Ex: stages of technology adoption. Technique: interviews, artifacts

6

Qualitative Research (cont.)Data Collection

A plan for data collection – the “heart” of the research designWho, where, what, when

Techniques for data collection:field observations/participant observationinterviewsartifacts (documents, photos, “made objects”, etc.)

Recording methodstaking notesrecording/tapingcollecting artifacts

7

Qualitative ResearchData Collection (cont.)Observational protocol:

Thinking out in advance what questions you want to answer through your field observations, interviews, or examination of artifacts.

•A “structure” for taking notes

•A “template” with various “slots” to fill in. (As in Homework assignment 1).

•For an interview: opening statement, slots for specific (demo-graphic) information, questions to ask, guidelines for follow-up

8

Qualitative Research (cont.)Data Analysis

Step 1. Prepare and organize collected data (notes, tapes, artifacts). Organization may be chronological, categorical, by source. Etc.

Step 2. Read with open mind – take notes.

Step 3. Classify into categories/taxonomies. Identify important attributes/dimensions of variation. Use a coding scheme to position elements of the data collected (answers in an interview, events observed, artifacts) within your classification scheme.

9

Qualitative Research Data Analysis (cont.)

Step 4. Identify larger themes and show how the categories of your classification scheme are related. Show how these themes provide explanations for the behavior observed.

Step 5. Summarize. Design figures, tables or other means for presenting your analytical scheme in a summary fashion.

Step 6. Conclusions. What will be the “lessons learned” or conclusions of the study.

10

Validity in Qualitative ResearchSome approaches to increasing credibility

Use data from different sources (“triangulate”) to show your analysis provides a coherent explanation

Show analysis to participants to confirm accuracy

Show results to other (impartial) researchers

Present negative or conflicting information honestly

Describe researcher’s background and possible bias

11

Qualitative Research (cont.)Writing the Proposal or Report

•Be clear about the study’s purpose, research questions, research strategy, and limitations.

•Be specific about who, what, when, where and how the data will be/were collected and analyzed.

•Be scholarly by reviewing prior work and explaining how the most relevant prior work is related to the current study.

•Be honest about problems encountered, researcher’s role, conflicting data.

•Present complex material verbally and graphically.

•Follow good technical writing style.

Qualitative Research Data collection methods

I. Observation 6 months – 1 year

II. Interviews Informal

III. Analysis of artifacts (documents, arts and crafts, tools)

13

I. Observation: Developing Behavioral Categories, i.e. “Coding”

“confused” vs.

(“clicked mouse at least 5 times on inappropriate menu” OR“gazed at interface with mouth open AND no mouse clicks or keyboard presses for 5 minutes”) AND “furrowed brows”

14

Developing Behavioral Categories

• How do you know when your definitions are good enough?

• Cohen’s Kappa measure of inter-rater agreement:

• PR(a) – PR(e) / (1 – PR(e))

• a = agreement observed

• e = agreement expected by chance

15

Refining a behavioral protocol

Draft Coding manual

Two or more observers code datafor pilot subjects

Is Kappa acceptable?

Coders reviewdifferences &update coding manual

Coding manual providesreliable measure,proceed with study

yesno

16

Coding Manual

• You should write your behavior identification rules down so that you could give them to someone else to follow reliably.

• You should also write down the sampling and coding methods you will use, as well as your recording instrument (e.g., paper form).

17

Quantifying Behavior in Observational Research

• Frequency Method– Record the frequency with which a behavior occurs within a

time period

• Duration Method– Record how long a behavior lasts

• Intervals Method– Divide the observation period into several discrete time

intervals (e.g., ten 2-minute intervals), and record whether a behavior occurs within each interval

18

Example: Code Posture Shifts

19

Coping With Complexity in Observational Research

• Event Sampling– Select one behavior for observation and record all

instances of that behavior

– It is best if one behavior can be specified as more important than others

Interviews

• Interviews– Structured, semi-structured, informal, retrospective– Informal interviews seem to be casual conversations,

reveal what people think and how one person’s perceptions compare with another’s

– Difficult to conduct ethically and productively

• Key Actor or Informant interviewing

• Specific questions: open ended or closed ended

Techniques

• Grand tour techniques – show me around

• Study of faculty attitudes: tell me about the College

• Good at exploratory stage, to identify topics for further study

The Ethnographic Interview (paper)

I. Grand Tour questionsA. General Overview Ask the informant to generalize, to discuss patterns of events.

• Could you describe a typical day on the job?

• Could you show me/tell me how you usually make a box?

B. Specific Tour Ask the informant about a specific incident or what he or she did on a certain day.

• Could you describe what happened at the management meeting yesterday, from

beginning to end?

• Tell me about the last time you used the crane.

C. Guided Tour Ask the informant for a tour of the workplace or to accompany him or her while doing a job.• Could you show me around the plant? Could I go on a sales call with you? D. Task-Related Grand Tour Ask the informant to perform a task to help you understand the context. • Could you draw a flow chart of how the aluminum moves through the plant, from raw metal to the finished product? • Could I watch you use the cutting machine and ask you questions about it afterwards?

• II. Mini-Tour Questions

• "Describe what goes on when you run the coil through the annealing machine."

• III. Example Questions

• "Can you give me an example of your supervisor giving you a hard time?"

IV. Experience Questions "Could you tell me about some experiences you've had working on the annealing machine?" V. Native-Language Questions • What do you call it when you mis-measure a piece? How do you refer to your work area?

26

III. Artifacts: Content Analysis

• Used to analyze a written or spoken record for occurrence of specific behaviors or events

• Archival sources often used as sources for data

• Response categories must be clearly defined

• A method for quantifying behavior must be defined

27

Example Study

• The CEO of Global Enterprises, Inc. is very worried about the low morale in the company, as evidenced by the amount of flame email she receives. She considers sending every office on a “ropes” course, but to do this would cost the company $10M. She asks you to do a study to tell how well her scheme might actually work in reducing her flame mail.

28

Analytic Induction (Znaniecki)Nonexperimental, Qualitative analogue to scientific method

1. Phenomenon tentatively defined2. Hypothesis is developed3. A single instance is considered to determine if

hypothesis is confirmed4. If hypothesis fails, then phenomenon or hypothesis

is redefined5. Additional cases are examined and, if the new

hypothesis is repeatedly confirmed, some degree of certainty results

6. Each negative case requires that the hypothesis be reformulated until there are no exceptions

29

Typical Use of Analytic Induction

• Say you’re interested in employee’s impressions of WizziWord.

• You interview 3 people, transcribe your notes, and categorize all important statements into themes– e.g. “It’s too slow.”, “It looks cool.”, etc.

• You interview 3 more people, categorize their comments.

• Repeat until no new (significant) categories/themes emerge.

30

Exercise

• What are your impressions of Clippy?

31

Meta-Analyses• Compare/Integrate “all” studies that have

investigated a given phenomena– E.g., use of a particular medication for a

particular disease

• Common in the literature (esp. medical)• Very methodical

– Search for articles– Eligibility criteria– Statistical analyses

Lecture 1 - Introduction 32

Ethnography

•Ethnography: The art and science of describing a group or culture.

• The observer is a human instrument

• Select a place and the people or activity

•Observe behavior and identify patterns

•A great variety of techniques to gather data

“(Participant) observation”: in natural setting

• “Participant” observation occurs when you interact casually and/or form relationships with informants

• How much you actually “participate” depends on the goals of the study.

Participant observation

• Advantages:– Offers insights into complex behavior– Identify the “right questions” for further study– Verify/correct self-reports

• Disadvantage: – Time consuming– Data collection is difficult– Problem of subjectivity

How to operationalize

• Field notes– Text– Diagrams, maps– Can result in numerical data

• Interviews (interviewer more clueful)

• Focus groups (facilitator more clueful)

What to observe

• Spatial relations

• Activities

• Communication– Verbal– Other

• Tasks– How work is allocated

Ethics

• Do not disrupt the activity your are observing

versus

• Do not mislead

• No formal rules about disclosing your role as a researcher when engaging in casual conversation – article suggests a point where you want to ask specific question

• Disclosure includes: right of refusal, confidentiality

Protecting confidentiality when data is unique

• Separate identify info from field notes entered into the computer

• People, organizations/companies, should be given fictitious names

How to be an effective observer

• Preparation

• Stay in the background

• Be factual and objective in your notes– (interpretation comes later)

• Taking notes:– Hand written usually– Type in to computer later

• EXPANDING NOTES

40

Case Study Research

Steps: design, conduct (data collection), analysis, write-up

Exploratory research and the role of prior theoryImpacts case selection, data collection Scientific method – observations should have the

potential to disconfirm

Case study research questions

Example: Telemedicine paper

Case study research

Design stepsDefine unit of analysisCase selectionCreation of a protocol (plan of work)

Data CollectionAnalysis and reporting

42

Case Study Research

I. Design steps:Select the unit(s) of analysis

(temporal, organizational, technological)Case selection (“sampling”??) – one or several

critical casetheory based (confirming or disconfirming)extreme v. typicalintensecriterion (e.g., budget > $X)convenience

43

Case Study Research – Design steps (cont.)

Use of a protocol: (the “plan of work”, should be required)

1. Overview of study, including overview of data collection strategy.

2. Details of data collection (sources, procedures)3. Interview guidelines and instruments4. Outline of the expected project report

Issues to be addressed in (2): access to the organizationresources sufficient to collect the data in the fieldscheduling of data collection activitiesproviding for unanticipated events

44

Case Study Research – Data Collection

The more different methods employed, the fuller the picture of the phenomena being studied.

-- Documents (meeting minutes, project reports, newsletters, manuals)

-- Archival documents (service records, system usage data)may provide quantitative information

-- InterviewsTypical: 95 interviews over 6 months

-- Field observations (when a visit is conducted): usually meetings. (also can observe user training, etc.)

-- Artifacts (problem reports – why not archival docs??)

45

Case Study Research – Data Collection

Selecting interviewees:a. maximum variation (preferred method)b. Homogeneousc. Snowball or chaind. Purposeful v. opportunistic

Benefits of semi-structured interviewsUnstructured when questions not known in advance

What is “triangulation”? (paper mentions construct validity)

When to STOP collecting data

46

Case Study Research – Data Analysis

Data analysis very different from analysis of experimentand survey data. Why?

Stages of analysis:Preliminary analysis (early steps)Within-case analysisCross-case analysis

Qualitative analysis most difficult and least standardizedpart of empirical research.

47


Goals of qualitative data analysis:Identify themesDevelop categoriesExplore similarities and differencesDescribe patterns that explain why(Propose models that predict)

48


Techniques for qualitative data analysis: Preliminary Stage1. Coding2. Database

What is a code? a word or short phrase attached to each segment (e.g., paragraph, answer to interview question) of the collected data, indicating the “presence” of that concept.

Codes can be arranged in (or derived from) a taxonomy.

A good taxonomy will yield codes that reveal patterns in the data.

49


Where do codes come from:Prior work or theoryStudy of initial data (defined “inductively”)Iterative nature of codingUse of independent raters to validate codes

The code book or code manual: desirable attributesDetailed description of each codeInclusion and exclusion criteriaExamples of collected data to illustrate each code

Development of higher-level “pattern codes” identify themes or relationships that are relevant to the study’s research questions

50


Case study database:•uninterpreted data•complete data (answers criticism of “selective quoting”)•analogous to raw data collected in experiment or survey

Contents of case study databaseField notes (interviews, observations)Documents (including transcripts)Quantitative data (including questionnaire data if any)Contemporaneous notes (reflective remarks)

51

Case Study Research – Data AnalysisTechniques for qualitative data analysis: within-case stageGoal: identify larger themes, relationships and propositions

Looking for larger themes and patterns:Pattern-matching

compare expected elements with actual dataperform cross-checking of interview transcripts

and other data collecteddesire two or more sources for each proposition

Explanation building:challenge tactics results approach

Use of charts, table, graphs, timelines to aid understanding

52


Techniques for qualitative data analysis: cross-case stage

Depends on availability of several cases

Two approaches:Analyze similarities and differences among cases

(e.g., factors, behaviors, results)If goal is theory-building, develop a theory using one

case and systematically compare itspropositions to other cases

53

Case Study Research – Write-up

Weakest part of the article

Goals:•Includes the goals of all professional writing, e.g., clarity, shows relationship to earlier work, data support conclusions

•For positivist case research, shows applicability (general relevance) to other examples with similar circumstances

•Constructive – propositions translate into “lessons learned” that offer guidance on how to make use of the results (do’s and don’ts)