Content Analysis of Open-Ended Responses

LIS 665 Teaching Information Technology Literacy * Dr. Diane Nahl

University of Hawaii * LIS Program * Spring 2013

LIS 665 Teaching Information Technology Literacy

2Nahl 2013

Content Analysis Methodology

Classification Analysis of Patterns and Themes Complete document analysis Learner-generated discourse, comments, and narratives Filtering narrative responses, comments, and expressions data via

naturally occurring clusters and making category titles

Elemental Analysis of Words and Phrases Parsing responses into word, phrase, and concept elements to define

“occurrences” Word or concept frequencies Percents or proportions of expressions in categories

Achieving and Defining Agreement 95% or better needed for inter-rater reliability

Content Analysis Steps 1-7 + Step 8 (Radcliff et al. Ch 13, pp. 146-151)

LIS 665 Teaching Information Technology Literacy

3

Content Analysis of Minute Writing Data: Question 2. What active learning methods have been used in LIS 665?

1. Copy and paste all of the responses to Question 2 in the raw data document into a blank document, or save a copy of the file to work on.

2. Read through all of the responses to Question 2 without marking any text.

3. Parsing: Read through all of the responses to Question 2 a second time and use a highlight color or bolding or other marking method to Parse the names of active learning methods mentioned in each response. Mark each mention of any technique.

Nahl 2013

LIS 665 Teaching Information Technology Literacy

4

Content Analysis of Minute Writing Data: Question 2. What active learning techniques have been used in LIS 665?

4. Parsing: Read through a third time to complete marking each mention of methods. You might have to mark longer phrases if responses describe instead of name methods.

5. Filtering: Read through the marked text and determine if the methods fall into more specific categories, e.g., models are more general than techniques so could form two categories. Elements of techniques could form another category. It is up to you to decide on the meaningful clustering and naming of categories.

6. Coding: Use different highlight colors to separate parsed text into categories.

Nahl 2013

LIS 665 Teaching Information Technology Literacy

5

Content Analysis of Minute Writing Data: Question 2. What active learning techniques have been used in LIS 665?

7. Categories: Copy and paste the parsed terms into category listings on the document.

8. Inter-rater agreement: Show your categories to one other person in class. Discuss any differences you detect in your parsed terms, category groupings, and names for categories.

9. Achieving inter-rater agreement: Come to agreement on categories and groupings by standardizing a definition for what each category stands for and write the definition under each category name, above the unique parsed elements.

Nahl 2013

LIS 665 Teaching Information Technology Literacy

6

Example: Content Analysis for a Published Research Article

An online survey was sent to several librarian lists and Second Life librarian groups. The open-ended survey responses were analyzed with content analysis and published: Ashford, Robin, Beth Kraemer, Denise Cote and Diane

Nahl. 2012. Academic Librarians and Virtual Information Services in Second Life. Journal of Library Innovation (January) 3 (1): 20-47. http://www.libraryinnovation.org/article/view/87

The following slides (7-12) illustrate portions of the processes involved in analyzing the open-ended comments of librarians using Second Life in their professional endeavors.

Nahl 2013

LIS 665 Teaching Information Technology Literacy

7

Open-Ended Raw Survey Data

Nahl 2013

LIS 665 Teaching Information Technology Literacy

8

Analyzing the Results: Parsing and Coding Phrases, Inter-Rater Reliability

Nahl 2013

LIS 665 Teaching Information Technology Literacy

9

Analyzing the Results: Categorizing the Comments

1.  Successeso Collaboration & Connecting o Professional Developmento Reference & Instruction Opportunities  o Socializingo Content Creation

2.Obstacleso Technical Difficulties o Steep Learning Curveo Insufficient Value  o Unknown Application

Nahl 2013

LIS 665 Teaching Information Technology Literacy

10

Figure 3. Reported Successes  

Nahl 2013

LIS 665 Teaching Information Technology Literacy

11

Figure 4. Reported Challenges          

      

Nahl 2013

Table 1: Academic Librarians in Second Life: Five Major Role Categories and Four Areas of

Difficulty N  comments made in

each category Successes Categories %  Total %

35 Collaboration & Connecting 29 %

42 Professional Development 31 %

54 Reference & Instruction 36 %

2 Socializing   1 %

19 Content Creation 13 %

        152 TOTAL 100 % 152   77%

     Challenges Categories

  6     Technical Difficulties   13 %

12 Steep Learning Curve   26 %

          16 Insufficient Value   35 %

12 Unknown Application   26 %

46 TOTAL 100 % 46   23 %

Grand Total All Comments 198 100 %

Ashford, Robin, Beth Kraemer, Denise Cote and Diane Nahl. 2012. Academic Librarians and Virtual Information Services in Second Life. Journal of Library Innovation (January) 3 (1): 20-47. http://www.libraryinnovation.org/article/view/87

LIS 665 Teaching Information Technology Literacy

13

Next Week

Radcliff et al. Ch 14; ACRL 2003, 2007; IL IQ

Bring your content analysis results to class for an exercise in summarizing textual/comment data

Graphing numerical results Bring your numerical data to class in a spreadsheet for an

exercise in making and discussing figures and tables

Instruction Unit due April 4 Part I: Individual report on all elements Part II: All Team member names on cover page.

Includes all materials used in session, instructional sequence, keys to worksheet scoring; and all activities, worksheets, and evaluations keyed to outcomes (ACRL SPIOs and ACS codes).

Nahl 2013