Topic Knowledge and Online Catalog Search Formulation

Topic Knowledge and Online Catalog Search FormulationAuthor(s): Bryce AllenSource: The Library Quarterly, Vol. 61, No. 2 (Apr., 1991), pp. 188-213Published by: The University of Chicago PressStable URL: http://www.jstor.org/stable/4308578 .

Accessed: 26/06/2014 06:33

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

The University of Chicago Press is collaborating with JSTOR to digitize, preserve and extend access to TheLibrary Quarterly.

http://www.jstor.org

This content downloaded from 130.217.227.3 on Thu, 26 Jun 2014 06:33:48 AMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=ucpress

http://www.jstor.org/stable/4308578?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


TOPIC KNOWLEDGE AND ONLINE CATALOG SEARCH FORMULATION'

Bryce Allen2

This research investigated the ways in which different levels of knowledge about a topic can affect searching for information on that topic in a library online catalog. It was found that people with high levels of knowledge use more search expressions, including more general and nonproductive expressions, than low- knowledge users. It was also found that high-knowledge users employed more search expressions that had not been contained in their statements of information need than low-knowledge users. These differences in vocabulary use and search expression formulation may be of interest to designers of online catalogs as they attempt to increase the responsiveness of catalog systems to the needs of individual users.

Purpose

The purpose of this research was to examine the effects of factual knowledge of a topic on one aspect of how individuals search for information in an online catalog: their use of vocabulary in search expression formulation. It explored the idea that people who know much about a topic may search for information differently than people who know little about the topic. Although this idea seems intuitively obvious, it has not been systematically investigated by library and information science researchers. This research focused on an analysis of the search expressions of users with different levels of factual knowledge of a search topic in their use of an online public access catalog (OPAC).

The underlying assumption of this and similar investigations of user characteristics is that "all users of information systems are not created

1. This research was supported by a grant from the University of Illinois Research Board. The assistance of Mr. Scott Drone-Silvers is gratefully acknowledged.

2. Graduate School of Library and Information Science, University of Illinois at Urbana- Champaign, 410 David Kinley Hall, 1407 W. Gregory Dr., Urbana, Illinois 61801-3680.

[Library Quarterly, vol. 61, no. 2, pp. 188-213]

? 1991 by The University of Chicago. All rights reserved.

0024-25 19/91/6102-0002$0 1 .00

188



TOPIC KNOWLEDGE AND CATALOG SEARCH 189

equal" [1]. A parallel assumption is that by knowing more about how different kinds of users interact with information systems, designers can create improved systems that are tailored to the particular needs of groups of users. One label that has been attached to this assumption is "cognitive modeling": the idea that systems can use models of users' cognitive processes to meet their information needs more effectively [2].

Specifically, this research was based on the assumption that those who design online public access catalogs can benefit from knowledge about how different types of users search existing catalogs. This understanding of user behavior can influence the search mechanisms and user interfaces that are implemented in OPACs. Research such as this is important to assist in the attempt to increase the effectiveness of major access tools for library collections.

There are a number of reasons to suspect that users' knowledge of a topic may affect the way that they formulate search expressions during an online public access catalog search.

Vocabulary Differences In a technical subject area, knowledge of a topic may entail familiarity with a specialized vocabulary that can be used in searching for information on the topic. An understanding of the specialized vocabulary of the topic may enable high-knowledge users to navigate through the syndetic structures of thesauri and to use appropriate terms in keyword searches. A low-knowledge user may have difficulty expressing the information need clearly or precisely because of a lack of familiarity with the technical vocabulary of the subject area and may be less confident about mov- ing to alternative expressions of the information need using either controlled or uncontrolled vocabulary. As a result, it would be expected that high-knowledge users would be able to express their information needs using specialized vocabulary and to use alternative terms if necessary to enhance their searches. These anticipated differences in search vocabulary may have an effect on the quality of retrieval. For example, use of precise technical vocabulary has been identified as a precision- enhancing technique, while the use of alternative expressions of a topic is recognized as a recall-enhancing technique.

Cognitive Differences Another difference between high-knowledge and low-knowledge individuals has been demonstrated in psycholinguistic research. Spilich, Ves- onder, Chiese, and Voss [3] and Chiese, Spilich, and Voss [4] have shown that knowledge of a topic affects individuals' comprehension and recall of narratives. People with much knowledge about the topic of a narrative understood and recalled that narrative better than people with



190 THE LIBRARY QUARTERLY

low knowledge of the topic. Voss, Vesonder, and Spilich [5] extended this research, demonstrating that topic knowledge also influences text generation. People with high knowledge were better able to express themselves in writing about a topic than people with low knowledge. Fincher-Kiefer, Post, Greene, and Voss [6] suggested that this effect of topic knowledge on understanding, recall, and generation of text is associated with mental models that provide structures through which people understand the topic. In other words, the topic knowledge is structured in the mind into a model of the topic, and this model influences how people comprehend, recall, and generate narratives on the topic. Recht and Leslie [7] found that topic knowledge affects comprehension and recall regardless of people's reading skills.

In summary, high-knowledge readers appear to have a well- developed mental model of the topic. Using this mental model, these readers construct an integrated understanding of the topic by associat- ing the new information provided by a text with existing knowledge. Low-knowledge readers are less able to integrate the topic into their existing knowledge and, consequently, are less able to understand the new information.

These experimental results may be applicable to understanding how individuals express their information needs when interacting with an online catalog. For example, their mental models may suggest alternative approaches to materials on the topic. High-knowledge searchers might approach an interdisciplinary topic like bioethics through search expressions drawn from philosophy, medicine, and law, while a low- knowledge searcher's mental model of the topic would be less likely to provide these interconnections. In this scenario, a high-knowledge user would achieve higher recall than a low-knowledge user.

It should be emphasized that the mental models discussed in this section are different from those investigated by Borgman [8]. Her research was concerned with the mental models that users build of the information system, and how these models may affect retrieval, rather than with the mental model of the topic that high-knowledge users may employ in vocabulary selection.

Previous Research Despite these reasons for believing that level of topic knowledge may affect how users formulate search expressions, previous research has not been able to discover differences in search expressions, or in the resulting information retrieval, that can be attributed to levels of topic knowledge. In part, this lack of evidence from previous research may relate to difficulties associated with measuring topic knowledge. Some investigations have included level of education of users as a variable. For




example, Thorngate and Hotta [9] investigated the ability of professors, graduate students, and undergraduates to replicate PsycINFO descrip- tors in their descriptions of the topics of articles. Although faculty and graduate students were able to generate more matches than undergraduates, it is not clear whether this ability related to factual knowledge of the topic of the articles, general familiarity with the field, or familiarity with information tools of the field. Similarly, Blair and Maron [10] compared the retrieval effectiveness of lawyers and paralegals in searching a large full-text database. Some individual differences were noted, but no systematic difference that could be attributed to topic knowledge was reported.

In important investigations of the knowledge used by professional searchers of bibliographic databases there has been no detailed investigation of topic knowledge. For example, Fenichel, in her dissertation research [11], collected data on the formal training in education (the subject area of the searches) of participants. These data were, however, not used in the analysis of search results. Instead, Fenichel's research concentrated on the familiarity of searchers with the database as a deter- minant of search effectiveness. Similarly, Bellardo's dissertation [12] examined character traits of searchers, and Howard's research [13] focused on training and experience in online searching. In summary, the research into the ways professional searchers differ has not focused on differences in topic knowledge.

Interestingly, one study that investigated users' level of topic knowledge focused on the possible effect of this knowledge on searches performed by professional searchers. Saracevic and Kantor [14, 15] asked users about their "internal knowledge": the amount of knowledge they felt they possessed in relation to the problem being searched. Levels of topic knowledge reported by users had no impact on the quality of the searches performed. It is possible that a greater impact would have been noted if the searches had been conducted by the users rather than by intermediaries.

With the increased emphasis in recent years on end-user searching of bibliographic databases, and the resulting voluminous literature, one would have expected some research on topic knowledge as a determi- nant of search effectiveness to have been reported. But this does not appear to be the case. The major emphasis of research on end-user searching has been on user knowledge of the search system, to the exclu- sion of any other type of knowledge. Neither the bibliographies by Siito- nen [16] and Lyon [17] nor the review by Mischo and Lee [18] contain references to investigations of topic knowledge.

Studies of online public access catalogs and their use, such as the seminal study funded by the Council on Library Resources [19], have




not analyzed whether the level of user knowledge is a variable affecting catalog use. A tantalizing question appears in the questionnaire used by Jones in his comparison of two different public access catalog systems: "How much do you know about what you are going to look up?" [20]. The tantalizing nature of this question relates to the fact that patron responses to the question are neither reported nor analyzed in the report of the research.

Bates's dissertation research [211 investigated both familiarity with the catalog and familiarity with the subject area in the ability of people to create subject headings that matched those used in card catalogs. She found that education in the subject area had no significant effect on the ability of participants to match library-generated access points. It should be noted that this research examined general domain knowledge rather than specific topic knowledge and that the participants were not actually doing a subject search in a library catalog. As a result, it serves as a starting point for the research reported here, in which specific factual knowledge of the topic was investigated as a factor influencing the generation of search expressions in an actual catalog search.

In summary, existing research provides a general background for this study of the effects of topic knowledge on search formulation. Although concern about levels of user knowledge of the search topic is evident in this research, there appears to have been no detailed investigation of this relationship.

Methodology

Participants were placed in simulated information retrieval tasks designed to elicit expressions of their information need. The first research task, responding to an open question on a simulated pre-online search request form, provided relatively few constraints on user response. This task was not limited by retrieval technology, although it may have been influenced by users' mental models of that technology. It was anticipated that this task would be particularly useful in determining vocabulary differences between high-knowledge and low-knowledge participants.

The second research task, the online catalog search, provided more constraints on the expression of information needs. There were, however, a variety of subject searching capabilities in the system that enabled users to try different search expressions. It was anticipated that this task would highlight the extent to which topic knowledge is used in searching a real information system.

Sixty students, evenly divided between those with a high level of topic knowledge and those with a low level, participated in the research. They were paid $5 for their participation, which lasted about an hour.




Selection of Search Topic and Participants In any population, it can be assumed that the majority lack specialized domain knowledge about any particular technical topic. One of the chal- lenges facing this research was to discover a pool of potentially high- knowledge participants who would be representative of the general population of library users in other ways. Since my previous research [22] indicated that academic background can have an effect on the expression of information needs, it was important to select a topic area in which all the people with high knowledge would not be from a single academic discipline. Means of controlling for other variables that might influence vocabulary selection are discussed below.

A suitable topic emerged in the 1989 press coverage of the Voyager 2 flyby of Neptune. A short article from Time detailing the recent history of planetary exploration in the context of Voyager 2's encounter with Neptune was chosen to represent that topic. It should be emphasized that the entire research centered on this single topic and that the results cannot be generalized to other topics.

On the campus of the University of Illinois at Urbana-Champaign there are three student organizations whose members would seem to be interested in the topic of planetary exploration: the American Institute of Aeronautics and Astronautics, the Astronomical Society at the Uni- versity of Illinois, and the Illini Space Development Society (a branch of the National Space Society). With the cooperation of these organizations, membership lists were obtained that contained the names of 220 individuals who might have a higher level of knowledge about this topic than individuals in the general student population. An examination of this pool of potential participants in the research showed that about 70 percent were enrolled in one of the engineering disciplines. Accord- ingly, it was decided to use a stratified sample that would produce an equal number of participants from engineering and nonengineering fields of study. Candidates were drawn at random from these membership lists until thirty participants agreed to take part in the research, evenly divided between engineering and other disciplines. Another group of candidates were drawn at random from the university student directory until thirty participants from the general student population agreed to take part in the research, divided between the engineering and nonengineering disciplines in the same proportion as the participants selected from the student membership directories.

Participants in the engineering group came from seven different departments, ranging from aeronautical engineering to industrial engineering. Participants in the nonengineering group represented fifteen different academic departments, from astronomy to psychology.

Participants completed a short (twelve-question) multiple-choice test to establish their level of factual knowledge of planetary exploration.




TABLE 1 TEST SCORES FOR LEVELS OF REPORTED FAMILIARITY WITH ToPic

Familiarity Number of with Topic Mean Test Score Standard Deviation Participants

Very familiar 10 1.9 16 Familiar 7.1 2.1 26 Unfamiliar 5.7 2.1 10 Very unfamiliar 3.6 1.3 8

Test questions were developed by two graduate assistants who indepen- dently compiled factual questions about the solar system and its exploration, using general reference sources to verify correct answers and to suggest alternative answers. These questions were then amalgamated into a single test that was pretested with both high-knowledge and low- knowledge individuals. The pretest enabled the questions that appeared to distinguish between high-knowledge and low-knowledge individuals to be selected. Examples of the selected questions are: (1) The first U.S. satellite probe to reach the Jovian planets was (Voyager 1, Mariner 10, Pioneer 10, Surveyor 7), and (2) Saturn's largest moon (Phoebe, Gany- mede, Atlas, Titan) is larger than Mercury.

To provide a measure of the validity of this test, all participants in the research were asked how familiar they were with the topic of the paper that they read. There was a fairly high correlation between scores on the test and self-reported level of familiarity with the topic (Spear- man's r [N = 60] = -.7197, t[58] = 7.894, P < .0001). The correlation coefficient is negative because the responses of "very familiar" were coded as 1, "familiar" as 2, and so on, while a test score of 1 indicated low familiarity with the topic. Table 1 contains the mean test scores for each level of reported familiarity with the topic. The median score on the multiple-choice test was 7.5. Thirty participants answered seven or fewer questions correctly, including six participants drawn from the membership directories of the student organizations, while the remaining thirty answered eight or more questions correctly, including eleven participants drawn from the general student directory. The de- mographics of these two groups are illustrated in table 2.

It was not possible to measure the motivation of the participants in the research, just as it is not possible to measure the motivation of library catalog users in general. It is possible that the high-knowledge participants, because of their knowledge, may have been motivated to do a different kind of search than the low-knowledge participants. There may have been intrinsic rewards for them, in addition to the extrinsic reward of the money that was paid to all participants. If the motivation




TABLE 2 DISTRIBUTION OF PARTICIPANTS IN THE RESEARCH

Characteristics Test Scores in Top 50% Test Scores in Bottom 50%

Area of study: Engineering 16 14 Other 14 16

Academic level: Freshmen 7 6 Sophomores 10 5 Juniors 2 4 Seniors 4 10 Graduate students 7 5

Gender: Male 25 17 Female 5 13

of high-knowledge participants was different, this fact would not invali- date the results of the research but would provide an additional explana- tion of those results.

Other Kinds of Knowledge It is clear from previous research that many kinds of knowledge (in addition to topic knowledge) affect how people perform information retrieval tasks. For example, knowledge of libraries in general, of the online catalog, of Library of Congress Subject Headings, of online searching, or of computers in general might affect the vocabulary used in searching the catalog. It was not possible to control all of these variables by balancing the numbers of participants (as was done in the case of academic departments). Rather, it was necessary to collect data that would enable the effects of differences in these kinds of knowledge to be controlled in analysis. Controlling for all of these knowledge types individually would have required an unacceptably large increase in the sample size. Accordingly, it was decided to use an "omnibus" means of controlling for knowledge-related effects. The effects of all of these kinds of knowledge can be seen in the difficulty experienced in doing the search: for example, unfamiliarity with the online catalog, or with the controlled vocabulary used for most subject searching, tends to make catalog searches more difficult. It follows that difficulty experienced during the search can act as a surrogate for the knowledge variables identified above.

In order to collect data on difficulty experienced during the search, and indirectly on the levels of various kinds of nontopic knowledge, participants were asked "How difficult was the library catalog search for




TABLE 3 DIFFICULTY WITH SEARCH REPORTED BY PARTICIPANTS

Number of Standard Difficulty Participants Mean Test Score Deviation

Very difficult 4 7.7 3.4 Difficult 21 7.7 2.8 Easy 31 7.0 2.7 Very easy 4 5.2 3.2

you?" Table 3 presents the responses to this question and the average test scores for each group of respondents. There was no significant difference among the test scores achieved by the four groups of respondents.

The Research Tasks Participants completed the research tasks individually, in a neutral (laboratory) setting. This was felt to be superior to the more ecologically valid library setting, in which there would have been numerous distractions and interruptions. Completing the research tasks in the laboratory may also have had the effect of neutralizing the influence of their surround- ings on the participants, since all participants were equally unfamiliar with them.

The researcher or an assistant was available to supervise each participant's completion of the research tasks and to answer questions. A policy was established that researchers would not respond to questions about the topic, or about the research tasks, but would respond to technical questions about the functioning of the online catalog.

Step 1.-All participants read the short Time article, which was tran- scribed into page format without illustrations. Reading instructions were: "Read the following article carefully. You may underline or make comments on the article if you wish. When you finish reading the article, do not turn back to reread, but proceed directly to the next page."

Step 2.-All participants completed an unrelated interpolated task of arithmetical calculations following the reading of the article. This task was designed to prevent mental rehearsal of article contents and to ensure that long-term memory for article contents was being used in the subsequent retrieval tasks. To minimize the potentially disruptive effects of anxiety about arithmetic, participants were assured that the results of their calculations were not going to be graded and were simply to provide a break between the reading task and the subsequent task.




Step 3. -Following this interpolated task, all participants were instructed as follows:

A few minutes ago you read a short article on a topic. Suppose that you are an author, and that you have been given the assignment of writing an article on this same topic for the Daily Illini. Although the article will be general rather than technical in approach, you want to make sure that it is accurate with regard to technical details, names, dates and so on.

To obtain the details you need to write this article, you want to read books and articles on the topic. You will look for books in the University Library's catalog, and to ensure that you get a complete listing of magazine and journal articles you will ask a librarian to do a search for materials on the topic. Please fill out the following form to request the library search for magazine and journal articles.

Request Form-Library Search

In the space provided below, describe the topic that you want to have searched. Include all details that you think would enable a librarian to complete a comprehensive literature search on this topic.

All participants had approximately the same amount of space (1.5 pages) in which to write their responses to this open question. It should be noted that all participants, regardless of knowledge level, were asked to assume the same role: that of an author writing for a campus newspa- per. This was done to ensure that all participants would be conducting the same type of information search. It seemed likely that high- knowledge users would normally tend to look for different kinds of information than those with little knowledge about the topic. In this research, the context of the search was specified so that the type of search conducted was kept as constant as possible. It is possible that specifying this context could have introduced an element of bias. Jour- nalism students, for example, are likely to view the task of writing a news story differently from other students. Fortunately, there were no journalism students in the sample, and there is no evidence that there was any systematic difference between groups of participants as to their view of the research task.

My dissertation [23] showed that users in a simulated information retrieval task of this sort can be influenced by the type of question asked on the online search form to include different levels of detail in their responses. The open question used in this research was chosen to minimize the effect of the structure provided by the question on responses of the participants.

Step 4.-After responding to this open question, all participants completed an online catalog search on the topic, using the online public access catalog of the University of Illinois at Urbana-Champaign in a




laboratory setting. Subjects were instructed: "Now you will do a library catalog search, using the University Library's online catalog (LCS/FBR), in order to find books that have been written on this topic. Please do this search exactly as if you were in the library."

This catalog uses an interface that guides users through their searches. The interface acts as a unified approach to a circulation system containing brief records of all materials in the library (LCS) and to a MARC-based system containing records for books cataloged since 1975 (FBR). The interface also performs a number of search functions automatically, and transparently to the user. Although a complete description of the catalog is out of place here, the subject searching capabilities are relevant to the assumptions of the research. Search expressions that use the Library of Congress Subject Headings (LCSH) are encouraged (a copy of LCSH was made available to all participants, and it was used by twenty of the sixty participants). These expressions are searched in the subject authority file, first as a string, then (if no citations are found) as a truncated string, then (if no citations are found) as a starting point for an alphabetic display of headings from the authority file. Subsequent steps may include a keyword search of the expression against subject fields and title fields. Pseudo-subject searches using keyword access to the title field are also possible and, in fact, proceed automatically if a title string search fails to produce citations. This sequence of approaches provides a thorough subject search, but it is somewhat inflexible, particularly in its preference for string searching over keywords and for authority file searching over full records.

Analysis Three main hypotheses were tested in this research. The first was that high-knowledge subjects would use more varied vocabulary than low- knowledge subjects in describing their information need. In order to test this hypothesis, written responses to the open-ended question on the simulated pre-online library search request form were subjected to lexical analysis. Common functions words were eliminated, and a list of the unique word stems used in the response was produced. Two subsets were created from this list of unique word stems: those that were also used in the Time article, and those that were not. To provide a slightly different measure of the nature of the vocabulary used in these written responses, the unique word stems from the responses were compared with the fifteen most commonly used word stems from the Time article. Again, two subsets were created: those that were found in the list of frequent word stems from the article, and those that were not.

The first hypothesis was considered to be supported if high- knowledge participants employed more unique word stems in re-




sponding to the open question (on average) than low-knowledge participants. It was also considered to be supported if high-knowledge participants employed more word stems that were not found in the Time article (on average) than low-knowledge participants.

The second hypothesis tested in this research was that high- knowledge subjects would use more search expressions in the library catalog search. To test this hypothesis, logs of the catalog searches were created and printed out for each participant. In the analysis of these logs, a "search expression" was defined as a word or phrase entered by the user to be searched. Sometimes these consisted of a single word (for example, "Voyager"), sometimes of a phrase (for example, "space flight"), and sometimes of a compound phrase consisting of a main heading and a subheading (for example, "outer space-exploration"). A search expression might also consist of a title or an author's name, although these types of searches were relatively infrequent. Twelve of the participants (20 percent) searched for one or more titles, five (8.3 percent) searched for one or more authors, and one (1.7 percent) searched for an author/title combination. The second hypothesis was held to be supported if high-knowledge participants used more search expressions (on average) than low-knowledge participants.

The third hypothesis was that high-knowledge participants would conduct more effective searches than low-knowledge participants. Ac- cordingly, a measure of search effectiveness was required. This research was not, however, an information retrieval experiment in the traditional sense. Participants were not asked to judge the relevance of the citations they retrieved. They were not instructed to conduct a comprehensive search but, rather, to "find books that have been written on this topic." As a result, recall, as the standard measure of search quality, was inappropriate. Similarly, no premium was placed on high precision. In order to test this hypothesis, an approximate measure of search effectiveness was devised. Two experienced reference librarians read the Time article and conducted independent catalog searches. The resulting bibliography contained over fifty citations to "books on the topic." Searches done by participants in the research were compared to the bibliography, and the number of citations that matched the bibliography citations were noted. The term "approximate recall" will be used to designate the ratio of the number of citations retrieved to the total number of citations in the bibliography because this measure seems related to the standard "recall" measure, and particularly to what Lancaster [24] has called "comparative recall." The third hypothesis was held to besupported if high-knowledge participants obtained higher approximate recall than low-knowledge participants.

As figure 1 indicates, the frequency distribution for the number of




20

16

14

~12

10

4-1 2 1.1 L 1j:

.-1 "151 I 111,11,, 1,, *1111

0 5 10 15 20 25 31 Number of Citations from Bibliography Retieved

FIG. 1.-Frequency distribution of citations from bibliography retrieved

citations from the bibliography that were retrieved by participants was bimodal, with one peak at 0 and another peak at 27. The peak at 0 is easily understood, but the peak at 27 may require some additional expla- nation. These were searches that employed the search expression "Proj- ect Voyager," which yielded 27 citations. It seems that participants who retrieved those citations felt that they had enough information for their purposes and discontinued their searches at that point.

Correlations between the scores achieved on the multiple-choice test and the various output variables were used to identify areas in which knowledge level had an effect on the information retrieval tasks. It should be emphasized that these correlations are the appropriate analyt- ical tool given the nature of the research. They do not, however, provide a thorough picture of the relationships among the variables.

Accordingly, whenever a significant correlation was found, subsequent analysis was conducted using analysis of variance (ANOVA), with level of knowledge and perceived search difficulty as independent variables in a 3 x 2 factorial design. The results of these analyses are provided below so that readers may more fully understand how the variables are related. They should not be taken as indicating any causal relationship; because of the correlational nature of the research such a relationship cannot be established.

Three levels of the knowledge variable were created using the following recoding: the eighteen participants who correctly answered nine




or more of the multiple choice questions were designated as high- knowledge, the eighteen participants who correctly answered five or fewer of the multiple choice questions were designated as low- knowledge, and the remaining twenty-four participants were designated as having a medium level of knowledge. Although this division was pragmatic, to give three groups of more or less equal size, there were other reasons for grouping participants in this way. The low-knowledge grouping was identical to the range of test scores achieved by participants who regarded themselves as "very unfamiliar" with the topic. The high-knowledge grouping contained the top 75 percent of the participants who regarded themselves as "very familiar" with the topic, and the top 25 percent of the participants who regarded themselves as "familiar" with the topic. Level of search difficulty, originally a four-level ordinal variable, was recoded as a binary (difficult/easy) variable.

Findings

Hypothesis 1 The first hypothesis was that high-knowledge participants would use more varied vocabulary than would low-knowledge participants in describing the topic of their information need to an intermediary. Analysis of the responses to the open-ended question on the search request form showed no significant correlations between test scores and type of vocabulary used. Frequency of use of word stems from the Time article, and of high-frequency word stems from the Time article, were the same for all levels of topic knowledge as measured by test scores. Subsequent exploratory data analysis showed no significant differences in the content of the responses based on area of study, level of study (freshman through graduate student), or gender. The first hypothesis was not supported by the results.

Hypothesis 2 The second hypothesis was that high-knowledge participants would use more search expressions in their catalog search. Analysis of the catalog searches showed a low but significant correlation between test scores and numbers of search expressions used (r[58] = .28, P < .03). The ANOVA indicated a significant effect for knowledge level, a significant effect for search difficulty, but no interaction between the two effects. Table 4 shows the results of this analysis, and figure 2 shows the average numbers of search expressions used by participants. These results sup- port the hypothesis that high-knowledge participants would employ more search expressions. They also indicate that the participants who




TABLE 4 ANOVA OF NUMBER OF SEARCH EXPRESSIONS

Effect SS df MS F P

Knowledge level 167.2755 2 83.6377 5.549 .0066 Difficulty level 154.4707 1 154.4707 10.249 .0026 Interaction 64.3113 2 32.1556 2.134 .1263 Error 813.8749 54 15.0718

reported difficulty with their searches employed more search expressions.

Subsequent analysis showed that some of the additional search expressions used by high-knowledge participants were general search expressions (for example, "Space Exploration") rather than more specific expressions (for example, "Project Voyager"). In assessing the level of specificity of search expressions, the principal criterion used was the "IS-A" relationship. The expression "Planet" was considered more gen-

12

0.~~~~~~~~~ tlo-

z

2 -

0 Low Medium High

Level of Topic Knowledge

FIG. 2.-Mean number of search expressions used by participants




TABLE 5 ANOVA OF NUMBER OF GENERAL SEARCH EXPRESSIONS USED IN SEARCHES

Effect SS df MS F P

Knowledge level 132.7150 2 66.3575 5.757 .0057

Difficulty level 123.1677 1 123.1677 10.686 .0022 Interaction 53.7653 2 26.8826 2.332 .1049 Error 622.4333 54 11.5265

eral than the expression "Saturn," because Saturn is a planet. Similarly, "Project Voyager" was considered more specific than "Space Explora- tion" because Project Voyager is an instance of space exploration. In doing this analysis, the level of specificity of "Project Voyager" and "Sa- turn" were considered as the base. Expressions were first divided into those that were of the base level of specificity, those that were more general, and those that were more specific. Two additional categories were created to encompass those search expressions that combined two levels of specificity. For example, an expression might be composed of a general main heading and a specific subheading (for example, "Space Exploration-Voyager") or of a specific heading with a general subheading (for example, "Voyager-Space Exploration"). Accordingly, the term "general search expression" designates search expressions that were more general than the base level of specificity and included no specific elements. There was a low but significant correlation between the number of these general search expressions used in searches and the test scores of participants (r[58] = .26, P < .04). Table 5 presents the ANOVA on this variable, and figure 3 shows the average numbers of general search expressions used by the groups of participants. Similar analysis by Bates showed that students (other than library science students) are "fond of broad terms" [25]. This research showed that people with high levels of topic knowledge or who express difficulty in searching are most likely to use general search expressions.

Hypothesis 3 The third hypothesis was that high-knowledge participants would obtain higher approximate recall in their searches. The results showed no correlation between test scores and approximate recall. Table 6 presents the ANOVA on this variable, and figure 4 portrays the average approximate recall for different groups of participants. It seems clear from this analysis that the quality of searches as measured by approximate recall was not affected by topic knowledge.




I -

9

8

a1

.2

Low Medium High

Levld of Topic Knowledge

FIG. 3.-Mean number of general search expressions used by participants

Other Findings Detailed examination of the search expressions revealed another area in which knowledge level influenced the retrieval process. When vocabulary used in responses to the open question about the topic and vocabulary used in searching the online catalog were compared, analysis showed a modest positive correlation between test scores and the number of search expressions constructed of vocabulary not in the responses (r[58] = .33, P < .01). Table 7 presents the results of the ANOVA on this variable, and figure 5 shows the average numbers of these search expressions containing "new" vocabulary used by the groups of partici-

TABLE 6 ANOVA OF APPROXIMATE RECALL

Effect SS df MS F P

Knowledge level .0066 2 .0033 .055 .9362 Difficulty level .1786 1 .1786 3.018 .0843 Interaction .0740 2 .0370 .625 .5438 Error 3.1952 54 .0592




0.45

0.4

0.35-

0.3-

<0.25-

0.2-

0. 15 Low Medium High


FIG. 4.-Mean approximate recall in searches

pants. The significant interaction effect indicates that participants who had a high level of topic knowledge, and expressed difficulty with the search, were most likely to introduce new vocabulary into the search.

Additional analysis was also completed on the outcomes associated with the search expressions. All search expressions entered by participants while searching the catalog were traced through the transaction logs and divided into three categories. The first category consisted of search expressions that led to the viewing of bibliographic records. For example, a user might enter the search expression "Neptune planet"

TABLE 7 ANOVA OF NUMBER OF SEARCH EXPRESSION TERMS NOT IN RESPONSES

TO OPEN QUESTION

Effect SS df MS F P

Knowledge level 79.1080 2 39.5540 5.507 .0069 Difficulty level 48.3272 1 48.3272 6.729 .0117 Interaction 77.2359 2 38.6180 5.377 .0076 Error 387.8500 54 7.1824




7'

6 J

5'

j4

Z3

2

I Easy

0 LOW Medium gh


FIG. 5.-Mean number of search expressions containing new vocabulary

and be shown several bibliographic records citing materials about the planet Neptune. The second category consisted of search expressions that led the user to a place in the authority list from which it was possible to select an alternative term, with this alternative term leading to the viewing of bibliographic records. For example, a user might enter the search expression "Voyager project." This would lead to the display of a number of subject headings, including "Project Voyager," which (if selected) would cause a number of citations to materials on this topic to be displayed. Finally, there were those search expressions that did not lead to the viewing of bibliographic records. The obvious example is the "zero hits" scenario, where the search expression failed to match any- thing in the database. A less obvious, but frequently encountered, scenario is where the search expression was so general that it retrieved too many citations for display. Another possibility is for a search expression to lead to a list of possible subject headings, none of which was selected by the user for display of records. Because these search expressions did not produce bibliographic citations, they were designated "nonproductive" search expressions.

There was a high correlation between the frequency of occurrence of




TABLE 8 ANOVA OF NUMBER OF NONPRODUCTIVE SEARCH EXPRESSIONS

Effect SS df MS F P

Knowledge level 74.2645 2 37.1323 4.093 .0216 Difficulty 125.6988 1 125.6988 13.857 .0008 Interaction 84.2277 2 42.1138 4.643 .0136 Error 489.841 54 9.0711

search expressions that did not lead to bibliographic citations and the frequency of occurrence of the general search expressions discussed above (r[58] = .73, P < .0001). This indicates that the participants who were using the general search expressions were also using nonproductive search expressions.

There were no differences between participants, either based on level of topic knowledge or difficulty experienced with the search, in either of the first two categories of search expressions. Participants used (on average) 1.4 of the search expressions that led directly to viewing bibliographic records and (on average) 1.65 search expressions that led indirectly to the viewing of bibliographic records. The noticeable (and statistically significant) difference was noted in the third category: nonproductive search expressions that did not lead to viewing records. Ta- ble 8 and figure 6 show the results obtained in analyzing this variable. Again, it was the participants with high topic knowledge and who expressed difficulty with their searches who used the greatest number of nonproductive search expressions.

In a follow-up investigation, search tactics employed by participants following the use of nonproductive search expressions were analyzed in detail. A total of twelve different search tactics, some reminiscent of Bates's [26] search tactics for online searches, were used. Table 9 gives the frequency of occurrence of each of these tactics. It seems clear from this table that attempts by participants to alter a search to produce more acceptable results were in the minority. By far the most commonly oc- curring tactic was to abandon a search formulation and to enter an entirely new one.

Discussion

These findings demonstrate that differences in factual knowledge of the topic being searched are associated with differences in search formulation in some of the ways suggested by research in cognitive science.



98 1 1/ 1 7,

04

2I

LOW Medium High Level of Topic Knowledge

FIG. 6.-Mean number of nonproductive search expressions

TABLE 9 SEARCH TACTICS USED FOLLOWING NONPRODUCTIVE SEARCHES

Search Tactic Frequency

1. Use a different term (that is, change both main heading and subdivision) 123 2. Change a subdivision (but keep main heading the same) 20 3. Give up and stop 16 4. Change a main heading (but keep subdivision the same) 13 5. Delete a subdivision (but keep main heading the same) 11 6. Use a synonym 10 7. Delete a main heading (turn previous subdivision into a main heading) 8 8. Do a title or author search 7 9. Add a main heading (turn previous main heading into subdivision) 6

10. Invert order of main heading and subdivision 6 11. Add a subdivision (but keep main heading the same) 5

12. Try exactly same search expression again I




The principal difference is that high-knowledge participants used more search expressions than low-knowledge participants. In doing so, they used terms that had not appeared in their topic descriptions, thus incor- porating additional vocabulary into their search formulations. Cognitive research predicts that high-knowledge participants would have greater familiarity with the technical vocabulary of the topic than would low- knowledge participants and would use alternative terms if necessary to enhance their searches. Evidence for this greater familiarity with the vocabulary can be found in the additional search expressions employed and in the introduction of novel vocabulary. The ability to use a larger number of search expressions and to introduce previously unused terms into those expressions is consistent with the idea that high-knowledge users are likely to be more familiar with the vocabulary associated with a topic.

Cognitive research also predicts that high-knowledge participants would have a well-developed mental model of the topic. One of the results of this research appears to be consistent with that prediction. The fact that high-knowledge participants used more of the "general" search terms might indicate that they were drawing on a broader understanding of the topic. If that is the case, then their better-developed mental model was leading them into an inappropriate search strategy.

The findings establish that differences in topic knowledge are associated with differences in search vocabulary. They also indicate that differences in search vocabulary are associated with differences in search quality and in the level of difficulty with the search reported by participants. From these results it would be possible to establish a causal chain from level of topic knowledge to vocabulary differences to search effectiveness. But this causal chain is not supported by the findings. In fact, there was no significant correlation between test score and level of reported difficulty or approximate recall. This pattern of results indicates that there may be an additional variable that has an effect on search vocabulary and effectiveness and that interacts with topic knowledge to affect search quality and level of expressed difficulty.

Findings from previous research on users of online systems suggest that familiarity with the online catalog may be that additional variable. In a number of the investigations of online searchers cited above, it was demonstrated that difficulty with searches could be attributed to a lack of familiarity with the search system or the database. It seems possible that the vocabulary differences and the difficulties reported by participants in this research can be attributed in part to a lack of knowledge of the online catalog. Although this interpretation goes beyond the findings reported above, it is based on a reasonably large number of




similar investigations. The causal link that this interpretation suggests is: topic knowledge and knowledge of the retrieval system interact to cause differences in search vocabulary, which in turn cause differences in search effectiveness.

Low-knowledge and medium-knowledge participants were not distin- guishable from each other in terms of search expression formulation and use. These participants used relatively few search expressions, regardless of their expressed difficulty with the search. They tended to use fewer general expressions and fewer nonproductive search expressions.

There were no differences in the statements of information need created by participants in response to an open question on a simulated pre-online search form. The vocabulary used by high-knowledge participants to describe the topic was virtually identical to that used by low- knowledge participants, although the high-knowledge participants had the flexibility to supplement this initial description with additional vocabulary when searching the catalog.

Implications

This research has a number of implications for information intermediaries and for designers of information retrieval systems. The first is that individuals who have a high level of knowledge about a topic may express their information need as if they had a low level of knowledge. It may not be possible for an intermediary to tell from the vocabulary presented in a statement of information need, or in a response to an open question, the level of topic knowledge of the patron. This rein- forces the importance of the reference interview in eliciting the information need of the patron.

The second implication is that high-knowledge users sometimes need to be directed to appropriate levels of specificity in their searches. These users are likely to try a larger number of expressions, some of which may be too general to permit effective retrieval. One speculation about this effect was briefly mentioned above: high-knowledge users may be more highly motivated to continue a search, trying every search expression that comes to mind, than are low-knowledge users. In a topic area in which they are expert, users may be more interested, or more stub- born, than in areas in which they are not knowledgeable.

This tendency of high-knowledge users to employ general search expressions presents a problem that can be solved in a variety of ways. One solution would be to train users to avoid general search expressions. Another would be to include general subject terms in the indexing of specific materials. A third would be to direct users from general search




expressions to more appropriate specific expressions, perhaps through controlled vocabulary searching with thesaural links. There are problems with all these solutions. The first presupposes a level of effectiveness for bibliographic instruction that has yet to be demonstrated. The second requires a change in cataloging practice that would be difficult to implement on both technical and political grounds. The creation of a system that would lead a searcher from an expression like "space exploration" to one like "Project Voyager" in a reliable and consistent manner is a challenge for system designers.

The third implication derives from the flexibility shown by high- knowledge users in applying "new" vocabulary in search expressions. This ability to create new search expressions for a topic might be productive if the records of the online catalog were richer in topic vocabulary. The relative scarcity of useful topic description in standard bibliographic records has been documented by Atherton's report on the Subject Access Project [27], and researchers such as Markey and Cal- houn [28] have experimented with means of enriching these records. One implication of this research is that high-knowledge patrons will be in a position to make constructive use of enriched records when they become more generally available.

Conclusions

This research has shown that level of topic knowledge has an impact on search expression formulation in OPAC searching. These results are consistent with the idea that high-knowledge users in this topic area had a greater familiarity with the vocabulary of the topic, and may suggest that they had a more developed mental model of the topic. This en- hanced familiarity with vocabulary and with the topic manifested itself in more search expressions, not all of which were productive.

Additional research is necessary to obtain a complete understanding of the impact of topic knowledge on information retrieval. This research studied only one topic area; other domains must be explored before general conclusions can be drawn about the effects of topic knowledge. Similarly, this research used only one online catalog. Replication of the research with other catalogs and more sophisticated information retrieval systems is encouraged.

It seems appropriate that catalogs (and other information retrieval systems) be designed to provide capabilities that will enable high- knowledge users to employ their expertise in searches for additional information. This research, by showing some of the ways in which high- knowledge users perform differently from low-knowledge users, has




identified some areas that require the attention of designers of online catalogs if they wish to facilitate searches by users with varying levels of topic knowledge.

REFERENCES

1. Borgman, Christine L. "All Users of Information Retrieval Systems Are Not Created Equal: An Exploration into Individual Differences." Information Processing and Man- agement 25 (1989): 237-51.

2. Daniels, PennyJ. "Cognitive Models in Information Retrieval: An Evaluative Review." Journal of Documentation 42 (1986): 272-304.

3. Spilich, George J.; Vesonder, Gregg T.; Chiese, Harry L.; and Voss, James F. "Text Processing of Domain-related Information for Individuals with High and Low Do- main Knowledge." Journal of Verbal Learning and Verbal Behavior 18 (1979): 275-90.

4. Chiese, Harry L.; Spilich, George J.; and Voss, James F. "Acquisition of Domain- related Information in Relation to High and Low Domain Knowledge." Journal of Verbal Learning and Verbal Behavior 18 (1979): 257-73.

5. Voss, James F.; Vesonder, Gregg T.; and Spilich, George J. "Text Generation and Recall by High-Knowledge and Low-Knowledge Individuals." Journal of Verbal Learn- ing and Verbal Behavior 19 (1980): 65 1-67.

6. Fincher-Kiefer, Rebecca; Post, Timothy A.; Greene, Terry R.; and Voss, James F. "On the Role of Prior Knowledge and Task Demands in the Processing of Text." Journal of Memoiy and Language 27 (1988): 416-28.

7. Recht, Donna R., and Leslie, Lauren. "Effect of Prior Knowledge on Good and Poor Readers' Memory of Text." Journal of Educational Psychology 80 (1988): 16-20.

8. Borgman, Christine. "The User's Mental Model of an Information Retrieval System: Effects on Performance." Ph.D. dissertation, Stanford University, 1983.

9. Thorngate, Warren, and Hotta, Miho. "Expertise and Information Retrieval." Knowl- edge: Creation, Diffusion, Utilization 11 (1990): 237-47.

10. Blair, David C., and Maron, M. E. "An Evaluation of Retrieval Effectiveness for a Full-Text Document Retrieval System." Communications of the ACM 28 (1985): 289-99.

11. Fenichel, Carol H. "Online Information Retrieval: Identification of Measures That Discriminate among Users with Different Levels and Types of Experience." Ph.D. dissertation, Drexel University, 1979.

12. Bellardo, Trudi. "Some Attributes of Online Search Intermediaries That Relate to Search Outcome." Ph.D. dissertation, Drexel University, 1984.

13. Howard, Helen. "Measures That Discriminate among Online Searchers with Different Training and Experience." Online Review 6 (1982): 315-27.

14. Saracevic, Tefko, and Kantor, Paul. "A Study of Information Seeking and Retrieving. II. Users, Questions and Effectiveness." Journal of the American Society for Information Science 39 (1988): 177-96.

15. Saracevic, Tefko; Kantor, Paul; Chamis, Alice Y.; and Trivison, Donna. "A Study of Information Seeking and Retrieving. I. Background and Methodology."Journal of the American Society for Information Science 39 (1988): 161-76.

16. Siitonen, Leena. "On-line Searching: Perspectives and Research on End Users." In Encyclopedia of Libra?y and Information Science, vol. 41, edited by Allen Kent. New York: Dekker, 1986.

17. Lyon, Sally. "Annotated Bibliography." In End-User Searching: Services and Providers, edited by Martin Kesselman and Sarah B. Watstein. Chicago: American Library Asso- ciation, 1988.




18. Mischo, William H., and Lee, Jounghyoun. "End-User Searching of Bibliographic Databases." In Annual Review of Information Science and Technology, vol. 22, edited by Martha E. Williams. Amsterdam: Elsevier Science Publishers, 1987.

19. Users Look at Online Catalogs: Results of a National Survey of Users and Non-Users of Online Public Access Catalogs. Berkeley: University of California, 1982.

20. Jones, Richard M. A Comparative Evaluation of Two Online Public Access Catalogues. London: British Library Research and Development Department, 1988.

21. Bates, Marcia J. "Factors Affecting Subject Catalog Search Success." Journal of the American Society for Information Science 28 (1977): 161-69.

22. Allen, Bryce. "The Effects of Academic Background on Statements of Information Need." Library Quarterly 60 (1990): 120-37.

23. Allen, Bryce. "Bibliographic and Text-Linguistic Schemata in the User-Intermediary Interaction." Ph.D. dissertation, University of Western Ontario, 1988.

24. Lancaster, F. Wilfred. Information Retrieval Systems: Characteristics, Testing and Evalua- tion. New York: Wiley-Interscience, 1979.

25. Bates, Marcia J. "System Meets User: Problems in Matching Subject Search Terms." Information Processing and Management 13 (1977): 367-75.

26. Bates, Marcia J. "Information Search Tactics." Journal of the American Societyfor Infor- mation Science 30 (1979): 205-14.

27. Atherton, Pauline. Books Are for Use: Final Report on the Subject Access Project to the Council on Library Resources. Syracuse, N.Y.: Syracuse University, 1978.

28. Markey, Karen, and Calhoun, Karen. "Unique Words Contributed by MARC Records with Summary and/or Contents Notes." In ASIS '87: Proceedings of the 50th ASIS Annual Meeting, edited by C. C. Chen. Medford, N.J.: Learned Information, 1987.



Documents

Topic Knowledge and Online Catalog Search Formulation