Pergamon Library Acquisitions: Practice & Theory, Vol. 21, No. 3. pp. 355-363, 1997

Copyright © 1997 Elsevier Science Ltd Printed in the USA. All rights reserved

0364-6408/97 $17.00 + .00

PlI S0364-6408(97)00064-1

THE FIRST ELSEVIER ELECTRONIC SUBSCRIPTIONS CONFERENCE OCTOBER 17-18, 1996

HEEMSKERK, THE NETHERLANDS

USER STUDIES: ELECTRONIC JOURNALS AND USER RESPONSE TO NEW MODES

OF INFORMATION DELIVERY

CLARE JENKINS

Sub-Librarian

British Library of Political and Economic Science

London School of Economics

10 Portugal Street

London WC2A 2HD

United Kingdom

Internet: c.jenkins@lse.ac.uk

Abstract--Measuring in-library use of printed journals has always been difficult. Even when libraries monitor the use of their journal collections, data collected rarely go beyond the level of title and are often incomplete. Participation in a European Union funded project on the delivery of electronic files of journals to end users (DECOMATE) has enabled the London School of Economics (LSE) to collect qualitative and quantitative data on the use of its journal collections. This paper describes how user statistics are being collected at the LSE. It looks at the difficulties that have been encountered in collecting the data and at the solutions that have been found. Comparisons will be made between patterns of use within a traditional library environment and user behavior as journal literature becomes available as electronic images, accessible over the network. The paper concludes with speculation about the benefits that may derive from collecting and analyzing user statistics. © 1997 Elsevier Science Ltd

K e y w o r d s - - U s e r studies, Electronic journals, Monitoring usage, Collection manage-

ment, Journal collection usage

355

356 C. Jenkins

INTRODUCTION

Over the past 18 months or so, the London School of Economics (LSE) has been taking part in a research project, in partnership with two other European university libraries (Tilburg University and the Universitat Autonoma de Barcelona), supported by funding from DG XIII of the European Commission. The aim of the DECOMATE project (DElivery of COpyright MATerials to End Users) is to provide end users with access, through each institution's own local network, to copyright materials in electronic form. During the project, the libraries have been receiving part of their journal holdings in electronic format directly from the publishers, under a licence agreement. Included in this material is a significant number of titles published by Elsevier, although other publishers are also contributing to our database. We made a conscious decision to restrict ourselves to a limited subject focus, for the purposes of the project, to have a discrete and manageable user group, and so that the electronic files, although modest in number, would nevertheless provide users with a critical mass of information. We have chosen the fields of economics and computer science, disciplines that are studied at each of the partners' institutions, and where there is an overlap of journal subscriptions. There have been several parts to the project: the development of generic software to allow searching of the bibliographic files and the retrieval of the image files of the journal articles; the development of monitoring software for user authentication and transaction logging; the implementation of a live, operational system on each of the three sites; the investi- gation of the copyright issues surrounding electronic publishing; a user evaluation study to investigate the impact of electronic distribution on end users; and the dissemination of our findings to the wider library and information community.

USER STUDY

The LSE has been a lead partner in the user study, and this paper will focus on our work on this aspect of the project. For someone like me, whose professional background is deeply rooted in collections, it is particularly interesting to be involved in a project in which a major objective is to study how collections are used. Leaving aside the requirements of the project, it has been extremely useful for us to glean a better understanding about the use of our journal collections. (I think it is generally acknowledged, in the library world, that we have notoriously little idea of how our journals are used!) My previous experience of monitoring journal usage dates back a number of years. Readers were discouraged from reshelving bound volumes, and library staff kept five-barred gates to record the frequency with which bound volumes of titles were consulted; we supplemented this information by asking users to initial slips of paper stapled to the covers of current issues. Very low tech and almost certainly not fully representative of use, but it was better than nothing, and it served a purpose when we had to assign priorities during a period of budget cuts.

In some libraries it may be possible to use loan statistics to provide information on usage, but many libraries do not lend journals, and even if they do, users will often prefer to consult this kind of material in the library. So loan statistics tend not to be a very reliable indicator of use. Other libraries have used impact factor analysis, where they analyze the articles referred to in publications produced by aca~lemics at their own institution to assess which journals are most frequently cited. Monitoring usage can be a very time-consuming process, and it is one that we have been able to undertake now only because we had staff effort dedicated to it within our project plan. I imagine few libraries would be able to sustain data collection of this type over a long period, unless the activity were linked to a specific research project, much though they might like to have the

Electronic Journals and User Response 357

information it revealed. This paper will touch on the following areas: the aims of the user study, the methodology used, and the preliminary findings.

Timing of the User Study Although DECOMATE itself has been running since March 1995, the first 15 months of the

project have been primarily concerned with the development of the DECOMATE software, the successful implementation of a live system on each site, and the negotiation of licence agreements between the libraries and publishers. The DECOMATE system went live at the beginning of October 1996. In the 4 months prior to this, we have been collecting data through in-depth interviews about the way our researchers use journal literature in the traditional print environment. Those we are interviewing have agreed to be "committed guinea pigs" who will try out the system over the next 5 months and give us their opinions about it. The preliminary interviews with this group will be followed by a second interview toward the end of the year, when we will look at our researchers' reactions to the DECOMATE system and the direct access it offers to electronic files. We needed to be careful, when scheduling the first interviews, that we did not time them too far in advance of DECOMATE becoming available. We were advised that to do so might raise their enthusiasm too soon, and that we then risked having them lose interest when they found that the system was still not available several months after being told about it.

The collection of user statistics, linked to the live running of the system, has been ongoing since October 1996 and we have not yet had time to carry out any statistical analysis of this data. I hope to give you a flavor of our preliminary findings, but at this stage in the project the picture is still, necessarily, incomplete. Most of these preliminary findings are based on data collected during interviews with our users. We are also conscious that the length of the project's user study (the project is due to end in February 1997) is very short--almost certainly too short for us to collect sufficient data for our findings to be meaningful. And we are currently exploring avenues that may enable us to continue our work over a longer time span.

Aims of the User Study The aims of the user study are to establish the usefulness of electronic distribution of copyright

material to the research community, understand what users require from an electronic document delivery service, and understand the impact of electronic distribution on user behavior. The study is being carried out at all three sites, although we shall be making no attempt to correlate the findings between institutions. We are seeking to collect and analyze data in the following areas:

1. Amount and type of usage--Who does what; have we been able to establish any trends, and, if so, what are they? Will there, for instance, be any foundation in my assumption that there would be difference in behavior depending on age and seniority? Was this just a false (and some might say simplistically biased) assumption?

2. Ergonomic issues---What do our users think about the design of the user interface? How intuitive did they find it? Are there features they would like that it does not offer? Are there aspects of the system design that irritate them, and, if so, what are they? How do they use the electronic f i les--Do they read on screen, or do they still want to print off the documents that interest them? Is the quality of the image files acceptable--on screen and when printed?

3. User acceptance--What will the pattern of use tell us about the users' acceptance of the system? How many returning users do we have? How far will our modes of data collection enable us to gather answers to these last questions?

358 C. Jenkins

METHODOLOGY

Data Collection From the outset it has been our intention to collect both qualitative and quantitative data. To

achieve this we are collecting data in four different ways. The data we collect will be separated from the name of the individual who contributed the information so that all of the information becomes anonymous, regardless of the method we use to obtain it. Qualitative data will be gathered through in-depth interviews and focus group meetings. Quantitative data will come from retro- spective analysis of log files and interactive online questionnaires.

In-Depth Interviews Part of our strategy has been to recruit a target group of users from our academic staff and

researchers who committed themselves to our project and to trying out the system. Participants have ranged from first year Ph.D. students to academics, who have been engaged in research for more than 20 years. Most have access to their own networked computer at the LSE. Two sets of interviews, conducted on a one-to-one basis, are being held with this target group. The first interviews were held over the summer, before the DECOMATE system was available for public use. The questioning focused on their current information seeking and literature usage patterns. We also attempted to build up a picture of each individual's affinity with their field, and with electronic services in particular. We hoped in this way to establish whether a user's background and familiarity with electronic services has any relationship to that person's perception of and response to the DECOMATE system. A second interview will be held toward the end of the project (probably around January 1997), when we shall be comparing responses with our findings from the first interview. At this time we shall be focusing more closely on users' direct reactions to the system; their perceptions of the pros and cons; their reaction to the user interface and how it compares with other electronic services they may use; and how their research behavior has changed as a result of having access to the system. We have seen this interview process as an important part of the user study, if we are to assess not only how users in an electronic environment appear to use networked information, but also how their research behavior has changed (if at all) as a result of the new technology. These interviews will provide us with our primary source of qualitative data and will give us a much richer picture than can be obtained from quantitative data alone.

Focus Group Meetings The focus group meetings complement the in depth-interviews and will explore similar issues

but in an open forum setting. The focus groups will include members of our target groups plus some users who have made use of the DECOMATE system on their own initiative. The emphasis of these meetings will be on sharing experience and exchanging information. We anticipate that such information will contribute a different perspective because of the group dynamics that will come into play, but will provide us with additional qualitative data. Focus meetings will take place twice during the course of the user study (October/November and December/January), but no meetings have yet been held so I am unable at this stage to give you information about the outcome of these group discussions.

Interactive Online Questionnaire An interactive online questionnaire has been included within the DECOMATE system design. The

questionnaire pops up randomly, when users log in, and they are offered a choice of answering

Electronic Journals and User Response 359

immediately or not at all. The design of the questionnaire is deliberately concise and simple to answer, to encourage as many people as possible to complete it. Users are asked to answer yes or no, or to select an answer from a list of multiple choices. The simple design also is intended to avoid errors of data analysis and to remove the possibility for misinterpreting answers. The focus of the data collection in this part of the user study relates to the ergonomics of the DECOMATE system, familiarity with electronic systems, and the use of DECOMATE on that day. The advantages of choosing this type of data collection over the more traditional printed questionnaire are:

• truly random sampling, based on the total current user population; • rapid feedback; • data collected in a standard, computer-readable form; • ability to control and vary by program module the volume of data collected; and • cost reduction (null response merely increments a counter).

A possible difficulty is that a large proportion of users, when presented with the questionnaire, will choose not to answer it, although our assessment has been that we might expect a response rate at least as good as we would have had from a paper questionnaire. We are, however, able to control the frequency with which the questionnaire pops up, so we could, if we wished, in the case of a low response rate turn the frequency up (or, alternatively, turn it down, if the response rate turns out to

be particularly high).

Retrospective Analysis of Log Files This aspect of our user study is, I suspect, the part that the publishers in the audience, at least,

will be most interested in. How are we logging system transactions and what transactions are we

logging?

HOW THE DATA ARE COLLECTED

Work on the system design has included the development of a monitoring toolkit to enable us to collect and analyze data relating to actual use of the system. I now want to look in more detail at what the monitoring package will do.

Authorization Module The authorization module operates at the point when a user logs on. To gain full access to the

system, users will need to register initially with the library as an authorized user. It is technically possible to configure the system to provide access to the bibliographic database--containing contents pages and abstracts--using a "gues t " logon, but no user will be able to access the articles database without first having registered and obtained a user ID and password. The logon ID identifies users down to the level of category and department, for example, a postgraduate student in the department of economics. It does not provide more precise identification than this. When the project teams discussed this issue, there was a general agreement that a unique identifier at the level of the individual would not be acceptable in terms of our users' rights to privacy over what information they were choosing to access. This decision means that some questions, in the context of user evaluation, will remain unanswered. For example, if we want to find out about returning

360 c. Jenkins

users we shall have to rely on data we collect via our interviews, focus group meetings, and questionnaires.

Transaction Logging Once a user is logged onto the system, the monitoring package will record, in the log files, data

on what has been retrieved from the DECOMATE system. In the current World Wide Web (WWW) environment, there are some limitations on what data we are able to record. So we will have statistics about data retrieval--for example, whether the bibliographic or the article database has been accessed; which search option has been used; which titles and issues have been searched; and whether the full article image has been retrieved. What we will not be able to know, in the Web environment, is how long our users have been logged onto the system or what they have subsequently done with the information retrieved, although of course it is precisely this which many of us would find most interesting!

So, then, to return for a moment to the kind of statistics we will be collecting by this method. The DECOMATE system contains two separate, but linked, databases: the bibliographic database and the full-text (image) database. There will be log files recording every transaction that takes place in each of these databases, so we will have a full picture of all activity at the server end. We won't, however, be able to log any activity at the client end. Data from the log files will be collected on each site and processed using an industry standard spreadsheet package into a format that will be the same for all three partners. This processed data will then be sent to the LSE for further analysis as part of the user study.

The current plan is to collect and analyze these statistics on a monthly basis, but the frequency could be varied and is within our control. Since the log files will contain raw data relating to all system transactions, there is, theoretically, scope to provide a multiplicity of statistical analyses. In practice we have agreed among ourselves on a dozen or so queries that we believe will provide the information that we need for the user study. If, with experience, we find that there are gaps in our coverage we merely need to add another query to our regular statistical analysis run. We can categorize the statistics we plan to collect under five headings: users, searches, frequency of use of journal titles and/or journal articles, logon sessions, and patterns of use, bibliographic to journal database.

WHY ARE WE COLLECTING THE DATA?

One answer to this question is that we are required, under the conditions of the project, to carry out a user study ! Seriously, I believe that in any innovative project of this kind, it is important to evaluate what is being done, to draw conclusions about the project's longer term future and to consider the strategic implications for the library. The strategic implications are likely to range from budgetary considerations (recurrent costs for the information technology infrastructure to support the initiative; recurrent costs of licences for electronic subscriptions) to collection devel- opment issues (should the library's collecting focus shift from paper to electronic, and, if so, how much; what are the long-term archival implications for such a shift and how do they relate to the library's own responsibilities in this respect?). At a more detailed level, the collection of statistics will allow us to gain:

• better understanding of how users use journal collections; • better knowledge of which journals are consulted and the intensity of use;

Electronic Journals and User Response 361

• better knowledge of which articles are consulted and the intensity of use; and • better understanding of how researchers move between the contents page/abstract and the full

article.

All of these are data that have been, if not impossible to collect in the traditional print environment, certainly so staff intensive as to be impractical. I should add that we have also agreed, with one of the publishers supplying us with electronic files, that we will share with them a subset of data relating to specific use of their published material. I can understand the rationale behind this request in the context of a research project that is experimenting with innovative methods of publishing and information delivery. However, I do have concerns about the validity of such a request becoming a contractual norm under any future licence agreements, since it appears to me that such a requirement is tantamount to asking the library to carry out the publisher's market research for free.

DIFFICULTIES IN IMPLEMENTING DATA COLLECTION

One potential difficulty in terms of the collection of qualitative data has been the problem of recruiting volunteers, particularly since we were trying to include a range of user types with different discipline interests. At the LSE, we tried to overcome this by inviting selected academics from the relevant departments to a presentation of the project and the prototype system accompa- nied by lunch. We won a number of recruits this way, plus promises (that were fulfilled, in the main) that they, in turn, would recruit additional volunteers from their students and colleagues. We committed ourselves to offering training and support for the system to encourage use. We also sent letters to individuals, on spec, and had some response from this approach including, as we had hoped, some users who were relative novices in terms of information technology (IT) applications.

We think that the limitations of the system in its current form (limited coverage in terms of both breadth and depth) may deter some people from making extensive use of it. At present the database is still quite small (some 80 titles in the case of LSE, with backruns extending only to 1995). Another limitation (although also a strength) is the restricted subject focus. We wonder if users who have not been formally recruited to the study and whose academic interests fall at the margins of the discipline may choose not to experiment with the system when they see it is available on the network. Many of our colleagues are already suffering from information overload. They need to feel there is adequate payback for them to become familiar with yet another electronic service.

The W W W environment has lots of advantages for our users. However, there has been one significant disadvantage for us in developing the system to run in a W W W environment. There are major limitations on the kind of data we are able to log. We can record all activities up to the point when a user has retrieved data from the database. From then on we have no means of knowing how that data are being used. So, for example, we will be unable to reach any conclusions, via the log files, about whether users read documents to any great extent (or at all) on screen, or whether the printed medium is still the preferred option. These, I suspect, are some of things we would all--librarians and publishers--most like to know.

WHO IS USING THE DATA?

The main user of the statistical data, is, I suppose, the library. Under the current project, of course, this activity is focused on the DECOMATE project team. If one looks beyond the life of the project and envisages a scenario where access to electronic journal files is still provided as a

362 c. Jenkins

regular part of the library's services, then the statistics will be of interest for collection development librarians, IT support staff, senior library managers, and publishers.

Benefits of Usage of the Data for the Library I have already indicated the categories of staff who have an interest in the statistics that can be

obtained from the DECOMATE system. The availability of this data, which provides us with information at a level of detail that has been impossible in the traditional print environment, will be of benefit in a number of ways. For example, the data will help us to gauge low- and high-use titles and help inform future selection and budgetary decisions. It will also help us to gauge high- and low-volume issue/article use within individual titles, as a means of managing file storage as the database grows in size. Assessing the cost benefit of investment in this particular kind of service will be enhanced by this data. Finally, the data will inform the process of strategic planning when developing networked services.

Benefits of Usage of the Data for the User Will all this data collection benefit the user? I think the answer is yes. It seems to me that the

library will be in a better position to understand user behavior and user needs. If it uses that information intelligently, it should be better able to make decisions on developments in collections and services that are based on objective facts rather than general supposition.

PRELIMINARY FINDINGS AND CONCLUSIONS

At this point, it would be usual to expect that I would offer you some general remarks based on conclusions we had been able to draw from the outcomes of our project. At the time of writing we are still less than 3 weeks into the running of a live system, so it is really too early to say very much. We hope to present more detailed findings at the DECOMATE Conference, which we will be holding in February 1997. However, we have been able to draw out a few points from the data we have collected during our interviews with our user group, and I thought I would finish by giving you a flavor of what has emerged at the LSE. Most of these points relate to qualitative rather than quantitative data.

1. The majority of our user group believes journals are very important for research. The rest rate them as important, and no one has categorized them as anything less than important.

2. The number of journals read on a regular basis varies from 0 to 15+. 3. The use of journals varies. Most people combine some current awareness with searching for

particular articles. A number of our users search paper copies for items on a particular subject.

4. Specific references are most frequently obtained from colleagues. Electronic sources and citations in other articles are the second most common way of finding references.

5. Most users photocopy articles of interest and expect to be able to print from a full-text electronic service.

Most of our users have been positive about the concept of the service and identify few drawbacks. However, there are some concerns. They are:

ELectronic Journals and User Response 363

1. Timeliness. Issues should be available promptly and, preferably, before the printed version; 2. Critical mass of information. Users have expressed a need to be able to search up to 25 years'

worth of material in a single search (5 years was the minimum coverage considered to be

useful); 3. System performance and quality of images; 4. Impact on print version and possible costs for end users.

The most frequently cited benefit is improved access. An electronic service will "overcome the physical obstacles" and will bring the "product of academic and scholarly work as close to the potential beneficiaries as humanly possible." There is also a perception that it will save researchers

time. It is still too early for us to say how the system will develop and what its long-term future will

be. Whatever the outcome, we will have a better understanding of the way our journal collections-- whether print or electronic--are being used by our researchers.

Acknowledgements The author would like to acknowledge the contributions made by Caroline Lloyd, also of the

BLPES, during the drafting of this paper.