Upload
berenice-watson
View
220
Download
1
Tags:
Embed Size (px)
Citation preview
Measuring the Impact of Networked Electronic Services: Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure for Libraries, Developing Assessment Infrastructure for Libraries,
State, and Other Types of ConsortiaState, and Other Types of Consortia
Presented By:Terry Plum, Simmons GSLIS
Brinley Franklin, University of ConnecticutMartha Kyrillidou, ARL
Gary Roebuck, ARLRaynna Bowlby, ARL Consultant & Simmons GSLIS
MaShana Davis, ARL Kristina Justh, ARL
Library Assessment Conference 2008University of Washington
Seattle, WA
http://www.minesforlibraries.org/
ARL New Measures Toolkit: ARL New Measures Toolkit: StatsQUALStatsQUAL®®
LibQUAL+® is a rigorously tested Web-
based survey that libraries use to solicit, track, understand, and
act upon users‘ opinions of service
quality.
LibQUAL+®
DigiQUAL®
The DigiQUAL® online survey designed for
users of digital libraries that measures reliability and trustworthiness of
Web sites. DigiQUAL® is an adaptation of
LibQUAL+® in the digital environment.
MINES for Libraries®
Measuring the Impact of Networked Electronic
Resources (MINES) is an online transaction- based survey that
collects data on the purpose of use of
electronic resources and the demographics
of users.
ARL Statistics™
ARL Statistics™ is a series of annual publications that
describe the collections, expenditures, staffing,
and service activities for Association of Research Libraries (ARL) member
libraries.
ClimateQUAL™
ClimateQUAL™: Organizational Climate
and Diversity Assessment is an online
survey that measures staff perceptions about:
(a) the library's commitment to the
principles of diversity, (b) organizational
policies and procedures, and (c) staff attitudes.
http://www.minesforlibraries.org/
What is MINES?What is MINES?
Action Research Historically rooted in indirect cost studies Set of recommendations for research design Set of recommendations for web survey
presentation Set of recommendations for information
architecture in libraries Plan for continual assessment of networked
electronic resources An opportunity to benchmark across libraries
http://www.minesforlibraries.org/
What is MINES? What is MINES? (cont’d)(cont’d)
Measuring the Impact of Networked Electronic Services (MINES)
MINES is a research methodology that measures the usage of networked electronic resources of a library or consortium by a specific category of the patron population.
MINES is a Web-based survey form consisting of 5 questions that is administered at the time of transaction.
MINES measures: User status and discipline/affiliation (who) Physical location (where) Primary purpose and reason of use (why)
MINES a part of Association of Research Libraries’ New Measures & Assessment Initiatives (since 2003).
MINES is different from other electronic resource usage measures that quantify total usage (e.g., Project COUNTER, E-Metrics) or measure how well a library makes electronic resources accessible (LibQUAL+®).
http://www.minesforlibraries.org/
Early Data Collection ActivitiesEarly Data Collection Activities
Academic Medical Libraries U of Connecticut Health
Center U of North Carolina U of Texas Medical Branch U of Texas Southwestern U of Utah U of Virginia Washington U
Main Universities U of Colorado U of Connecticut U of North Carolina Oregon State U U of Utah U of Virginia Washington U
Data was collected at seven main campus libraries and seven academic health sciences libraries in the U.S. between 2003 and 2005.
http://www.minesforlibraries.org/
Early Data Collection ActivitiesEarly Data Collection Activities
More than 45,000 networked electronic services uses were surveyed.
At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities.
http://www.minesforlibraries.org/
Recent Data Collection Recent Data Collection Activities via ARLActivities via ARL
Ontario Council of University Libraries (OCUL)
University of Iowa Libraries University of Macedonia
http://www.minesforlibraries.org/
Questions AddressedQuestions Addressed
How extensively do sponsored researchers use the new digital information environment?
Are researchers more likely to use networked electronic resources from inside or outside the library?
Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?
Are particular network configurations more conducive to studies of digital libraries patron use?
http://www.minesforlibraries.org/
MINES for LibrariesMINES for Libraries®® Survey Form Survey FormFive Questions and a Comments BoxFive Questions and a Comments Box
Methodological Considerations:Methodological Considerations:Experience with the MINES SurveyExperience with the MINES Survey
Terry PlumAssistant DeanSimmons GSLIS
Library Assessment Conference 2008University of Washington
Seattle, WA
http://www.minesforlibraries.org/
Issues with Web surveysIssues with Web surveys
Research design Coverage error
Unequal access to the Internet Internet users are different than non-users
Response rate Response representativeness
Random sampling and inference Non-respondents
Data security
http://www.minesforlibraries.org/
MINES StrategyMINES Strategy
A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.
Random moment/web-based surveys are employed at each site.
Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time. IRB waiver or approval
Libraries with database-to-web gateways or proxy re-writers offer a comprehensive networking solution for surveying all networked services users during survey periods.
http://www.minesforlibraries.org/
MINES StrategyMINES Strategy
Placement Point of use Not remembered, predicted or critical incident
Usage rather than user What about multiple usages Time out ? Cookie or other mechanism with auto-population or
more recently counting invisibly with a time out. Distinguish patron association with libraries.
For example, medical library v. main library. But what if the resources are purchased across
campus for all. Then how to get patron affiliation?
http://www.minesforlibraries.org/
Web Survey Design GuidelinesWeb Survey Design Guidelines
Web survey design guidelines that MINES followed: Presentation
Simple text for different browsers – no graphics Different browsers render web pages differently
Few questions per screen or simply few questions Easy to navigate Short and plain No scrolling Clear and encouraging error or warning messages Every question answered in a similar way - consistent
Radio buttons, drop downs ADA compliant Introduction page or paragraph Easy to read
Must see definitions of sponsored research. Can present questions in response to answers – for example if
sponsored research was chosen, could present another survey
http://www.minesforlibraries.org/
Quality ChecksQuality Checks
Target population is the population frame – surveyed the patrons who were supposed to be surveyed - except in libraries with outstanding open digital collections.
Check usage against IP. In this case, big numbers may not be good. May be seeing the survey too often.
Alter order of questions and answers, particularly sponsored and instruction.
Spot check IP against self-identified location Spot check undergraduates choosing sponsored research –
measurement error Check self-identified grant information against actual grants Content validity – discussed with librarians and pre-tested. Turn-aways – number who elected not to fill out the survey Library information architecture -- Gateway v. HTML pages –
there is a substantial difference in results.
http://www.minesforlibraries.org/
Documenting the Purpose and Use of Documenting the Purpose and Use of Electronic Resources:Electronic Resources:
Experience with the MINES SurveyExperience with the MINES SurveyBrinley Franklin
Vice Provost for University LibrariesUniversity of Connecticut
Library Assessment Conference 2008University of Washington
Seattle, WA
http://www.minesforlibraries.org/
““It is useless to tell the acquisitions librarian that half the It is useless to tell the acquisitions librarian that half the monographs ordered will never be used, unless we can specify monographs ordered will never be used, unless we can specify
which 50% to avoid buying.” (Galvin and Kent, 1977)which 50% to avoid buying.” (Galvin and Kent, 1977)
http://www.minesforlibraries.org/
Reliance on Vendor StatisticsReliance on Vendor Statistics Vendor statistics, while more reliable than in the past, are still maturing.
http://www.minesforlibraries.org/
The most popular current method of measuring usage of electronic resources by libraries is not through web-based usage surveys, but through vendor supplied data of library patron usage or transaction usage.
Web-based usage surveys are increasingly relevant in the collection of usage data to make collection development and service decisions, to document evidence of usage by certain patron populations, and to collect and analyze performance outputs.
Brinley Franklin and Terry Plum, “Successful Web Survey Methodologies for Measuring the Impact of Networked Electronic Services (MINES for Libraries®)” IFLA Journal 32 (1) March, 2006
Measuring Digital Content UseMeasuring Digital Content Use
http://www.minesforlibraries.org/
A web-based transactional survey that collects data on users’ demographics and their purpose of use. It is administered in real time over the course of at least a year using a random moments sampling plan.
MINES for Libraries® has been administered at 40 North American universities in the last four years. More than 100,000 North American networked services users have been surveyed using a standard protocol.
Measuring Digital Content UseMeasuring Digital Content Use
http://www.minesforlibraries.org/
Library User SurveyLibrary User Survey
http://www.minesforlibraries.org/
Library User SurveyPatron Status
http://www.minesforlibraries.org/
Library User SurveyAffiliation
http://www.minesforlibraries.org/
Library User SurveyLocation
http://www.minesforlibraries.org/
Library User SurveyPurpose
http://www.minesforlibraries.org/
Sample Survey Data File GeneratedSample Survey Data File GeneratedOther UConnInstruction/Education/Departmental (Non-Funded)
Researchhttp://newfirstsearch.oclc.org/done=referer;dbname=WorldCat;autho=100122319;FSIP12:36:5012/3/2004Off CampusUConn Faculty12.101.104.84
Family StudiesInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/cgi-bin/jstor/gensearch12:37:4312/3/2004Off CampusUConn Undergraduate Student12.101.104.84Non-UConnOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:08:4112/3/2004Off CampusNon-UConn12.18.36.40
Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html13:31:2912/3/2004Off CampusNon-UConn12.76.131.246
Non-UConnOther Activitieshttp://magic.lib.uconn.edu/index_real.html12:11:0612/3/2004Off CampusNon-UConn130.132.86.156Agriculture & Natural ResourcesInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html12:33:5712/3/2004
Off CampusNon-UConn131.128.89.54EducationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.euromonitor.com/womdas/12:57:4412/3/2004
Off CampusNon-UConn134.241.135.70Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:28:5212/3/2004
Off CampusNon-UConn134.74.42.82Business AdministrationOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:56:4612/3/2004In the LibraryUConn Faculty137.99.1.122
Liberal Arts & SciencesOther Activitieshttp://www.siam.org/journals/simax/simax.htm12:52:1712/3/2004On Campus - StorrsUConn Graduate Student137.99.1.122
EngineeringInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:04:3112/3/2004On Campus - StorrsUConn Graduate Student137.99.1.219
Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/pqdweb?RQT=31812:16:3312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105
Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/journals/07322399.html12:16:5212/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105
Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:29:5312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175
Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:48:4112/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175
Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/login?COPT=SU5UPTAmVkVSPTImREJTPTE3MjErMysxNkJD&clientId=4899612:04:2312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175
http://www.minesforlibraries.org/
24%
25%
8%
43%
Graduate Students
Faculty, Staff,Research Fellows
UndergraduateStudents
All Other Users
40%
31%
25%
4%
28%
11%
21%
40%
On Campus, Not in the Libraryn = 6,391
Inside the Libraryn = 9,172
Off-Campusn = 4,953
MINES for LibrariesMINES for Libraries®®
Demographics by Location of UserU.S. Main Libraries
http://www.minesforlibraries.org/
14%
14%
69%
3%
Graduate Students
Faculty, Staff,Research Fellows
UndergraduateStudents
All Other Users
Inside the Libraryn = 4,047
On Campus, Not in the Libraryn = 7,090
45%
24%
29%
2%
31%
16%
48%
5%
MINES for LibrariesMINES for Libraries®®
Demographics by Location of UserOntario Council of University Libraries
Off-Campusn = 9,163
http://www.minesforlibraries.org/
Purpose of UsePurpose of Use
Are users engaged in coursework, funded (or unfunded) research, public service, patient care, or other activities?
http://www.minesforlibraries.org/
26%3%
5%
SponsoredResearch
Instruction
Other
Other SponsoredActivities
*72% of sponsored research usage of electronic resources occurred outside the library; 83% took place on campus.
63%
14%2%
21%
11%
62%
25%2%
6%
58%
34%2%
On-Campus, Not in the Libraryn = 9,460
In the Libraryn = 9,733
Off-Campusn = 7,790
Overall Usen = 26,983
66%
MINES for LibrariesMINES for Libraries®®
Purpose of Use by LocationU.S. Main Campus Libraries 2003-05
http://www.minesforlibraries.org/
8%6%2%
16%
26% 42% Sponsored Research
Coursework
Other Research
Other Activities
Teaching
Patient Care
MINES for LibrariesMINES for Libraries®®
Purpose of UseOCUL Scholars Portal Users
In a sample of 20,300 electronic resources uses at OCUL Libraries, there were four uses outside the library for each use in the library.
http://www.minesforlibraries.org/
AnalysisAnalysis
Web deliverables: Cross-tabulations in HTML for all institutional
data Interactive crosstabs for all institutional
Print deliverables: Summary tables Final report
http://www.minesforlibraries.org/
http://www.minesforlibraries.org/
Web InterfaceWeb Interface
http://www.minesforlibraries.org/
OCUL Scholars Portal Usage OCUL Scholars Portal Usage AffiliationAffiliation
http://www.minesforlibraries.org/
OCUL Scholars Portal Usage OCUL Scholars Portal Usage User StatusUser Status
http://www.minesforlibraries.org/
OCUL Scholars Portal UsageOCUL Scholars Portal UsageLocationLocation
http://www.minesforlibraries.org/
OCUL Scholars Portal UsageOCUL Scholars Portal UsagePurpose of UsePurpose of Use
http://www.minesforlibraries.org/
Cross Tabulations Affiliation by Purpose of Use
100.0%5.6%26.2%2.4%16.2%7.5%42.0%TOTAL
100.0%4.2%13.6%0.7%14.4%4.5%62.6%Social Sciences
100.0%2.4%31.8%0.4%11.1%9.7%44.6%Sciences
100.0%5.0%7.4%2.1%10.9%22.8%51.9%Other
100.0%5.7%32.0%8.6%18.4%5.5%29.7%Medical Health
100.0%9.4%2.6%0.9%12.8%6.8%67.5%Law
100.0%6.7%9.5%0.5%21.0%10.8%51.5%Humanities
100.0%9.4%5.6%1.3%20.6%6.9%56.3%Fine Arts
100.0%6.3%23.3%0.3%24.0%2.5%43.5%Environmental Studies
100.0%24.0%11.8%0.8%17.1%5.4%40.9%Education
100.0%16.0%10.8%0.9%30.0%7.6%34.8%Business
100.0%3.7%46.3%0.6%17.7%7.6%24.0%Applied Sciences
TotalTeachingSponsoredPatient
CareOther
ResearchOther
ActivitiesCourseworkAFFILIATION
PURPOSE OF USE
http://www.minesforlibraries.org/
100.0%5.6%26.2%2.4%16.2%7.5%42.0%TOTAL
100.0%1.9%5.9%0.9%7.7%7.8%75.8%Undergraduate
100.0%12.7%51.6%2.1%20.6%9.5%3.5%Staff
100.0%2.5%26.8%8.7%20.8%35.2%6.0%Other
100.0%5.2%17.7%16.5%13.1%24.1%23.5%Library Staff
100.0%3.2%45.4%2.5%25.5%3.9%19.5%Graduate Professional
100.0%25.6%42.6%4.4%21.2%4.7%1.5%Faculty
TotalTeachingSponsoredPatient
CareOther
ResearchOther
ActivitiesCourseworkUSER STATUS
PURPOSE OF USE
Cross Tabulations User Status by Purpose of Use
http://www.minesforlibraries.org/
Cross TabulationsCross TabulationsLocation by Purpose of UseLocation by Purpose of Use
100.0%5.6%26.2%2.4%16.2%7.5%42.0%TOTAL
100.0%5.7%42.2%0.9%17.9%4.0%29.2%On-campus
100.0%4.6%19.9%4.1%17.3%7.0%47.2%Off-campus
100.0%7.9%12.3%1.2%10.8%14.9%52.8%Library
TotalTeachingSponsoredPatient
CareOther
ResearchOther
ActivitiesCourseworkLOCATION
PURPOSE OF USE
http://www.minesforlibraries.org/
4,388Other
925Course Reading
620Recommended Librarian
6,090Reference/Citation
2,436Recommended Colleague
10,219Important Journal
FrequencyReason for Use
(n=20293)
21.6%Other
4.6%Course Reading
3.1%Recommended Librarian
30.0%Reference/Citation
12.0%Recommended Colleague
50.4%Important Journal
PercentReason for Use
(n=20293)
OCUL Scholars Portal Usage OCUL Scholars Portal Usage Reason for UseReason for Use
http://www.minesforlibraries.org/
http://www.minesforlibraries.org/
http://www.minesforlibraries.org/
http://www.minesforlibraries.org/
ResourcesResources
Web interfacehttp://minesforlibraries.org
ARL New Measures and Assessment Initiatives: MINES for Libraries®
http://www.arl.org/stats/initiatives/mines/index.shtmlplace
Articles & Presentationshttp://www.arl.org/stats/initiatives/mines/minesresources.shtml
http://www.minesforlibraries.org/
Contact UsContact Us
ARL Staff: MaShana Davis, Technical Communications Liaison Martha Kyrillidou, Director, ARL Statistics and Service
Quality Programs Gary Roebuck, Technical Operations Manager
Project Management: Brinley Franklin, Vice Provost for University Libraries,
University of Connecticut Libraries Toni Olshen, Business Librarian, Peter F. Bronfman
Business Library, York University Terry Plum, Assistant Dean for Technology and Director,
Simmons Graduate School of Library and Information Sciences
http://www.minesforlibraries.org/
Questions