Upload
ray-garcia
View
222
Download
0
Embed Size (px)
Citation preview
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
1/31
Evaluation of Website Comprehensibility
Evaluation of Website Comprehensibility .......................................................... 1
Comprehensibility Project .............................................................................. 4
Defining Comprehensibility .......................................................................... 5
Understanding the Evaluators Expertise .......................................................... 7
Defining the General Public Audience .............................................................. 8
STEM Topics ................................................................................................. 9
Web Site Sources ........................................................................................ 10
Website Pre-qualification for General Public Audience ................................... 11
Site Selection Process ............................................................................... 13
Comprehensibility Rating Process ............................................................... 15
Evaluation of Website Comprehensibility ........................................................ 15
Site Selection .......................................................................................... 15
Review Process ....................................................................................... 15
Review Scaffolding ................................................................................... 16
Review Duration ...................................................................................... 16
Rating Duration ....................................................................................... 16
Training Period ......................................................................................... 16
Comprehensibility Survey Questions ............................................................. 17
Site Classification ..................................................................................... 17
Reading Speed and Linking ....................................................................... 17
Prior Topic or Site Knowledge .................................................................... 17
Concept Recall......................................................................................... 18
Overall Informational or Instructional Rating ............................................... 18
Detailed Rating Method ............................................................................. 18
Information Content Rating .................................................................... 18
Internet Construction Rating ................................................................... 19
Post Evaluation ........................................................................................ 20
Processing Notes ..................................................................................... 20
Rating Matrix ............................................................................................. 21
Evaluation of Website Comprehensibility ........................................................ 21
Ray Garcia, June 2006 Confidential, University of Arizona 1
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
2/31
Evaluation of Website Comprehensibility
Comprehensibility Model Implications ............................................................ 22
Evaluation of Website Comprehensibility ........................................................ 22
Appendix ................................................................................................... 26
Science Taxonomies .................................................................................... 26
Science Content Standards .......................................................................... 26
Science as Inquiry .................................................................................... 26
Physical Science ...................................................................................... 26
Life Science ............................................................................................. 26
Earth and Space Science ........................................................................... 26
Science and Technology ............................................................................ 26
Science in Personal and Social Perspectives ................................................. 26
History and Nature of Science .................................................................... 26
Science Center Exhibits ............................................................................... 27
Bridge of Knowledge ................................................................................ 27
Vivarium ................................................................................................. 27
24-Hour Observatory ................................................................................ 27
Earthworks .............................................................................................. 27
Earth and Planetary Materials .................................................................... 27
Reality Science Exchange .......................................................................... 27
Heath and Wellness .................................................................................. 27
Agriculture and Life Sciences ..................................................................... 27
Library of Congress Science Classification ...................................................... 28
Cybernetics .......................................................................................... 28
Mathematics ......................................................................................... 28
Astronomy ........................................................................................... 28
Physics ................................................................................................ 28
Chemistry ............................................................................................ 28
Geology .............................................................................................. 28
Natural history ...................................................................................... 28
Botany ................................................................................................ 28
Zoology .............................................................................................. 28
Human anatomy ................................................................................... 29
Ray Garcia, June 2006 Confidential, University of Arizona 2
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
3/31
Evaluation of Website Comprehensibility
Physiology ........................................................................................... 29
Microbiology ........................................................................................ 29
Ray Garcia, June 2006 Confidential, University of Arizona 3
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
4/31
Evaluation of Website Comprehensibility
Comprehensibility Project
The University Of Arizona Science Center is conducting a research project with the
UofA Eller School of Management MIS Department to determine whether an
automated process can be used to source and evaluate web site pages for adequate
usage for learning purposes. The learning purposes are focused on self-directed free
choice learners who would use the web as a resource for finding information to
satisfy a learning objective. The automated process will attempt to simulate the
selection process for learning content that has a higher degree of potential
instructional value to the learner due to its ease of access to the main concept of the
information and ease of comprehensibility. The availability of enormous variety of
information on the web presents a challenge to the average person in selecting the
web pages that best suit their learning objectives, skills, and comprehension level.
The research involves the participation of four people who will apply their judgment
in the review and evaluation of web site pages. The evaluation criteria will be tojudge the web pages for comprehensibility of the hypertext information space
presented in the web page. We provide a description of the comprehensibility
criteria to the evaluators prior to conducting the review.
The review process consist of accessing the web site page, finding and reading the
central concept, linking to related pages as necessary to understand the central
concept, and evaluating the web site pages for adequacy for learning or instructional
purposes. The evaluation uses a rating system to rank the web pages relative to the
criteria and relative to other pages reviewed. Approximately eight hundred web site
pages will be reviewed with an average of 10 minutes allocated per web site page.
We are interested in the scanning for the web page site similar to what is typicallydone when browsing the web site when seeking information.
The focus of the website is within the Science, Technology, Engineering, and Math,
(STEM) Domains. The selection of web site pages is based on a typical search
strategy using keywords. This requires the evaluators to have a general interest in
STEM but without any particular expertise in the area. The evaluators would have
the following general profile:
Four people with similar professional profile.
Internet and Media Literate with average daily internet Web Browser usage of
1 - 2 hours or more.
Information literate so that they can discriminate between questionable
information sources and reliable sources.
Adept at selecting internet content for learning activities either for themselves
or for others to use. They should have selected, evaluated, and
Ray Garcia, June 2006 Confidential, University of Arizona 4
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
5/31
Evaluation of Website Comprehensibility
recommended website/pages for others such as the public, parents, friends,
children, peers, etc. that they thought were appropriate.
Active self-directed free choice learners. They have learned something not
taught in school within the last year. They have read a non-fiction book,
visited a museum, watched a documentary, or participated in any free choice
learning activity where they can describe what they learned from the
experience or activity.
Some interest in STEM without any particular expertise in any given area.
They should have internal criteria for what makes a web site suited for
learning that they can express.
We have professional librarians who are not researchers and interface with the
public to select a broad range of appropriate web sites and pages for people to use to
learn about a topic.
The evaluation project was scheduled for July 2006 through August 2006.
The data generated from the project is used to understand what characteristics of a
web page may be more suited for learning insofar as ease of comprehensibility than
other web pages given the same level of interest and motivation in the learning
activity. It may require that we interview the evaluators after the review process to
understand why certain pages where rated higher than others. The specific ratings
provided by each evaluator will not be shared other that to the person providing the
data.
The results will permit the research to more accurately simulate the human
judgment in the selection process and rate pages for comprehensibility moreeffectively. A computational model will be created to simulate the human judgment.
A research paper will be published describing the model and how it was created.
Summary statistics from the rating process and a description of how the data was
generated will be included in the research paper.
Defining Comprehensibility iii
The main interest in comprehensibility is the extraction of the instructional/learning
value of the information content in the web page and the related linked web pages.
The instructional value is a measure of the knowledge contained within the page,
including the degree of application, analysis, synthesis, inquiry or any other narrative
that provokes learning activity. If the information primarily motivates topic interest
in an entertaining manner this is of equal learning value.
Comprehensibility is the degree to which a web page provides direct access to the
substance of the information in the hypertext space without distractions. A
hypertext space is the web page being viewed plus all related linked pages which are
necessary for the reader to visit to understand the information within the related web
Ray Garcia, June 2006 Confidential, University of Arizona 5
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
6/31
Evaluation of Website Comprehensibility
pages. A web page with high comprehensibility will focus on the central concept of
the text narrative and would only use visuals when they are more appropriate for
explaining a concept or audio when it is not spoken language and is relevant to the
concept. A highly comprehensible web page will limit the number of concepts
explained in the text to what can be easily scanned within two page scrolls with each
paragraph building on the concept. The use of the hypertext links are limited to
related concepts that are necessary to understanding the hypertext narrative where
clicking on the link the reader maintains a meaningful context and only optionally is
required to return to the previous page. The reader should easily be able to
determine where they are in the hypertext space relative to where they started.
Comprehensibility encompasses ease of finding and understanding of the concepts
presented assuming that the reading level of the text is equal or less than the
reading level of the evaluator. It does not require that the reader understand all the
text in absolute terms but is able to find meaning and make sense of the text relative
to their framework of understanding or mental map of the topic. If the reader
believes they understood some of the concepts presented this is adequate to
determine the degree of comprehensibility.
Comprehensibility would focus on the main topic with a narrative treatment that is
suited for quick scanning of the web page and uses summary presentations to isolate
the main ideas. The summary may be presented as, the use of short paragraphs
that are easy to scan, use of bullet points, tables, or other presentation that allows
for quick scanning of the information to find the central ideas.
Comprehensibility is evidenced by web page narrative that is suited for short term
memory that allows for ease of linking through and selectively discovering the
meaning relevant to the reader. It is web page narrative that combines both the text
on a single page and the associated pages navigated through links while still
maintaining the context of the related topics.
Comprehensibility of the hypertext space would allow the reader to know how they
navigated through the space and return to any previous page to traverse a different
path. Comprehensibility is enhanced by a site search and site map when a site trail
is not available. Findability of information adds to the comprehensibility.
Comprehensibility is enhanced by consistency and uniformity of the presentation of
the pages on the site. The use of graphical elements only adds to comprehensibility
when it shows information that is more easily presented by the graphical element.The arbitrary use of colors, fonts, background, images distracts from the ease of
reading the text.
The usability and accessibility of the website contribute to comprehensibility. If a
site is not easy to use or cannot be adjusted for accessibility then its
comprehensibility is diminished.
Ray Garcia, June 2006 Confidential, University of Arizona 6
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
7/31
Evaluation of Website Comprehensibility
Comprehensibility puts the readers attention on the most relevant information
quickly and build confidence in the reader that the information has utility for learning
to a degree that the reader will be satisfied that the time spent with the text as
having some value for learning.
Comprehensibility also includes the sense of credibility and trust and access to the
source, author and date of the information being presented. Without trust the
reader is unlikely to try and understand the concept or question it when found.
The opposite of comprehensibility is clutter. Clutter is the presentation of more
than a few related concepts on a single page or the partitioning of the page into
different sections. This is evidenced by Advertising, multiple topics on the same
page, and site navigation links, in areas that present a distraction. Clutter draws the
reader attention away from the central concept of the page or makes it difficult to
find the main idea. Clutter is when the noise on the page is more than 20% of the
page information content. Clutter is when it is difficult to find information and
understand it. Clutter is common on e-commerce and news sites. Clutter includeschatter and ramblings typical of blogs. Clutter includes images that do not add
information to the text or is a concise visual of an idea.
Understanding the Evaluators Expertise
We require a brief professional biography from each evaluator to understand the
level of experience in selecting web content for the general public. The evaluators
should mention any activity that is relevant to the selection of websites such as
participation in social tagging, submitting sites to a virtual library or directory or
search engine, editorial or writing experience, web site development experience, or
any other Internet related experience that would influence their judgment in
selecting websites.
The evaluators will be asked to review this document prior to starting the activity.
They may provide feedback and suggestions to have the document more closely
reflect the criteria that they use in evaluating web site as Librarians.
We may ask the evaluator to optionally participate in a determination of their
Internet competency to establish a comparative level of experience. This may be
through a survey or by an interview process.
The evaluator may optionally participate in determining their preferred cognitive
style using the Cognitive Style Index or simply state whether they believe they are
more analytical or intuitive. The purpose of this information is only to understand
how to interpret the judgment of the ratings.
The ratings are done by the evaluators on a best efforts basis. We would prefer
that the evaluators engage in the rating process in sessions no longer than a few
hours per day to avoid fatigue or skewing of the judgments. If at any time confusion
Ray Garcia, June 2006 Confidential, University of Arizona 7
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
8/31
Evaluation of Website Comprehensibility
over applying the rating criteria occurs we would like to be informed and given the
opportunity to discuss the issue.
The evaluators are allowed to discuss the project with their peers to expand the
professional judgment being applied but only within the first 25 website evaluations.
We want to avoid a shift in the judgment due to outside influences. The first 25
evaluations are considered the training period to practice applying the rating and to
adjust the process or rating if necessary.
Periodic communications during the evaluation process will provide feedback on the
status. Comments provided for particular websites would be discussed to
understand the rating. We are as interested in why a website is disqualified or rated
poorly as we are in why a website is rated appropriate. The rating statistic will not
be discussed with the evaluators during the project to avoid skewing the judgment.
After all the rating is completed we will analyze the data and provide the summarized
results to the evaluators.
We ask the Evaluators to classify the websites, state any prior knowledge, recall the
concepts presented on the website, provide an overall rating, and provide a specific
rating. This should model the selection process that a Librarian uses.
Defining the General Public Audience
The general public is assumed to have an average reading level of the 8th grade.
Reading competency would be sufficient to understand a newspaper or magazine.
The public is assumed to have access to the Internet either at home, work, library,
or public internet kiosk. The content is to be viewed in an Internet Browser and not
a cell phone or other internet enabled device.
Internet and Information Literacyiii is not assumed other than at a very rudimentary
level. The competency level in discriminating selection is assumed to be at a typical
Media Literacy level, which is relatively poor, often biased, and focused on
entertainment. For this audience the Internet may be falsely assumed to be
accurate and not equated to other media outlets. This is the audience that may be
familiar with Yahoo, MSN, AOL and possibly Google. They would use searching
strategies that are limited to single keywords in the search box and may expect that
the results in the first two pages would be satisfactory and valid. They would be
competent enough to use the Internet to find information and extend that activity to
learn about a topic.
The general public is not assumed to have an email account outside of work, buy
from e-commerce sites like Amazon or E-Bay, or use their personal computers for
other purposes. We do not want to bias the study to the top two million frequent
Internet users that are early adopters, know how to create a web page, or work in
STEM, and use computers as a tool. We believe that this population of users may
Ray Garcia, June 2006 Confidential, University of Arizona 8
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
9/31
Evaluation of Website Comprehensibility
also lack appropriate search and information literary skills but they are likely to
overcome any deficiencies through the application of other skills.
Full competency in all the Browser features is not assumed. The typical Internet
user may not be aware of the configuration options for the Browser, parental
controls, history, search options, research, bookmarking, or optional features which
help the user gain control of their browser.
English reading proficiency is presumed for multi-lingual individuals. The websites
selected will be in English and may not have translations or may not be machine
translated without problems. Spanish Language Comprehensibility may be
supported in the future and tools to help English as a Second Language improve
comprehension. These are beyond the scope of this study but we believe that the
findings will be useful for European Languages other than English.
The general public is the audience that is targeted by spamming, phishing, viruses,
chain emails, and other activity that attacks the vulnerable Internet user. These are
the users that may not apply criteria that help them discriminate sources of
information.
Education level or age is not assumed to provide the user with an advantage. Older
professionals who do not need to use the Internet for work may have difficultly in
learning new Internet skills. Younger generations who have a lot of experience with
console games may not be any better at selecting credible and useful content on the
Internet.
Our belief is that the need for Librarians to assist the public in selecting appropriate
sources of information and learning material is even greater with the use of the
Internet across all segments of the population.
STEM Topics
The topics that are included in the evaluation are from Science Technology
Engineering and Mathematics. We have included four taxonomies for consideration,
National Academies Press (NAP), Flaudrau Science Center Exhibit subjects, Library of
Congress, and UNESCO. These taxonomies were chosen for their representative
sampling of STEM topics across a broad range of purposes related to the general
public. NAP publishes books for educators in Science, Engineering, and Medicine.
NAP Science Content Standards is expressed as conceptual phrases for science topics
to cover for each k 12 grade. Flandrau Science Center taxonomy is related to the
types of exhibits that may be designed for the public. The taxonomies from the
Library of Congress and UNESCO represent a view of how STEM information may be
cataloged and used by librarians or researchers. The taxonomies may be used as
sources of keywords used in searching for content. This mimics the expressed
interest by an individual to a librarian who would use the request to determine the
Ray Garcia, June 2006 Confidential, University of Arizona 9
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
10/31
Evaluation of Website Comprehensibility
area to start the search and the associated terms to use. (See appendix for sample
taxonomies.)
Web Site Sources
The sites are selected from Wikipedia, DMOZ.org, and the open web. Wikipediaand DMOZ were chosen due their wide adoption by many search engines as the
encyclopedia and directory sources. While both of these sources are questionable
they reflect the state of the information available on the web and accessed by the
general public. The Wikipedia repository and DMOZ directory are freely available to
any website to reuse therefore much of the content and links appear on many
websites. The repository and directory are managed by volunteers. The volunteers
are not qualified through any certification process. The volunteers are individuals
who have an interest in the topic. They provide text and selections of the websites
within the topic category, accept or reject website submitted by the public, and
summarize a description of the website. The wikipedia entry or directory listing may
have several volunteers working within a topic but generally there are not more than
three volunteer editors and typically only one. The volunteers may post a short
biography of their expertise or interest in the topic but may choose to remain
anonymous. The encyclopedia repository and directory taxonomy of DMOZ have
grown substantially and has many cross links between portions of the websites
organization. The encyclopedia and directory reflects the many ways of organizing
the topics and may appear disorienting at first due its a lack of a consistent audience
perspective and level of competence of the volunteers who created it.
DMOZ represents an organic directory of public websites. It is not as qualified as a
public library directory and therefore would have many more website of questionablevalue. The volunteers do not have consistent criteria for deciding which website is
accepted for inclusion in the directory. DMOZ has enough coverage that most
keyword searches will return a variety of choices of web sites. Similarly, Wikipedia is
organically edited by volunteers without specific editorial control. The quality of
entries varies and the authority of the information cannot be verified. Wikipedia and
DMOZ may be considered the lowest common denominator of web content created
and accessed by the general public. DMOZ has over 100,000 links related to Science
and Wikipedia has over a million pages in its encyclopedia. These are large corpus of
documents that should provide adequate variability in the site content and
construction.
We refer to Wikipedia and DMOZ as sources due to the general public selection of
the content and the links. A public search engine would return many links that
include Wikipedia pages and overlap with DMOZ links as well as content that may be
better or worst than these two sources. The public search engines lack of adequate
filtering of the results is therefore used as the third source of web site pages for
evaluation. A keyword search in a public search engine would return the most
Ray Garcia, June 2006 Confidential, University of Arizona 10
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
11/31
Evaluation of Website Comprehensibility
popular websites based on link structure, not based on information conceptual
relevancy, which results in a lack of adequate filtering necessary to guide the user to
the most qualified source of information. With expert level searching strategy and
persistence it is possible to find quality web sites but this requires competency in
Internet searching and Information Literary neither of which is common within the
general public.
Our site selection goal is to find web site pages that reflect what the general public
would find plus sites that represent the full spectrum of what is available that is
better than the lower common denominator from Wikipedia and DMOZ. We want to
include Web site pages that would be recommended by a Librarian to the general
public interested in a particular topic. We expect that a Librarian may have expert
searching strategies and knowledge of quality sources as well as qualitative
judgment on information that is well beyond the competency of the general public.
We are interested in included web site page that reflect the level of quality that is
representative of a professional Librarian. The Librarians are asked to include web
sites page that may be of poor quality but easy confuse by the general public as a
valid source as well as web site pages that are of the highest information quality.
We prefer to exclude website that may already be included in a virtual library due
to the pre-determine quality of the website. We excluded any sources that are not
freely accessible from any where on the web. Digital libraries and web sites that
require paid subscription are excluded. We assume that the general public could go
to the local library or use the virtual library for their information and instructional
needs. We are interested in helping the general public when they are not accessing
quality directories.
A sample selection of web sites may be presented to the evaluator three times to
test for consistency in the rating judgment. The web sites will be presented within
the normal process and with time delays between the reviews. For example, if a web
site is evaluated on Monday it may appear again on Wednesday. The justification is
to allow enough time in between the reviews, such that the evaluator may not
remember that they already reviewed the site, or recall the ratings they provided
previously.
Website Pre-qualification for General Public Audience
Each selected web site will be pre-qualified for suitability for rating by reviewing the
sites prior to the rating activity and eliminating any sites that are not for the General
Public Audience seeking informational or instructional content. This will allow the
evaluators to focus on only rating web sites that may be of some informational or
instructional value.
The following types of website would be eliminated from the evaluation due to the
purpose of the site not being solely informational or instructional for the general
Ray Garcia, June 2006 Confidential, University of Arizona 11
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
12/31
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
13/31
Evaluation of Website Comprehensibility
would need to use good sense criteria to evaluate the site owners reputation and
the content contained within the web site.
Collaboration or News site used for communications such as groups, blogs,
forums, polls, open wiki, image galleries, email. These sources are available and
known to many Internet users and would not generally be confused with other
sources of information.
Highly Interactive sites without a clear instructional value, includes flash
introduction pages. These sites are difficult to process and tend to focus on
entertainment media.
Inappropriate web sites, includes all sites that would be blocked from further
review due to safe filtering. These included gaming, pornographic, and other
offensive content.
Unknown due to site error, broken links, slow loading, installation of software or
other problem that hinders access to the information. Any sites that essentiallydo not function are eliminated.
Site Selection Process
The web sites are selected by the research team and the librarians using the
Science Content Taxonomies provided. This is to allow for variety in the site
selections across, prior knowledge, interest, preferences, search strategies, and
sources. The four members of the research team will provide 200 sites. The four
Librarians will provide 600 sites each. This will total 800 sites for evaluation by the
Librarians. Only 25% of the sites being evaluated by any given Librarian would have
been submitted by that Librarian.
Each person may use their own search strategy for the highest quality web sites.
To include variety the researchers may follow a set sequence with the intention of
including both quality and poor site examples for the Evaluators. The search process
is as follows:
1. Go through each second level topic the taxonomy and pick a topic to use as
the keyword and associated concepts in the search.
2. Start with Wikipedia and find any pages that directly relate to the concepts.
3. Navigate through the internal and external links from Wikipedia. (It ispossible for the subject not to be represented in Wikipedia in which case just
go to the next step.)
4. As topically relevant sites are found submit them for rating. Include poor
and quality web sites as they are found.
5. Using DMOZ search for sites related to the topic.
Ray Garcia, June 2006 Confidential, University of Arizona 13
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
14/31
Evaluation of Website Comprehensibility
6. Navigate through the DMOZ directory to find topic relevant sites.
7. As sites are reviewed navigate through any links on the page to find topic
relevant pages
8. Using Google, Gigablast.com, Ask.com or another search engine, with only
the keyword find pages.
9. Use advance searching to find pages and include terms like tutorial, learn
and exclude courses, lesson plans, or other terms as necessary
10.Go to any other known sources like Nasa, AMNH, National Geographic
11.Submit sites as they are found. Include variety in the selection with poor
and quality sites represented. Go to the next topic in the taxonomy.
We do not expect either the researchers or the Librarians to spend much time
searching for web sites for each term. The intention is to make a fast determination
of the website and not the rate them during the selection process. We want a
superficial determination of whether the website should be included for rating based
on the classification of the website previously described.
To store the web sites that are found we are using FURL from Furl.net. This is a
small utility that works with most internet browsers to help bookmark websites of
interest. The reason we selected Furl was due to its export feature which will permit
the researchers to load the data into our rating process. Each researcher should
install and test Furl before the site selection process begins.
All the web site pages will be selected prior to starting the rating process. This is to
allow for adequate representation of the topics and quality in the sites selected. The
site list will be reviewed periodically by the researchers in summary prior to the
evaluation to make recommendations to the Librarians in their choices. We want to
avoid a skewing of data based on any preferences or biases or prior knowledge in a
particular topic.
It is possible that not all topics or levels of quality are represented in the open web.
Web content is often developed to satisfy popular topics therefore science topics that
are of special interest may not be adequately represented.
It may be a common practice for Librarians to use multiple sources for information
to cross check facts as a technique to validate the information source. All the
sources used in this technique should be included for evaluation in the selectionprocess.
Ray Garcia, June 2006 Confidential, University of Arizona 14
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
15/31
Evaluation of Website Comprehensibility
Comprehensibility Rating Process ivvvi
The evaluator will login to the project website and select the URL presented. They
will review the website and evaluate it using the criteria described above. Once they
have read/scan/skim the website to a degree that they are confident in their abilityto make a judgment the evaluator will exit the site and complete a survey form. The
survey will ask some background questions, require that the website be categorized,
and rated for quality. The process is repeated with the next URL.
It is possible for the process to be paused at any point and restarted at a later
time. Once a site is selected the evaluator should complete the survey as a
continuous activity without distraction. The start and end time of the website review
is tracked by the application. The total number of pages visited within the site is
monitored.
Site Selection
The sites are pre-selected and presented in a particular order to each evaluator. No
description of the site is provided prior to it being presented. This removes the web
searching activity and the contextual evaluation of search results. Reviewing the site
in this manner is different from normal behavior since it is not typical for one to be
presented with a web page without some indication of what the web site is about.
The rational for this is to force a complete evaluation of the web site on its own
merits. The evaluator will need to scan the website to find the relevant information
and navigate through the website until they are confident that they understand what
is available and can rate the website content and construction adequately.
Review Process vii
The review process should consider the rating questions but only after the web site
is read/scan/skimmed. It is permissible to scan or skim the contents and not read
the contents if the web site is designed for this type of reading. The evaluator
should use the same process they would use if they had a person ask them to find a
web site on the topic, they found the site through some search process, and they
need to make a quick determination as to the appropriateness of the web site to the
persons goal. The assumption is that the topic is under represented and the website
is not available in a Virtual Public Library. The person may have a goal to either find
information about a topic or have a specific learning objective. The evaluator may
assume that the website being recommended is not the only source they would
provide.
Ray Garcia, June 2006 Confidential, University of Arizona 15
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
16/31
Evaluation of Website Comprehensibility
Review Scaffolding
It is acceptable for the evaluator to use reference sites or dictionaries to
supplement their understanding of the topic. The assumption is that the evaluator is
learning about the Science topics and may need some assistance in specific
vocabulary used within the Science Domain. This activity should only be necessaryfor a few words on the website. If the website has links to help understand specific
concepts this should be positive if the Evaluator needs to find other sources to help
understand the content then this should be accounted for in the rating. If sites are
found that are helpful in the scaffolding process the site should be included for
review.
Review Duration
The evaluation should remain within an average of 10 minutes for each site with
not more than 20 minutes for exceptional sites. If a site is commanding too much
time to comprehend this should be indicated in the rating. It is not necessary tostruggle with poor websites. For websites that are exceptionally interesting it also
not necessary to learn about the topic beyond understanding that the information is
of useful for learning. The websites that appear useful but are unusual or unfamiliar
are the websites that warrant time to understand how to rate them accurately.
Rating Duration
The rating should be done quickly under 5 minutes once the evaluator is familiar
with the rating questions. We are interested in the fast judgment applied in a real
setting where the evaluator is making a recommendation to an individual who is
waiting. This is similar to the behavior that one may have for their own searching
where quick judgments are made based on limited information in the search results
and a review of the first web page seen of the website. The difference is in the
expert judgment being applied using consistent criteria with a focus on adequacy and
appropriateness of the information for learning. The assumption is that web sites
and pages that are comprehensible are easy to read and understand and can be
evaluated quickly.
Training Period
The site selection process will have a training period the validate that content exist
for the range of topics in STEM. The training is also to help the Librarians refine an
effective search strategy and filtering of web sites. The research may review the
initial 25 web sites selected by each Librarian to help establish adequate topic and
quality variety in the selection.
The librarian evaluator will rate 25 to 50 web sites as a trial period to understand
the survey process and level set the criteria. They will pause to allow the
Ray Garcia, June 2006 Confidential, University of Arizona 16
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
17/31
Evaluation of Website Comprehensibility
researchers to review the survey data and make any adjustments to the process.
Depending on the consistency of the rating the data may be discarded.
Comprehensibility Survey Questions
Site Classification
The web site is to be classified before rating. Websites that are not suited to the
general public audience as defined in this document are not to be rated. This step in
the process only validates that the web site belongs in the evaluation data.
Options: dropdown, default is General Public Audience Informational or Instructional Site,
Audience Specific Content, Entity Information Sites, Connector Sites, Content Format non-
HTML, and Sites with Restrictions
Reading Speed and Linking
We are interested in how the site is reviewed. Web sites may be skimmed to
understand the structure, scanned to find specific sections, read to find the main
concept, read for understanding, read for curiosity. Each reading technique is
subject to degrees of comprehensibility and therefore would impact the judgment
during the evaluation. We would like the Librarians to review the web sites on its
own merit and when the web site invokes deeper review to pursue their interest in
the topic within the short time period allotted for each web site. This forces
judgment about what is necessary to review for each web site to make an
evaluation.
The web is not a linear medium therefore clicking on links is common and
necessary to understand the information. The degree to which one links through an
web site page and retraces ones paths is an indicator of the information hypertext
space. The clicking through of links is considered part of the reading process for web
site pages.
Options: dropdown, skimmed to understand the structure, scanned to find specific sections,
read to find the main concept, read for understanding, read for interest
Options: dropdown, did not link through, active browsing of the website
Prior Topic or Site Knowledge
Do you have prior knowledge or interest in the topic of the website with well formed
notions or opinions? Prior knowledge does not include coursework in formal
education or general curiosity in the topic. Stable interest with an intrinsic
motivation to learn the topic where the person actively pursued a learning goal is
what we are interested in identifying.
Options: radio button, True or False; Default: False
Ray Garcia, June 2006 Confidential, University of Arizona 17
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
18/31
Evaluation of Website Comprehensibility
Concept Recall
After reviewing the website read the description and website title and think about
whether this adequately capture the concepts you remember reading. The title and
description should be an adequate representation of what the website or page is
about.
Does the site description and directory category not match the site contents?
Options: radio button, True or False; Default: False
Overall Informational or Instructional Rating
General impression of the website based on what was learned and the presentation
facilitating the finding and reading of information and its suitability for the general
public. Is the information easy to understand, useful, and credible? This is a
judgment using intuition based on the content and appearance of the website.
Would the evaluator recommend this site as a source of informational or instructionalcontent to the public?
Options: radio button
1) 1 Poor, 2 Low, 3 Avg, 4 High, 5 Best
Detailed Rating Method
The rating method uses a five point scale with each point contributing to the value
of the overall rating. The scale is from one to five with one indicating that the
specific criteria do not add value and five indicating that it adds maximum value.
These are relative values based on judgment across the sites reviewed.
The ratings are normalized for the evaluator across all the web sites. The
normalized rating is adjusted using the evaluators weighted criteria then compared
to the other evaluators.
Options: radio button
1 Poor, 2 Low, 3 Avg, 4 High, 5 Best
Information Content Rating
Information Value (text content)
Is it easy to find and understand the main topic and concepts in the text? Is it
readable within the comprehension level of the general public? Are the web pages
informative?
Does the website adequately cover the topic? Does it have sufficient useful
information? Is the background and context for the topic provided?
Ray Garcia, June 2006 Confidential, University of Arizona 18
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
19/31
Evaluation of Website Comprehensibility
Is the audience clear? Is it relevant to the audience? Is the information scope
adequate? Do the links help understanding the information? Are terms defined?
Information Credibility (credibility)
Does the content build confidence in its validity (expertise) and can be verified
(trustworthiness)? Does the content appear to have a editorial control?
Is the Authority of information known? Are the sources credible? Are the sources
listed and obtainable? Is the site owner available? Is the author expertise and
credentials listed? Can they be contacted? Is there a discussion of the topic or
opinions of the quality or a rating shown by peer review? Is there accountability for
the information?
Is the information objective, valid, and accurate? Is the information current? Is
the information free of biases or are the biases easy to determine?
Are there any broken links? Is this a small website that looks amateurish? Does it
link to other websites where the information does not look credible? Are their
typographical errors? Is it hosted by a 3rd party such as Geocities, Yahoo, or AOL?
Media Instructional Value (visual design clarity)
Does the use of graphics, icons, animation, or audio enhance the clarity of the
information? Is it necessary to communicate the concepts?
Is the site free of visual clutter? Is the site free of advertisements and unrelated
information? The site does not include popup ads or techniques that distract the
reader. Are the advertisements placed away from the information so that they are
easy to ignore?
Internet Construction Rating
Affective Attention (overall appeal)
Does the website make it easy to focus attention on the most relevant information?
Does the website look professionally designed?
Are the fonts, sizes, colors, background and sectioning of the site easy to read? Is
the content designed to be printed or have an option to have a printed version of the
same content?Do the format, appearance, and aesthetic of the website enhance the information?
Is the web site organization uniform, consistent, and familiar? Is the page heading
and summary representative of the information on the web site?
Ray Garcia, June 2006 Confidential, University of Arizona 19
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
20/31
Evaluation of Website Comprehensibility
Organization (structure, navigation)
Is the content presented for reading on the web? Does it make appropriate use of
short paragraphs, list, tables, headings, and links, to focus easily on the main ideas?
Is it easy to scan or skim?
Are the links related to the topic and make it easier to understand the information?Are the links clearly organized? Are the navigational links separated from the links
within the narrative? Are the links to external sites relevant to the topic and clearly
indicated? Are links to media files indicated?
Usability (functionality, interactivity)
Do the Web site pages load quickly? Is the site easy to use? Does it have a search
feature and site map or menu that is organized? Is it easy to find the home page?
Does it provide an indicator of where the web page is relative to the site navigation?
Is the back button feature operable?
Does the site adequately support usability and accessibility?
If the website is highly interactive does the activity enhance the informational or
instructional value of the website? Is the interactivity necessary or the easiest
means of communicating the concept?
Post Evaluation
Comment on the evaluation process if necessary to explain an exceptional rating
and characteristic that was not included in the rating questions but was an important
factor in the evaluator criteria.
A calculation of the rating may be done and compared to the initial general
evaluation. If an inconsistency is found a re-evaluation may be done to correct for
the judgment. If some factor accounted for the discrepancy then a comment would
need to be provided to explain the difference.
Processing Notes
The survey time should be less than the time used to review the website. The
evaluation survey facilitates the rating process by displaying criteria questions the
evaluator should think about before providing the rating.
The evaluator is assumed to have an adequate comprehension level to review
general web pages on Science and Technology, and enough interest in the subject
matter to be motivated to find the central concept and attempt to understand it.
Ray Garcia, June 2006 Confidential, University of Arizona 20
8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective
21/31
Evaluation of Website Comprehensibility
Rating Matrix
The rating matrix established the internal criteria of each factor considered in the
overall rating. It indicates when a factor is dependent on others or if it may be
considered independently. The relative value of each factor is proposed. This matrixis used to weight the ratings against each other to emphasize one factor more than
another.
Each rating indicates the following:
D = Dependent
I= Independent
L = Familiarity may overcome hurdle, can learn to ignore factor
C = Constant Factor no matter how often it is reviewed
< > = Relative importance between the factors
IV IC MV AA SO UA
Information Content Rating
Information Value IV I C < > > < >
Information Credibility IC I C > > > >
Media Instructional Value MV I C >