Evaluation of the NSDL and Google for Obtaining

Pedagogical Resources

Frank McCown, Johan Bollen, and Michael L. Nelson

Old Dominion UniversityComputer Science Department

Norfolk, Virginia, USA

ECDL 2005September 21, 2005

Useful K-12 Educational

Content

The Entire Web

ECDL 2005 7

Related Work

Many studies compare web search engines’ ability to find “relevant” documents

Human evaluators (typically students) are used Top 10 or 20 results are evaluated None have used NSDL or evaluated educational

usefulness of search results Sumner said educators expect DLs to save them

time over using web search engines because DLs filter their content1

1T. Sumner et al.: Understanding Educator Perceptions of “Quality” in Digital Libraries, (JCDL 2003)

ECDL 2005 8

Virginia Standards of Learning

BIO.3 The student will investigate and understand the chemical and biochemical principles essential for life. Key concepts include

a) water chemistry and its impact on life processes;

b) the structure and function of macromolecules;

c) the nature of enzymes; …

http://www.pen.k12.va.us/VDOE/Superintendent/Sols/home.shtml

ECDL 2005 11

Evaluators

ECDL 2005 12

Query Ratings

Were the search terms well chosen?

1 = Strongly agree

2 = Agree

3 = Neutral

4 = Disagree

5 = Strongly disagree

Average rating 2.08 Median rating 2

ECDL 2005 13

Link Ratings

Were the search results useful for educating a student about the Learning Statement?

1 = Strongly agree2 = Agree3 = Neutral4 = Disagree5 = Strongly disagree6 = N/A

Wilcoxon signed rank test revealed a statistically significant difference (p < 0.05) between the NSDL and Google ratings.

ECDL 2005 14

Calculate Precision

Google’s precision = 145/380 = 38.2%

NSDL’s precision = 57/334 = 17.1%

results search of number total

ratings 2 or 1 of number precision

ECDL 2005 15

Search Result Ratings

All domains

ECDL 2005 16

Search Result Ratings

Median ratings for each domain

ECDL 2005 17

Ranking of Search Results

How well did Google and NSDL rate their search results?

Performed Spearman Rank correlation between rankings and ratings

Google: rho=0.125, p=0.001 ** NSDL: rho=0.057, p=0.173

ECDL 2005 18

How Did this Happen?

Why did Google perform better than NSDL? Google has more stuff

8 billion pages vs. a few million? But Google’s stuff is not screened for educational

quality Google and NSDL show different results

38 queries 6 duplicate results in the top 10 results

25% of NSDL results not indexed by Google

ECDL 2005 19

How Did this Happen?

More NSDL results are inaccessible or login-protected (NSDL = 9.3%, Google = 5.3%) 81% of inaccessible NSDL scores were from

http://nsdl.lon-capa.org/ Some NSDL resources were not for K-12

students 17% of NSDL results were from http://arXiv.org/

ECDL 2005 20

Improvements for NSDL

Provide advanced search features Provide the ability to target the grade level

appropriateness of information Rank results based on relevance

ECDL 2005 21

Thank You

Questions?Evaluation results, data files, slides:

http://www.cs.odu.edu/~fmccown/research/nsdl_google/