1 Michigan Teacher Network michiganteacher.net Transaction Log Analysis & Michigan Teacher Network Marcia A. Mardis Merit Network at the University of

1

Michigan Teacher Network • michiganteacher.net

Transaction Log Analysis & Michigan Teacher Network

Marcia A. MardisMerit Network at the University of

Michigan

2


Working Definitions User centered design is a methodology that concentrates on product

design and development from user, user task, and user interface perspectives. A UCD process functions in parallel with a product development process to:

Collect Data: About users, their tasks, and their current work processes. Analyze Data: To determine how to redesign the workflow, and to identify usability

requirements and goals. Design and Develop: Scenarios, information architectures and grammars, and user

interface prototypes. Test: Design and conduct iterations of usability assessments and tests. Deploy: Follow up with measures of usability and user satisfaction

Transaction log analysis (TLA) uses transaction logs to discern attributes of the search process, such as the searcher’s actions, the interaction between the users and the system, and the evaluation of the results by the searcher. (Jansen & Pooch, 2000).

3


What can you do with TLA?

Characterize user sessions Length of time and time of day Number of pages viewed Where coming from

Characterize of user queries Number of revisions Number of terms used Use of logic and advanced search techniques

Characterize search terms Topic Frequency of terms

4


Key Authors & Studies

Spink, Jansen, Wolfram & Saracevic Excite studies Meta-evaluation Frameworks, general taxonomies

Bryan-Kinns & Blandford Digital library TLA (vs. Web TLA) Interactional traps (misunderstanding; premature bailout)

Silverstein & Henziger AltaVista search logs Comparison of theoretical vs. actual user models

Abbas Teachers & Children’s info seeking (her own work and lit review) Influence of metadata and search scaffolds

5


Consistent Findings from Studies

Very few queries use advanced search features like boolean; half of those that do contain mistakes

Highly skewed use of terms: few used a lot, a lot used infrequently

Distribution of terms bears no relation to distribution of content

Users look mainly at the first page of results only

Most sessions contain only 1 query Most queries are very short (3 terms or less)

6


Caveats

Are the exhibited searching characteristics due to some unique aspect of the system or collection?

Are the user samples from the same or different populations?

Are the searching characteristics really applicable to all users?

What are the implications of users who do not understand what they are searching (metadata vs. full-text)

What is the tradeoff between user privacy and data collection?

Real searchers have different information seeking processes Are the topic classification schemes used consistently? Is the data collection tool accurate?

7


What are information seeking

behaviors? Actions and expressions exhibited

(recursively) during: Query formation

Topic and concept identification Keyword generation

Information collection Source selection Source navigation (Searching/browsing)

Results evaluation Quality reflection Query revision Source reselection (choose another tool)

8


Why is it important to look at these

behaviors? For tool developers

User interface design Training/Help content and design Knowledge of where need to ask questions might arise

For questioners Understand where asking questions may be integrated into

information seeking process

For responders Phrasing and types of answers given Understanding of barriers to use

9


What’s consistent in the literature?

Time is consistently a big factor in teacher information seeking (Fitzgerald, 2001; Pittsley, 2000, etc.)

Teachers say quality is important (Sutton, 1999; Fitzgerald, 2001; Mardis, 2003, etc.)

Teachers don’t take time to evaluate quality (Lazonder, 2001)

Teachers are willing to interact (Fitzgerald, 2001; Shumar, 2003; Renninger, 2003)

Teachers consult colleagues (Sutton, 1999; Mardis & Hoffman, 2003, etc.)

10


Higher ed faculty and K-12 teacher

behaviors: compared Instructors Primary research (use of email, discussion groups) fieldwork

data, archival materials, and gray literature Citation rhetoric (Bates, 2001) Very little instruction on using Web for course resources

Teachers Information tends to be for self-generated queries Teachers adapt resources (Sutton, 1999; Fitzgerald, 2001) Teachers use each other as primary sources of information

11


Adults and K-12 teacher behaviors:

compared Topic familiarity determines amount of time

spent searching and reading. (Kelly & Cool, 2002)

Both groups are not familiar with search engine structure and function (Bradley, 1999)

No structured search models exist for either group

12


Children and K-12 teacher behaviors:

compared Little extensive, controlled study has been

done (Dresang, 1999) Children overperceive their abilities to use

search tools; teachers bail out more quickly (Dresang, 1999; Fidel, 1999)

Both groups excel in systems designed for them (Hirsh, 1997; Dresang, 1999)

Younger students and teachers favor browsing (Dresang, 1999)

13


What are some key ideas?

Save time Describe richly (Sutton, 1999) Quick response is essential Recommend ready-to-use resources with clear application and

evaluation/assessment Assure quality (rating/peer review) (anecdotal) Question formation is still weak link Don’t assume technology skills can/will be

gained(Weglinksy, 2000) Users need a lot of support (documentation, online help)

(Pollock, 1999)

14


About Michigan Teacher Network

(MTN) Founded in 1998 as state clearinghouse Contains over 8500 professional learning,

professional support, and curriculum resources correlated to the Michigan Curriculum Framework

Receives additional funding from the NSF NSDL Receives 500,000 (summer)-700,000(fall/winter)

page views per day Executes 750 (summer)-950(fall/winter) search

strings per day Governed by Merit privacy policy

15


Research Questions

What do MTN users look for? Do MTN users look for classroom planning

information at home? Do users use too many or too few search

terms? Do users seem to understand what they’re

searching?

16


Characteristics of Windows Examined

2/19-3/5 3/26-4/9 4/29-5/14

N 10870 7462 5896

Mean Searches per day 776 530 420

Mean words per string 2.48 1.97 1.99

Mean time of search 3:12pm 2:40pm 2:38pm

17


Broad Category of Searches

Search Category

3000

2000

1000

0

Data collected from April 29-May 14, 2003

18


Top Searches 2/19-3/5/2003

classroom management 300

sciences 102

lesson plans 90

sound 87

states 91

history 72

social studies 71

economics 64

elementary education 64

technology education 58

19


Top Searches 3/26-4/9/2003


physical education 60

lesson plans 55

educational assessment 51

fractions 47

private schools 41

weather 39

rainforests 35

music 34

life science 31

20


Top Searches 4/29- 5/14/2003


listserv(s) 146

poetry of kids 109

science 48

lesson plans 41

private schools 37

zookeeper 31

plants 30

plagiarism 28

organizational skills 27

21



Curriculum Area Searches

Curriculum Area

800

600

400

200

0

22


1

3


2

23


Conclusions

Many users exhibit “Search Engine Habits” Search for items beyond scope of collection Seem to be expecting to search site text, not metadata Perform searches on browsing categories

24


Conclusions

Most searches occur after school MTN users seek classroom management help

the most Science is the most searched curriculum area Users search very broad or very narrow

concepts with few terms Users seem to want objects Users repeatedly type in the same search

25


Users misunderstand sometimes…

Ozzy Osbourne Motown Ho Down Health priorities in Australia Bay City Boating and Fishing Recipes (food) Spices and herbs Buying a car SARS Mazda rx-8 Bush declares war

Documents

1 Michigan Teacher Network michiganteacher.net Transaction Log Analysis & Michigan Teacher Network Marcia A. Mardis Merit Network at the University of