Upload
alexia-hunt
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
Tried and Tested?Evaluating Data Entry Consistencyin the Component Model Prototype
Brit Hopmann, July 1, 2008
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Presentation
• Background & aim consistency check
• Images & data
• Analysis & results (in 3 steps)
• Problems & possible solutions
• Conclusion & discussion
data entry
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Background & aim consistency check
Background
• Problems with data entry ambiguity
• Consequences: decreased recall and precision
Aim
• Check and improve data entry consistency
• Solutions for existing problems?
• Changes?
data entry
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Images
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Data
• 5 images
• 25 features (+ 2 discarded)
• 6 respondents
Analysis in 3 steps
• Categorizing the answers (per image / per feature)
• Analysis per image and per feature
• Problems and possible solutions
Analysis
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 1
example: answers for image4
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 1
Categorizing the answers (per image / per feature)
3 categories
category consistency
times percentage
no problem 57 45,6%
indecisive 16 12,8%
problem 52 41,6%
There were many problematic descriptions.
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 2
Summary per watermark (5 images)
None of the images was described unanimously.
image no prob ≈ problem
per image
image 1 15 2 8 fine
image 2 13 3 9 fine
image 3 10 3 12 problematic
image 4 11 3 11 indecisive (≈)
image 5 8 5 12 problematic
total 57 16 52
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 2
Summary per feature (25 features + 2 discarded)
Described consistently 5-7 features*
3 of 5 images not consistently described 10-8 features*
* Depending on quality of definitions
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 3
How problematic are specific features?
1. Is or might the feature be problematic with other watermarks?
2. What caused the problem and can it be solved easily?
-> image quality, definition, subjective description
3. What is the distinctive value of a particular feature?
4. Is there a possible solution?
5. Which problems will remain problematic after having applied a specific solution?
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 3
What could contribute to a solution?
• Grouping
• Allow assigning multiple features
• Drawback: increased decision effort
• Extra tool
• For instance graphical indicator of angle
• Drawback: exceptions such as rotated scans
• Clarify definitions
• For instance description of additional elements
• Relatively straightforward to apply
• Otherwise
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Crucial Question
Which features will still pose a problem with a particular solution applied?
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Conclusion
Problems that will eventually remain
• Eyes: alignment, distance, position
• Horns: endings, orientation, inside
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Conclusion
Remaining problems
• Ears: endings, roundness, width
• Eyes: shape
• Head: shape
• In-between horns: shape
• Nose: concomitance with other elements, shape
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Discussion
Features that evoke subjective descriptions
(shape!) cause problems
Thinking about possible changes in the model
• Clarify certain issues: clear definitions, instruction
• Expand model: extra tool, grouping
• Simplify model: adjust or take out features
data
Reconsider role of component model within Bernstein
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Discussion
Questions and Comments?
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?