Embed Size (px)
DESCRIPTION
Web accessibility for people with disabilities is a highly visible area of research in the field of ICT accessibility, including many policy activities across many countries. The commonly accepted guidelines for web accessibility (WCAG 1.0) were published in 1999 and have been extensively used by designers, evaluators and legislators. W3C-WAI published a new version of these guidelines (WCAG 2.0) in December 2008. One of the main goals of WCAG 2.0 was testability, that is, WCAG 2.0 should be either machine testable or reliably human testable. In this paper we present an educational experiment performed during an intensive web accessibility course. The goal of the experiment was to assess the testability of the 25 level-A success criteria of WCAG 2.0 by beginners. To do this, the students had to manually evaluate the accessibility of the same web page. The result was that only eight success criteria could be considered to be reliably human testable when evaluators were beginners. We also compare our experiment with a similar study published recently. Our work is not a conclusive experiment, but it does suggest some parts of WCAG 2.0 to which special attention should be paid when training accessibility evaluators
Citation preview
On the testability of WCAG 2.0 for beginnersFernando Alonso, José Luis Fuertes, Ángel Lucas González, Loïc Martínez
DLSIIS. Facultad de Informática. Universidad Politécnica de Madrid. Spain
7th International Cross-Disciplinary Conference on Web Accessibility
W4A2010
W4A2010
W4A2010
Contents
On the testability of WCAG 2.0 for beginners
Introduction
WCAG 2.0 Overview
Testability in WCAG 2.0
The experiment
Results of the experiment
Discussion and conclusions
I
W
T
E
R
D
W4A2010
On the testability of WCAG 2.0 for beginners
Introduction
Special attention to Web accessibility
Access to ICT is key in providing access to
our society
WCAG 1.0 (1999)
Experiment on WCAG 2.0 testability for beginners (students in an intensive
course)
I
W
T
E
R
D
WCAG 2.0 (2008)
GoalsTechnology-independent &
testable
W4A2010
W4A2010
On the testability of WCAG 2.0 for beginners
Informative
Testable sentences
Basic goals for web designers
Foundation for web accessibility Principles
Guidelines
Success Criteria
Sufficient techniques
Advisory Techniques Failures
I
W
T
E
R
D
WCAG 2.0
W4A2010
On the testability of WCAG 2.0 for beginners W4A2010
Testability in WCAG 2.0
• Each criterion is written as a statement that will be either true or false when specific Web content is tested against it
Success criteria are testable
• Machine testable• There is a known algorithm to determine whether a
technique has been implemented • Reliably human testable
• Can be tested by human inspection• At least 80% of “knowledgeable” evaluators agree
Definition of testability (techniques)
I
W
T
E
R
D
W4A2010
On the testability of WCAG 2.0 for beginners W4A2010
Factors related to testability
Objectivity of the language
used
Clarity of the language used
Openness of techniques and failures
I
W
T
E
R
D
W4A2010
On the testability of WCAG 2.0 for beginners
The context for the experimentI
W
T
E
R
D
• One week intensive course (ATHENS programme). March 2009.• Contents: introduction to functional diversity and design for all,
W3C-WAI, WCAG 2.0 (jigsaw sessions), evaluation of accessibility
• Assessment: participation in the jigsaw sessions and one exercise (accessibility evaluation)
The course
• Evaluate one web page• Lack of tools: manual evaluation (using a spreadsheet)• Focus on the 25 level-A success criteria• Values for each SC: pass, fail, partial, N/A, unknown
The exercise
W4A2010
On the testability of WCAG 2.0 for beginners
The web pageI
W
T
E
R
D
W4A2010
On the testability of WCAG 2.0 for beginners
The experimentI
W
T
E
R
D
• Results provided by the 17 students• Merging partial with fail• In 4.1.2 there were two “unknown” values (we ignored them)
• Compared with ratings provided by two instructors• Had to agree on 12 SC!
The data
• Are the 25 level-A success criteria “reliably human testable”?• 80% agreement• The agreed value has to be the correct one
Goals
W4A2010
On the testability of WCAG 2.0 for beginners
ResultsI
W
T
E
R
D
0 5 10 15
1.1.1 Non text content
1.2.1 Audio-only and Video-only
1.2.2 Captions
1.2.3 Audio Description
1.3.1 Info and Relationships
1.3.2 Meaningful Sequence
1.3.3 Sensory Characteristics
1.4.1 Use of Color
1.4.2 Audio Control
2.1.1 Keyboard
2.1.2 No Keyboard Trap
2.2.1 Timing Adjustable
2.2.2 Pause, Stop, Hide
2.3.1 Three Flashes
2.4.1 Bypass Blocks
2.4.2 Page Titled
2.4.3 Focus Order
2.4.4 Link Purpose
3.1.1 Language of Page
3.2.1 On Focus
3.2.2 On Input
3.3.1 Error Identification
3.3.2 Labels or Instructions
4.1.1 Parsing
4.1.2 Name, Role, Value
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
Fail
0 5 10 15
1.1.1 Non text content
1.2.1 Audio-only and Video-only
1.2.2 Captions
1.2.3 Audio Description
1.3.1 Info and Relationships
1.3.2 Meaningful Sequence
1.3.3 Sensory Characteristics
1.4.1 Use of Color
1.4.2 Audio Control
2.1.1 Keyboard
2.1.2 No Keyboard Trap
2.2.1 Timing Adjustable
2.2.2 Pause, Stop, Hide
2.3.1 Three Flashes
2.4.1 Bypass Blocks
2.4.2 Page Titled
2.4.3 Focus Order
2.4.4 Link Purpose
3.1.1 Language of Page
3.2.1 On Focus
3.2.2 On Input
3.3.1 Error Identification
3.3.2 Labels or Instructions
4.1.1 Parsing
4.1.2 Name, Role, Value
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Pass
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
Fail
The majority of students gave an incorrect response in 9 success criteria!
W4A2010
On the testability of WCAG 2.0 for beginners
Results: reliabilityI
W
T
E
R
D
13 success criteria are not reliably evaluated with a 64% threshold!
W4A2010
On the testability of WCAG 2.0 for beginners W4A2010
Results: insufficient majority (<64%)
• There is a non-interactive flash-based animation• Students failed to indentify it as “video-only” content
1.2.1 Alt to audio-only & video only (58%)
• Some did not think of tabindex as a way of programmatically determine the sequence• Some failed to identify that sequence affected the meaning
1.3.2 Sequence for reading content (53%)
• Some students failed to detect that one function (the link of the flash animation) was unreachable with the keyboard
2.1.1 Keyboard access (58%)
• Some considered that the SC passed although there were no time limits• Some considered that the animation did set time limits
2.2.1 User-adjustable timing (53%)
I
W
T
E
R
D
W4A2010
On the testability of WCAG 2.0 for beginners W4A2010
Results: wrong majority
• Students failed to indentify the animation as video content that required audio-description
1.2.3 Audio-description (0%)
• A few failed to identify that the page contained info, structure and relations…• The majority thought that the SC passed although the headings were wrong
1.3.1 Information, structure and relationships (12%)
• The students failed to identify that some links were distinguished only by small differences in color
1.4.1 Color use (23%)
• There is a link “skip navigation” but it is not well implemented. Students failed to detect this.
2.4.1 Bypass groups of content (35%)
I
W
T
E
R
D
W4A2010
On the testability of WCAG 2.0 for beginners W4A2010
Results: wrong majority
• Only one discovered that tab order did not preserve meaning and operability
2.4.3 Navigation focus order (6%)
• The language is identified… but it is not the right language
3.1.1 Natural language of the page (47%)
• The user could search for the default value “search” with no validation
3.3.1 Error identification (29%)
• In the search form there is a label and a button, but some students said it was not enough
3.3.2 Labels or instructions (18%)
• The form had incomplete internal information that was not detected by the students
4.1.2 Name, role, value (18%)
I
W
T
E
R
D
W4A2010
On the testability of WCAG 2.0 for beginners W4A2010
Results: reasons for lack of reliability
I
W
T
E
R
D
SC Category Comp. Knowl. Effort
1.2.1 Insufficient X
1.2.3 Wrong X
1.3.1 Wrong X X
1.3.2 Insufficient X X
1.4.1 Wrong X
2.1.1 Insufficient X
2.2.1 Insufficient X
2.4.1 Wrong X X
2.4.3 Wrong X X
3.1.1 Wrong X
3.3.1 Wrong X X
3.3.2 Wrong X
4.1.2 Wrong X
W4A2010
On the testability of WCAG 2.0 for beginners
Discussion: Brajnik’s experimentI
W
R
E
R
D
• Brajnik, G. “Validity and Reliability of Web Accessibility Guidelines” Proceedings of ASSETS’09 (October 25-28, Pittsburg, Pennsylvania, USA). ACM Press. 2009. 131-138.
Reference
• More evaluators (35)• Two web pages• Evaluated both WCAG 1.0 and WCAG 2.0
• We have a common subset of success criteria• The evaluators had to assess how difficult it was to decide
applicability and value
Differences
W4A2010
On the testability of WCAG 2.0 for beginners
Discussion: percentage of agreementI
W
R
E
R
D
5 success criteria particularly weak in both experiments!
W4A2010
On the testability of WCAG 2.0 for beginners
ConclusionsI
W
R
E
R
D
• 13 success criteria with low reliability• Similar global conclusion reached by Brajnik• Reasons for lack of testability: comprehension, lack of
knowledge and lack of effort
WCAG 2.0 is not testable in our experiment!
• Low number of students• Short training time• Only one page
Limitations of our experiment
W4A2010
On the testability of WCAG 2.0 for beginners
RecommendationsI
W
R
E
R
D
• 7 success criteria• Could lead to rewriting proposals, but we need more experiments
Comprehension
• 4 success criteria• Will impact on how we teach WCAG 2.0
Lack of knowledge
• 7 success criteria• It was an intensive course• We need to increase motivation
Lack of effort
W4A2010
On the testability of WCAG 2.0 for beginners
Future work: more experimentsI
W
R
E
R
D
0102030405060708090
1001.
1.1
Non
tex
t con
tent
1.2.
1 A
udio
-onl
y an
d …
1.2.
2 Ca
ption
s1.
2.3
Aud
io D
escr
iptio
n1.
3.1
Info
and
Rel
ation
ship
s1.
3.2
Mea
ning
ful S
eque
nce
1.3.
3 Se
nsor
y Ch
arac
teri
stics
1.4.
1 U
se o
f Co
lor
1.4.
2 A
udio
Con
trol
2.1.
1 Ke
yboa
rd2.
1.2
No
Keyb
oard
Tra
p2.
2.1
Tim
ing
Adj
usta
ble
2.2.
2 Pa
use,
Sto
p, H
ide
2.3.
1 Th
ree
Flas
hes
2.4.
1 By
pass
Blo
cks
2.4.
2 Pa
ge T
itled
2.4.
3 Fo
cus
Ord
er2.
4.4
Link
Pur
pose
3.1.
1 La
ngua
ge o
f Pag
e3.
2.1
On
Focu
s3.
2.2
On
Inpu
t3.
3.1
Erro
r Id
entifi
catio
n3.
3.2
Labe
ls o
r In
stru
ction
s4.
1.1
Pars
ing
4.1.
2 N
ame,
Rol
e, V
alue
ATHENS 2009
ATHENS 2010
CS Eng.
Thank you for listening!
Loïc Martínez-Normand ([email protected])
7th International Cross-Disciplinary Conference on Web Accessibility
W4A2010
