1 QA Focus – Supporting JISC's Digital Library Programmes Quality Assurance For Digital Library Programmes: A Practical Workshop Brian Kelly UK Web Focus

QA Focus – Supporting JISC's Digital Library Programmes1

Quality Assurance For Digital Library Programmes:A Practical Workshop

Brian Kelly

UK Web Focus

UKOLN

Email: [email protected]

http://www.ukoln.ac.uk/qa-focus/http://www.ukoln.ac.uk/qa-focus/

Workshop Aims • Learn about QA for

digital library work• Hear about QA

Focus's activities• Identify QA

requirements in metadata & service deployment

• Work on solutions• Agree on follow-up

actions

Workshop Aims • Learn about QA for

digital library work• Hear about QA

Focus's activities• Identify QA

requirements in metadata & service deployment

• Work on solutions• Agree on follow-up

actions


Timetable

14:00 Why Do We Need QA? / What Is QA Focus? [BK]

14:20 What Can Go Wrong In Your Project? [ALL]

14:50 Report back [Group Reporters]

15:15 Coffee

15:30 The QA Focus Approach [BK]

15:45 QA For Metadata – An Example [MG]

16:05 Embedding QA in your Project [ALL]

16:30 Report Back [Group Reporters]

16:40 What Next? (Get some answers and win a prize!)

17:00 Finish (?)


About Us

The workshop organised by:• QA Focus – a JISC-funded advisory service

provided by UKOLN and AHDS which supports JISC's digital library programmes (5/99, and now FAIR and X4L)

• Brian Kelly, QA Focus project manager (and UK Web Focus)

• Additional support provided by: Marieke Guy (UKOLN): formerly QA Focus

officer and now SPP and E-Prints UK project manager

Gareth Knight (AHDS): QA Focus officer

Note Amanda Closier (UKOLN) will be working on QA Focus from January

Note Amanda Closier (UKOLN) will be working on QA Focus from January


Aims of the Session

The aims of the QA session are:• To provide participants will an awareness of the

importance of quality assurance (QA) in ensuring that project deliverables are (a) functional (b) interoperable (c) widely accessible and (d) re-purposable

• To find out about QA Focus's resources and gain an understanding of the QA Focus methodology

• To make use of the QA Focus methodology in an area of your work

• To have an opportunity to look and evaluate QA Focus resources

• To agree on areas to be addressed by projects


Why Do We Need QA?

Why Is QA Needed Now?In the past JISC had standards-based philosophy but:

• Approach of "let a 1,000 flowers bloom"• Diversification as no killer application then

(it could have been Gopher!)

Things are different now:• The Web is the killer application• There is now a need for interoperability (not just self-

contained independent Web sites)• JISC Programmes increasingly intended as service

development and not 'speculative' research• There is a need for publicly-funded services to be

seen to provide value for money

Intr

od

uct

ion


Technical Areas Addressed

The areas addressed by QA Focus are:• Standards• Digitisation• Access (Web)• Metadata• Software • Service Deployment

The areas of standards, digitisation and access/Web have been addressed and a wide range of advice is currently available.In this session we will focus on the areas of metadata, software and service deployment

The areas of standards, digitisation and access/Web have been addressed and a wide range of advice is currently available.In this session we will focus on the areas of metadata, software and service deployment


Group Exercise 1

• Split into a number of small groups• Choose a reporter• Address the question "What can go wrong with

my project?"• The focus should be primarily on technical

challenges (i.e. not 'all the staff leave')• Address primarily the areas of metadata, software

and service deployment

E


Report Back – Exercise 1

What can go wrong with your project?What have you identified?


Coffee Break

We should now break for coffee


What Can Go Wrong?

We have previously identified the following:• A lack of understanding of appropriate standards

and best practices• A failure to make use of appropriate implementation

architectures• A lack of awareness of failure to comply with

appropriate standards and best practices• A lack of awareness of checking & validation tools

and procedures (and their limitations)• Using quality control rather than implementing

quality assurance procedures

QA Focus is addressing these high level areas

QA Focus is addressing these high level areas


Approach Taken By QA Focus

Two possible approaches to avoiding problems:

Enforce• Inspect all project’s work• Strict auditing, with penalties for no-compliance

Encourage• Developmental:

Working with communityExplaining reasons for complianceDocumenting examples of best practicesProviding advice on implementation & monitoringDeveloping a QA methodologySeeking to embed QA in working practices

Intr

od

uct

ion

The latter approach is felt to be preferable, especially in a HE context

The latter approach is felt to be preferable, especially in a HE context


QA Focus Activities

QA Focus activities:• Focus groups identified key concerns (in areas of

standards, implementation and service deployment)• Surveys profiled approaches, best practices and

common problems• Provision of focussed advice primarily addressing:

Standards and best practices Implementation issues Compliance checking

• Case studies describing examples of best practices• Development of a QA methodology• Validation of approach • Embedding QA methodology within projects


Survey FindingsSurveys of compliance of 5/99 project entry points helped to profile community and identify common problems and best practices

Su

rvey

s


Documentation: AdviceA

dvi

ce

Advisory briefing documents are being producedThese are:

• Brief, focussed documents

• Informed by findings of the surveys

• Over 50 briefing documents have been published to date

Advisory briefing documents are being producedThese are:

• Brief, focussed documents

• Informed by findings of the surveys

• Over 50 briefing documents have been published to date


Documentation: Case Studies

Case Studies have been commissioned

These are:

• Written by projects themselves (helping to achieve 'buy-in')

• Describe the solution adopted to a particular problem

• Include details of lessons learnt – not just a press release!

• Over 30 published

Case Studies have been commissioned

These are:

• Written by projects themselves (helping to achieve 'buy-in')

• Describe the solution adopted to a particular problem

• Include details of lessons learnt – not just a press release!

• Over 30 published

Ad

vice


Surveys For FAIR & X4L Programmes

QA Focus now supports JISC's FAIR and X4L programmesWe have developed a self-assessment interface for projects

http://www.ukoln.ac.uk/qa-focus/surveys/fair/web-self-assessment/

http://www.ukoln.ac.uk/qa-focus/surveys/fair/web-self-assessment/

QA

Fo

r W

eb

The aim is that projects will (a) check and, if necessary, fix their home page; (b) check/fix other key pages and (c) address underlying causes of problems

"I have found the webpage very useful for testing the technical compliance and accessibility of our website""Many thanks for this - what a really useful tool!"

"I have found the webpage very useful for testing the technical compliance and accessibility of our website""Many thanks for this - what a really useful tool!"


Putting it Together

We have:• Explained why projects should implement standards

and best practices (they don't always know)• Provide case studies on approaches by projects• Described implementation architectures• Described various checking tools

But how do we ensure projects :• Check deliverables for themselves • Have systematic checking procedures

There is a need to describe:• Tools & architectures for projects for check

compliance with standards & best practices• Embed systematic QA procedures

But how do we ensure projects :• Check deliverables for themselves • Have systematic checking procedures

There is a need to describe:• Tools & architectures for projects for check

compliance with standards & best practices• Embed systematic QA procedures

To

ols


The QA Focus Methodology

The QA Focus methodology which we are advising projects to deploy:

• Based on well-established QA principles• Seeks to be lightweight and achievable• Seeks to benefit the projects themselves, as well as

interoperability benefits to JISC programmes

Consists of:• Documented policies• Procedures for monitoring compliance with policies• Audit trails• Sharing and collaboration

QA

Fo

cus

Met

ho

do

log

y

See <http://www.ukoln.ac.uk/qa-focus/documents/briefings/briefing-30/>


Applying QA – Web Standards

Policy: The Web site will use XHTML 1.0 and CSS 2.0 standards

Architecture: The Web site will be based on XHTML templates and SSI

Monitoring: New and updated pages validated using ,validate and ,cssvalidate. Every month ,rvalidate will be used

Exceptions: HTML derived automatically (e.g. Save As HTML in PowerPoint) need not comply with standards. The files will be stored in a standard directory to enable such files to be excluded from checks

QA

Fo

cus

Met

ho

do

log

y


Applying QA – Accessibility (1)

Policy: The Web site will strive to attain WAI A guidelines.Consistent accessibility shortcuts will be used.An accessibility policy will be published.

Architecture: The Web site will be based on XHTML templates which comply with WAI A.

Monitoring: New and updated pages will be validated using ,bobby. A monthly batch checker will be used and audit reports published (to enable any trends to be spotted).

Exceptions: A list of permitted exceptions will be provided.

QA

Fo

cus

Met

ho

do

log

y

Note that University of Edinburgh Library have a useful Web accessibility policy

Note that University of Edinburgh Library have a useful Web accessibility policy


Applying QA – Accessibility (2)

Policy: The organisation has no accessibility policy

Authors are free to implement their own accessibility shortcuts (if at all)

Architecture: No centralised policy covering authoring tools or architecture will be provided

Monitoring: No monitoring will be carried out

If you don't have a written policy, the unwritten policy may well be frightening!

If you don't have a written policy, the unwritten policy may well be frightening!


Self Assessment Toolkit

Further Deliverables

We are developing a self-assessment toolkit for projects to use, by individual projects or across project clusters

The toolkit consists of:• Self-assessment questionnaires• Examples of QA procedures• Documented examples of use of testing tools• Advice on standards and best practices• Case studies• FAQs• …

Nex

t S

tep

s


Self Assessment Toolkit (1)N

ext

Ste

ps


Self Assessment Toolkit (2)

The toolkit is also available in paper format, suitable for use in workshops, meetings, etc.

This approach will be used in today's QA workshop

Toolkits on Metadata, Software and Service Deployment will be used


QA For Metadata – An Example

Presentation By Marieke Guy


Group Exercise 2

Embedding Quality Assurance Into Your Project WorkIn your groups use the QA framework to define:

• Appropriate policies• Appropriate checking procedures• Audit trails

In the areas of metadata, software and service deployment

You may use the metadata, software and service deployment toolkits.


Report Back – Exercise 2

Please provide a brief report back on:• Any significant issues which arose• How you envisage addressing the issues covered

in the exercise?• What additional support you may need


Comments On Exercises

Your project deliverables may not be interoperable because:

• You were unclear of the purpose of the metadata / software or there were differing views across project partners

• You chose inappropriate standards for your purpose or your level or expertise, funding, timescales, etc.

• Your modelling of your metadata (or database) meant that it was not interoperable with others

• You chose an inappropriate implementation architecture• You failed to implement appropriate cataloguing, etc. rules• You had cataloguing rules, but no processes for spotting

problems, deviations, ambiguities, etc.• You haven't allocated sufficient resources to implement the

above

You will need to implement QA procedures to address such issues

You will need to implement QA procedures to address such issues


Hype

QA implemented

Thoughts On Timeliness of QA

Early adopters

Despondency

Realism & service deployment

MaturitySoftwareDigitisationWebMetadataWeb ServicesSemantic Web

MaturitySoftwareDigitisationWebMetadataWeb ServicesSemantic Web

Note that QA for metadata is at an early stage. We are currently still developing and evaluating the potential for technologies such as OAI, IMS, …

Note that QA for metadata is at an early stage. We are currently still developing and evaluating the potential for technologies such as OAI, IMS, …


Next Steps – Metadata

QA for metadata is at early stages (little documented work found).Thoughts on QA for metadata:

Syntax Check syntax on embedded DCCataloguing Rules Ensure projects have appropriate

cataloguing rulesInteroperability Ensure metadata can interoperate

with third partiesFitness for purpose Ensure metadata is fit for its purpose Change control Ensure architecture for managing

metadata can cope with change control (cf. Exploit Case Study)

Nex

t S

tep

s

See <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#metadata> and <http://www.ukoln.ac.uk/qa-focus/documents/case-studies/#metadata>

See <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#metadata> and <http://www.ukoln.ac.uk/qa-focus/documents/case-studies/#metadata>


Next Steps – Software

QA for software development is mature, but there may be a need to engage in culture change within the sector

Thoughts on QA for software:Specification Process for developing systems

spec, agreeing spec with stakeholders, etc.

Methodology Documentation of systems development methodology (cf UML case study)

Testing Documentation of testing methodology

Deployment Awareness of deployment challenges

Nex

t S

tep

s

See for example <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#software>

See for example <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#software>


Next Steps – Service Deployment

QA for service deployment is important to protect investment in project funding (but there is a lack of awareness in some cases).

Development Architecture Early public documentation of development architecture

Service Architecture Awareness of likely service architecture and liaison with service

Security Awareness of security concerns Licensing Licensing issues

ScenariosSoftware developed using cool open software environment – but JISC Service has no knowledge of environment.Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns.

ScenariosSoftware developed using cool open software environment – but JISC Service has no knowledge of environment.Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns.

Nex

t S

tep

s


Approaches For Web

Examples of areas which may be considered for QA for Web is illustrated.

We intend to provide such advice on solutions for metadata, software an service deployment once we know the answers and have obtained case studies.

Web ToolkitThe Web toolkit seeks to address quality assurance issues for the provision of a Web site. 1. Purpose Of Your Web SiteHave you identified the purpose of your Web site? There can be a number of roles for your project Web site: information about the project; for use by project partners; access to project deliverables, etc. You may wish to take different approaches depending on the different roles (e.g. use of standards; access conditions; preservation policy, etc.) For further information see:

briefing-15, 2. Standards For Your Web SiteHave you chosen the standards to be used on your Web site?? Use of open standards such as HTML and CSS are important in order to maximise access to resources, ensure resources function correctly and to facilitate reuse of your Web site. For further information see:

briefing-01, briefing-35, …case-study-02, case-study-05, …

Web ToolkitThe Web toolkit seeks to address quality assurance issues for the provision of a Web site. 1. Purpose Of Your Web SiteHave you identified the purpose of your Web site? There can be a number of roles for your project Web site: information about the project; for use by project partners; access to project deliverables, etc. You may wish to take different approaches depending on the different roles (e.g. use of standards; access conditions; preservation policy, etc.) For further information see:

briefing-15, 2. Standards For Your Web SiteHave you chosen the standards to be used on your Web site?? Use of open standards such as HTML and CSS are important in order to maximise access to resources, ensure resources function correctly and to facilitate reuse of your Web site. For further information see:

briefing-01, briefing-35, …case-study-02, case-study-05, …


What Next? (1)

QA Focus materials:• Available for you to use• Feedback needed• Book token will be provided for 2 people providing

best feedback (see note)

QA for metadata, software and service deployment• Currently under development• Your input today is valuable• Why not contribute a case study?


What Next? (2)

Embedding QA In Your Work• There is a need for QA to ensure interoperability

and protect JISC's investment• QA Focus have recommended a self-assessment

approach, rather than external checkers• This is reliant on projects implementing

appropriate QA procedures• You will need to consider how to embed QA into

your project work

Please complete the form in which you'll list your action plansThis is for your personal use.

Please complete the form in which you'll list your action plansThis is for your personal use.


What Next? (3)

Future ProgrammesQA Focus is recommending that for future JISC programmes:

• Programme calls require projects to indicate the approaches they will take to QA

• Initial reports to JISC should describe in more detail the projects: QA methodology Criteria for

standards Intended service deployment environment

Your current project can provide a valuable opportunity to gain experiences in these areas

Your current project can provide a valuable opportunity to gain experiences in these areas


Conclusions

We have described:• The approaches to QA which have been taken by

the QA Focus project• The resources which have been developed for the

community by QA Focus• A systematic methodology for QA

You have:• Addressed the QA requirements for your project• Started to employ the QA methodology• Identified areas for you to address• Identified areas in which you can contribute to the

community

Time for a drink!Time for a drink!

Documents

1 QA Focus – Supporting JISC's Digital Library Programmes Quality Assurance For Digital Library Programmes: A Practical Workshop Brian Kelly UK Web Focus