Approved Assessor Handbook

How to use this guide

Contents

1. Document Management a. Glossary of Terms

2. What is an Approved Assessor? 3. How to become an Approved Assessor

a. Application Process 4. Approved Assessor Life-Cycle

a. Stage 1 - On-boarding b. Stage 2 – Preparation Phase c. Stage 3 – Live d. Stage 4 - Off-boarding

5. Expectations of an Approved Assessor 6. Governance 7. DAQ Standards

This Handbook is a guide for any organisation interested in applying to become an Approved Assessors for the NHS Apps and Wearables Service.

This handbook is also available via the Apps and Wearables Guidance Pages.

This guide covers:

What is an Approved Assessor?

How to become an Approved Assessor

The role of an Approved Assessor.

Expectations of an Approved Assessor.

The governance underpinning the Operating Model.

Document Management

Programme Reviewers

Role Name

Programme Director Hazel Jones

Operations Programme Manager Marcus Hinde

Product Programme Manager Laura Jones

Glossary of Terms

Term / Abbreviation What it stands for

AAOM Approved Assessor Operating Model

Assessor Candidate An organisation that has yet to pass the Application process to become an Approved Assessor

Assessor Organisation An organisation that has passed their application and is within their preparation phase, awaiting approved assessor status

Approved Assessor Organisation

An organisation approved by the Service to assess apps against the DAQ standard and has passed their Preparation Phase, and so achieved ‘Approved’ status.

Assessor Account Manager

The point of contact for an (Approved) Assessor Organisation, responsible for managing the organisation’s portfolio of work within the DAP.

Assessor Legal Agreement

The legal document that describes the agreement between NHS Digital and an Approved Assessor

Assessor Marketplace The area where Assessors can advertise their services to potential Developers. Developer’s select their Approved Assessors from this marketplace.

Auditor NHS Subject Matter Experts (SMEs) who audit the assessments carried out by the Assessors

Commissioner Groups that can commission apps for publishing on the NHS Apps Library. Commissioners can include: CCGs, CSU and Trusts.

Developer An organisation or individual who has an app for publication on the NHS Apps Library

Developer Legal Agreement

The legal document that describes the agreement between NHS Digital and App Developers.

Digital Assessment Questions (DAQ)

A set of questions Developers need to answer and pass, forming a standard that their products must adhere to in order to have their digital tool published on the NHS Apps Library

DAQ Acceptance Criteria

The criteria used by Assessors to guide them in their assessments.

Digital Assessment Portal

The portal on which: App Developers provide answers to the DAQ; App Assessors assess apps against the DAQ standard; and Auditors audit the Assessors.

Governance Group A group of NHS subject-matter experts from across the areas of the DAQ.

NHS Digital

NHS Digital are the authority responsible for: generating the Health Apps Assessor Agreement and Developer Licences; ensuring published DAQ standards are up to date and adhered to; providing approval of Assessor Candidates; and the audit of Approved Assessors.

NHS England NHS England are the authority with responsibility for clinical governance and oversight of the NHS Apps Library assessment process

Preparation Phase The period between an organisation conducting assessments as an 'Assessor' and demonstrating the experience required to received full 'Approved Assessor' status

Reviewer An Assessor Organisation’s subject matter expert responsible for reviewing and approving individual question answers.

What is an Approved Assessor?

The Apps and Wearables service is a part of the NHS Empower the Person portfolio and aims to enable people to find trusted digital health tools to help with their healthcare needs and goals. This will be achieved through an easily accessible tool, the NHS Apps Library,

Central to this strategy is ensuring any Products advertised on the NHS Apps Library have met a required standard. That standard is outlined in the published Digital Assessment Questions (DAQ).

An Approved Assessor is an organisation approved to review a Developer’s answers to the DAQ on behalf of NHS Digital. Their responsibility is to assess apps and digital health tools against the published DAQ standards on behalf of NHS Digital

How to become an Approved Assessor

The Digital Assessment Questions are designed to assess that digital health tools are trusted and safe to use.

The DAQ covers the following areas:

• Eligibility

• Company and Product registration

• Evidence of Outcomes

• Clinical Safety

• Data Protection

• Security

• Usability and Accessibility

• Interoperability

• Technical Stability

Each area demands subject matter expertise from a Reviewer, without which they will not be able to knowledgeably assess whether an answer demonstrates the standard required by the DAQ.

To make sure an Approved Assessor Organisation has the knowledge to assess Developer submissions to the DAQ, they must pass an application process and go through a Preparation Phase before they receive ‘Approved Assessor’ status. Until they receive that status they are referred to as;

▪ An ‘Assessor Candidate’ - Organisation's status prior to successful application ▪ An 'Assessor Organisation' - an Organisation that has passed the application process

and is within their Preparation phase. ▪ An 'Approved Assessor Organisation' - An organisation that has completed their

Preparation phase and been given full 'Approved' status

All definitions are outlined in the Glossary of Terms.

The application form’s purpose is to understand whether the applying organisation has the operational and commercial capability to fulfil the Approved Assessor role. This includes demonstrating the organisation’s ability to review Developer submissions across the subject areas of the DAQ.

Application Process

• Access - From the Guidance Pages you can email to register you interest in the role. Your email will be followed up with an application form to complete and return

o Guidance Pages - https://developer.nhs.uk/digital-tools/become-an-assessor/

o Email - m.health@nhs.net • Submission - Applications should be submitted to the same email -

[m.health@nhs.net]. NHS Digital will acknowledge all applications by email. • Application review – Applications will be reviewed by NHS Digital and a team of

subject matter experts. • Notify – Assessor Candidates will be notified of NHS Digital’s decision and reasons

for it by email. Unsuccessful Candidates will receive feedback within two weeks of the decision.

The application submission and review cycle will take place in waves. This is to help the

programme manage demand.

The first wave will operate as a Pilot to coincide with the Digital Assessment Portal’s

transition to a Public Beta phase.

Figure 1 – Becoming an Assessor

Approved Assessor Life-Cycle

The role of an Approved Assessor is to review a Developer’s answers to how their product meets the standards of the DAQ.

Based on these answers they must then recommend whether the Developer’s product meets the standard required to be published on the NHS Apps Library.

NHS Digital has created a Digital Assessment Portal (DAP) to facilitate this process and the ongoing assessment dialogue. Both Developers and Approved Assessors will have access to this portal.

Developers will use it to submit their answers for review by Approved Assessors. This portal will be the single point of access for publication to the NHS Apps Library.

Following a successful application Assessors Candidates are provided access to the DAP. This forms the first of a four stage Approved Assessor Lifecycle.

1. On-boarding 2. Preparation Phase 3. Live 4. Off-boarding

Stage 1 - On-boarding

Once an Assessor Candidate successfully passes the application process they are referred to as an

Assessor Organisation.

The Assessor Organisation will then be given access to the Digital Assessment Portal via a named

person, who will be provided with Assessor Account Manager rights. This role will then be able to

create Reviewer roles so that individuals in their organisation can carry out assessments in their

specific subject areas on behalf of the Assessor Organisation.

Your organisation will need to sign the Assessor Legal Agreement and upload a copy to the Digital

Assessment Portal before being on-boarded. This outlines your legal responsibilities and

expectations.

On-boarding process:

• Application – Applications are submitted and reviewed. Successful applications will be on-boarded.

• Obligations – Assessor Candidates are required to sign a legal agreement defining their relationship with NHS Digital and their responsibilities regarding their role as an Approved Assessor

• Digital Assessment Portal – Your organisation’s Account Manager will be provided access to the Digital Assessment Portal (DAP) and asked to set up your organisation’s account.

KE

Y T

AK

EA

WA

YS

Step 4 – Approved

Assessor Status If NHS-D is confident in the organisation’s ability to assess apps they will grant full ‘Approved Assessor’ status. The organisation will be able to use NHS Approved Assessor Logo and market their services.

Step 2 –

Onboarding Following a successful application, Assessor Candidates become Assessors and are given access to the Digital Assessment Portal

Step 1 – ApplicationOrganisation applies to become an Assessor. Applications are reviewed by NHS-D.

Step 3 – Preparation Phase Assessors undergo an agreed

preparation phase to demonstrate their ability to successfully assess apps

An Approved Assessor is

an organisation approved to review apps and digital

tools on behalf of the NHS

to the latest DAQ standard.

To become an Approved

Assessor, your organisation will need to undergo due

diligence checks and have

access to individuals with

minimum qualifications or

experience in:

• Clinical Safety

• Data Protection

• Security

• Accessibility and

Usability

• Interoperability

• Technical Stability

Appro

ved A

ssessors

Step 5 – Live Approved Assessors work with App Developers to help them achieve the DAQ standard and feature on the Apps Library

Step 6 – LifecycleAssessors work with App Developers to ensure they maintain the DAQ standard as their product or the DAQ evolves

Annual Audit

Early Audit

Sign Deed

DAQ evolution Off-Board

Becoming

an Approved

Assessor

Figure 2 - On-boarding Process

Stage 2 – Preparation Phase

Once an Assessor Organisation has been on-boarded and begins their role, they will undergo a Preparation Phase. The Preparation Phase allows the Assessor autonomy to carry out their role as per ’business as usual’ whilst having additional support from NHS Digital as they develop and demonstrate their skill set to gain 'Approved' status.

During the Preparation Phase, NHS Digital will carry out an ‘audit’ of the Assessor’s assessments. Processes and decision need to be checked for adherence to their legal obligations and the DAQ standards to which they have been reviewing.

After passing the audit the NHS Digital will approve the change of the organisation’s status from Assessor to Approved Assessor. This will give them permission to use the Approved Assessor logo.

If after this initial period NHS Digital is not fully confident in the Assessor Organisation's ways of working and assessment capabilities, they will look to extend the Preparation Phase timeframe. Assessors will be provided information on areas to work on and how they can improve and develop into Approved Assessors. Following the extension another audit review will take place.

Preparation Phase:

• Start - Assessor begins services. • Document - Assessor are encouraged to document their ways of working (processes,

communications, etc.) and anything they think pertinent to their Preparation Phase review.

• Preparation Phase Audit - NHS Digital will audit the organisations ways of working and assessment results within the DAP

• Decision - NHS Digital decides whether to award 'Approved Assessor' status or to extend the Preparation Phase.

After passing the Preparation Phase -

• Branding – Approved Assessor Organisations are given permission to refer to themselves as ‘Approved Assessors’ in their operations, including using the official role logo. Branding guidelines will be provided.

• Marketplace – Your organisation will be added to the list of available Approved Assessors – This is how Developers choose an Assessor. It also allows Assessors to advertise other pertinent services.

Figure 3 - Approved Assessor Preparation Phase

Stage 3 – Live

Approved Assessors will be able to carry out assessments for Developers for as long as they maintain 'Approved Assessor' Status.

A process of annual auditing will continue throughout an Approved Assessor’s life time with the NHS Apps and Wearables Service. This is to ensure performance remains in line with DAQ’s standards and aligns to NHS Digital's expectations and values. Audits will be carried out by subject matter experts from NHS Digital.

It is the responsibility of Approved Assessors to retain access to people suitably experienced and qualified in line with the latest published DAQs. This will ensure quality is maintained and that Approved Assessors can effectively assist Developers through the assessment process.

To help with assessments, Approved Assessor Organisations can add 'Reviewers'. to their account and assign them individual areas of responsibility within an assessment. For

example, you could assign your internal security expert to review a submitted Security section.

Assessments are necessary whenever a new product is submitted by a Developer, but also when the DAQ or the Product changes in a way that makes a substantive difference to previously provided answers. If that happens, Approved Assessors must help Developers through re-assessment.

Re-assessments are also required on an annual basis.

Live Process:

• Selected - Developers choose an Approved Assessor from the Marketplace list. This will match you to their submission, enabling you to view their answers and begin assessments.

• Assess - You will guide Developers through the assessment process, helping them to demonstrate their product meets the DAQ standards

• Approve - You will approve DAQ submissions that have met the standard for Apps Library publication

• Re-assess - Developers will require guidance through the re-assessment process as the DAQ or their product iterates. Or when their annual anniversary occurs.

Figure 4 - Approved Assessor BAU

Stage 4 - Off-boarding

As an Approved Assessor you are not tied in for life, and if you decide reviewing is no longer for you, NHS Digital will work with you to efficiently off-board you from the NHS Apps and Wearables service. You must email NHS Digital your intention to leave.

The most important part of off-boarding is maintaining operational continuity so that no Developer’s experience is adversely affected. Therefore, the first part of off-boarding is making sure any in-flight assessments are not affected. With this in mind, before you can off-board, you must finish the assessments in your pipeline of work.

Another important part of the off-boarding is ensuring the public is not confused by the change in status of an Organisation. Therefore it is important any branding references to being a NHS Digital 'Approved Assessor', including using the logo, are removed from company branding and marketing materials immediately.

Organisations can also have their Approved Assessor status removed for compliance reasons. In such cases NHS Digital will notify the Assessor Account Manager by email.

Off-boarding:

• Notify – You must email NHS Digital of your desire to leave the role • In-flight Assessments - Off-boarding will pause either, until all planned assessments

are complete, or responsibility of an assessment has been transferred to another Approved Assessor of the Developer's choice.

• Marketplace - Organisation will be removed from the Approved Assessor choice list within the Marketplace for selection by Developers

• Branding - You must immediately cease using any reference to being an 'Approved Assessor'.

• DAP Access - Access to your Digital Assessment Portal account will be removed.

Figure 5 - Off-boarding process

Evidence for Effectiveness

NICE recently published their standards regarding Evidence for Effectiveness for digital health tools.

This Approved Assessors Operating Model does not currently allude to reviews on the effectiveness

of apps and discusses assessment via technical questions only.

However, following future iterations of the Digital Assessment Questions to incorporate the

principles of categorisation, assessment of clinical risk, and reviews of evidence for effectiveness, the

Approved Assessors Operating Model will evolve to reflect this.

Whilst this operating model continues to be developed, some Approved Assessors may be able to

review effectiveness data from lower risk apps, subject to decisions by NHS England and discussions

between the Approved Assessor and NHS Digital.

The assessment of apps with a higher risk or more complex functionality will most likely be retained

entirely for review internally until additional assessment routes are confirmed. Therefore, whether

apps can be agreed for review between Approved Assessors and Developers will be confirmed in

the initial stages of the assessment process, before official selection within the Digital Assessment

Portal.

The discussed standards are available online here –

https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-

for-digital-health-technologies

Expectations of an Approved Assessor

The core responsibility of an Approved Assessor is to work with Developers to help them to demonstrate their products meet the required DAQ Standard for publication on the NHS Apps Library, this is achieved through successfully completing the DAQ.

Approved Assessors should guide Developers through the assessment process. They must be knowledgeable enough across the DAQ subject areas to advise a Developer on what their answer must demonstrate to meet the DAQ Standard. This must include the ability to advise on how a product must iterate to meet DAQ Standard.

The Digital Assessment Questions incorporates industry best practice, regulatory standards and directives required of healthcare and wellbeing products and technologies. To ensure it remains a representation of these, it will continuously iterate. A roadmap of planned iteration is published on the Guidance Pages

A responsibility of an Approved Assessor is to be aware of planned DAQ changes (via the NHS Digital DAQ Roadmap) and ensure their knowledge keeps up with those changing standards. This will ensure they can assess DAQ submissions adequately and provide advice when necessary.

DAQ Roadmap is published here - https://developer.nhs.uk/digital-tools/daq/

Approved Assessors must sign an Assessor Legal Agreement with NHS Digital before acting as an Approved Assessor. These obligations need to be kept at all times.

Expectations:

• Assessment - Assess Developer's DAQ submissions and work with them to help them achieve the required standard for publication.

• DAQ Standard - Understand and adhere to the standards of the DAQs. • Legal Obligations - Meet all required legal obligations as per the signed

Assessor Legal Agreement.

Governance

The Live Services team will manage the procedures and processes to run the NHS Apps and

Wearables Service. They are governed by a Live Services Board.

Clinical governance and oversight of the assessment process for the NHS Apps Library will be

provided by NHS England.

The below explains the governance controls in place throughout the Assessor Lifecycle to ensure

standards.

Application

▪ In addition to any legal or commercial review necessary, a team of subject-matter experts internal to NHS Digital, review any applications to become an Approved Assessor.

▪ They look for evidence that the organisation has demonstrated the ability to complete assessments on behalf of NHS Digital.

▪ This group also take responsibility for ensuring the application form reflects the latest experience requirements needed for the Approved Assessor role. They will take advice from NHS England as necessary.

Preparation Phase and Auditing

The Preparation Phase is assessed following the standard Audit process, therefore the governance of both will be discussed together here.

• An Auditor will use the Digital Assessment Portal (DAP) to conduct all audits.

• Using the DAP, they review both the Developer’s submitted answers to the DAQ and the Approved Assessor Organisation’s assessment results and decision making.

• Auditors are checking for consistency across assessments and alignment to the DAQ’s standard.

• Auditors will provide commentary on the assessments. This commentary will be made available to the Approved Assessor Organisation.

• Audits will be conducted by a team of NHS Digital subject-matter experts. They may seek guidance from anyone they require it from, such as from commercial or legal teams.

NHS Digital expect to audit Assessors once every 12 months, unless there's reason to suspect a breach of contract or legal agreement.

Performance

▪ KPI measurements such as response time will also be considered during audits. These indicators will be gathered through Approved Assessor interactions with the Developer via Digital Assessment Portal.

▪ Measurements are used to assess and improve overall performance of the Service, rather than purely for the review of individual organisational.

DAQ Standards

The Digital Assessment Questions form the standard required from an app or digital health tool to be published on the NHS Apps Library.

The standard is a combination of industry best practice, regulation and legislation. The technical assessment stage is separated out into seven areas, covering a broad range of criteria. The breadth is designed to make sure products meeting the standard have been thoroughly reviewed.

It is vital Approved Assessors and their Reviewers are familiar with the DAQ's sections and knowledgeable in the areas.

To continue to reflect best practice and the changing requirements of digital health tools, the DAQ will continuously evolve and iterate.

These iterations are outlined in the DAQ Roadmap, available here: https://developer.nhs.uk/digital-tools/daq/