EHR-S Conformance ConsiderationsEHR-S Conformance Considerations

Lynne S. Rosenthal

National Institute of Standards and Technologylynne.rosenthal@nist.gov

August 2004

2

Today’s ObjectivesToday’s Objectives

Present basic information about conformance and conformance related topics

Address:– What is Conformance?

– What does it mean to conform?

– How do you know a system conforms?

– What do you need to do testing?

– Who does the testing?

3

BackgroundBackground

Standards not enough to ensure interoperability– Standards only meaningful if implemented in a consistent way

Need to ensure that implementations adhere to the standard– What is expected of implementations in order to claim conformance –

i.e., what are the requirements?

– How will we know if an implementation conforms?

• test suites, test tools

Different ideas of what conformance is– Past experience may have affected view of conformance

4

TerminologyTerminology

Conformance

Specification Conformance Clause

Conformance Testing test suite, test tools

Validation process, reporting

Certification certificate, brand

5

Terminology - ConformanceTerminology - Conformance

CONFORMANCE – the fulfillment of a product, process or service of specified requirements. (ISO Guide 2)

– The requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification

CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

6

Terminology - Conformance TestingTerminology - Conformance Testing

CONFORMANCE TESTING – a way to determine directly or indirectly that relevant requirements are

fulfilled. – Serves as a communication between buyer and sellers

• Buyers increased confidence• Sellers substantiate claims

– Necessary, but not sufficient, for interoperability

Note: referred to as conformity assessment in ISO Guide 2

7

Terminology - Testing ProcessTerminology - Testing Process

VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite– ensures that testing can be repeatable and reproducible

– ensures that conclusions are consistent with facts presented in the evaluation

CERTIFICATION - acknowledgement that a validation was completed and the criteria established for issuing certificates (brands) was met.

8

Testing IngredientsTesting Ingredients

Standard Conformance clause

Conformance TestingTest assertions

Test suite (test software, test scripts, test criteria)

Validation Process - policy and procedures for testing

Certificationqualified bodies to do the testing and certification

control board - advisory and arbiter

9

Conformance IssuesConformance Issues

Conformance Clause should address1. What Needs to Conform

2. How to conform and claim conformance

3. Subdividing a specification: Profiles, Levels, Modules

4. Variability

a. Extensions

b. Options

c. Implementer defined values

d. Alternative approaches

10

What needs to conform? What needs to conform?

Identify who or what will implement the specification What ‘class of products’ will be developed

– What products are considered an EHR-S?

– Does it make sense to mention these?

Why care? – It helps in defining the scope of the specification

– It helps the reader know what is being targeted by the specification

What are the conditions that need to be met in order to claim conformance

11

Profiles, Levels, ModulesProfiles, Levels, Modules Profile is a subset of the overall specification that includes all

the functionality necessary to satisfy the requirements of a particular community of users. – Use to target needs of specific constituencies or purpose (care setting)

Levels are nested, hierarchical subsets of the specification. Level 1= core that all must implement, subsequent levels include lower level + added functionality– Use to facilitate incremental development and implementation (system core)

Modules are discrete collections of semantically-related units of functionality that do not necessarily fit into a simple hierarchical structure. – Use when functionality can be implemented independently of one another

(ePrescribing, CPOE, PHR)

Reference: Subdivide a Specification (extracted from W3C Quality Assurance documents)

12

Profiles, Levels, ModulesProfiles, Levels, Modules

EHR-SDSTU

NursingHomeProfile

Ambul.CareProfile

System Core

Care Setting CoreCare setting core

XX ZZ YY

3

System Core

Care Setting Core

CPOE

ePrescibe

PHR

13

ExtensionsExtensions Extensions incorporate additional features beyond what is defined

in the specification If allowed:

– shall not re-define semantics of existing functions

– shall not break existing functions

Standardized method for extending the specification – Define how to ‘simplify’ or ‘add’ functions for care settings

– Provide wording so vendors can articulate their extensions in a consistent way

Why?– Helps ensure extensions defined in a consistent manner – leading to

predictable understanding and handling aids interoperability

14

Options = discretionary choicesOptions = discretionary choices

Provide the freedom to make choices about:– Whether or not to support a function and/or features of that

function– Which functions, features, or values to choose from a list of

possibilities– Implementation dependent features or values – Alternative approaches

Indicated by choice of words: Should, May, Recommend Make sure need for the option is real If the option is implemented, it must be implemented

correctly (as defined in the specification)

15

More on optionsMore on options

Implementation Defined Features– may be necessary to address environmental conditions

(code sets and vocabulary, locality differences - e.g., language, time zones, dependencies on other technologies, or just to be flexible)

Alternative Approaches– specification may describe several different ways to

accomplish an operation (e.g., choice of bindings)

– specify whether or not all approaches need to be implemented

16

Conformance TestingConformance Testing

Requires: a test tool/method and testing program Test method:

– Way to determine requirements are met– E.g., test tool, test suite, checklist, etc

Testing is Falsification Testing– Can show presence of errors not their absence– Prove non-conformance, can never prove conformance

Testing Program = Doing the testing– 1st party (self-test), 2nd party (buyer), 3rd party (independent)

CONFORMANCE TESTING – a way to determine directly or indirectly that relevant requirements are fulfilled.

17

Testing and Certification Testing and Certification Considerations

Not all specifications need a testing program– Risks of nonconformance vs. cost of creating and running

testing program How formal should testing be?

– Determined by the market or regulation– Self testing, 2nd party (buyer), 3rd party testing– Self-certifying vs. independent certification

Testing program– Who will ‘own’ the testing program– What are the policy and procedures– Who will do the testing – testing organizations?– Who will issue the certificates?– Who will handle disputes Control board– Who will do maintenance?

18

Testing and CertificationTesting and Certification

Enables quality products, competitive markets, more choices Must be acceptable to sellers and credible to buyers

Reference: Conformance Testing and Certification Model for Software Specifications

19

Conformance GuidanceConformance Guidance

All specs to contain a conformance clause– Identifies: what and how to conform– Any special designations of conformance (e.g.,

conform to core levels)

– Ways that conforming implementations can vary– How to make a claim of conformance

Traceability to the specification

Specification profiles EHR-S

Conformance statements test cases tests

20

Conformance Clause ExamplesConformance Clause Examples Example 1

– Conformance of Generators is defined in terms of conformance to a particular profile of CGM.

– A Profile conforms if it meets all the requirements in ISO xxx and is structured in accordance with the Rules for Profiles

Example 2– There is no concept of conformance to this standard because there is no normative content.

This approach is taken for several reasons, chiefly … Example 3

– Within the context of this specification, conformance can be claimed for markup, document types, module implementations, documents, generators, and interpreters. In most of these cases, two levels of conformance are available: simple conformance and full conformance. Simple conformance means that the conforming object supports the minimal content model of the ruby element in section 2.1, i.e. only simple ruby markup. Full conformance means that the conforming object supports the maximal content model of the ruby element in section 2.1, i.e. that both simple and complex ruby markup are supported.

Example 4:– DICOM 1 (PS3.1-2003) points to PS3.2-2003, DICOM Part 2: Conformance – Each part of the 1073 family of standards contains a conformance section.

21

ReferencesReferences

1. Conformance Testing http://www.oasis-open.org/cover/conform20000112.html

Defines conformance and conformance testing and describes the components of a conformance testing program.

2. Conformance Testing and Certification Model for Software Specifications

http://www.itl.nist.gov/div897/ctg/conformance/bulletin-conformance.htm

Overview of testing and certification, including identifying key roles, activities and products involved in any conformance testing and certification program.

3. W3C Quality Assurance Framework: Specification Guidelines

and W3C Quality Assurance Variability in Specificationshttp://www.w3.org/QA/WG/ Public versions available in September 2004.

How to write better specifications with a focus on conformance issues