LRI Validation Suite Meeting

October 4th, 2011

Agenda• Action Item List• Test data update

– Selection of core message set– Review of lab results test category spreadsheet– First set of test messages– NIST data sets and management of test data

• Review policy proposal for validating receiver processing of terminology• Review ELINCS Test Plan and Test Tool (Met on Thursday)• Update on LIS Test Plan Template• Update on EHR Test Plan Template• Spreadsheet Analysis/Juror Document

– Overview and purpose of spreadsheet– Mapping CLIA requirements to HL7 Elements– Identifying Reportable Conditions to PH Lab Results– IG Example Messages (IG Analysis Call)

• Tooling Update• Face-to-Face Meeting Plans• Planning

Draft Test Messages• Completed/Located Draft Messages

– Hematology– Chemistry– Microbiology (has been reviewed by Riki)– Special Chemistry (Lipid Panel)– Susceptibility– Other (Surgical Pathology )

• Work in Progress– Routine Urinalysis– Complete Urinalysis

• Provides at least one message for each of the categories we identified• We will use these to finish test plan skeletons, to develop data

management spreadsheet, create test case variations, etc.• Disclaimer: These messages are not verified to be correct and do not

yet meet the technical/structural requirements specified in the IG.

Draft Test Messages• Hematology

– Hemoglobin 718-7 – use as example for minimally populated message• Started mapping to IG

– Potentially maximally populated message (with no repeats)– Unknown patient– OID version– Non-OID version

• Chemistry• Microbiology (has been reviewed by Riki)• Special Chemistry (Lipid Panel)• Susceptibility• Other (Surgical Pathology)• Work in Progress

– Routine Urinalysis– Complete Urinalysis

ELINCS I• Met with Walter Sujansky and team on 09/27/2011• Generally speaking, in-line with validation suite approach• Automated LIS Validation and Inspection Testing for EHR (EHR Display and

mostly CLIA)– Other data items not specified as CLIA requirements were not examined– I think we can relax our requirement that all required elements need to be inspected;

some will be implied (a lot of MSH, set id, etc.—it is just part of the message processing)

– Other items will need to be inspected (access to DB) – Did not test “RE” usage elements; did test for conditionals when feasible– LIS testing was context free testing

• About 20 test messages covering the various scenarios (e.g., P, F, C).– Messages posted on Wiki– We can adopt these messages with modifications– Limited in scope (in terms of the breath of lab results)

• Agreed that an “Expert Reviewer” is needed to access terminology equivalence– ELINCS was self testing tool (CCHIT did the certification) – We’ll follow same model (we’ll provide recommendations for “Expert Reviewers)

ELINCS II• Since Labs and trading partners will have worked out

mappings we can’t necessarily expect a particular mapping for our test cases (test data sheets)– This is OK; we can validate to make sure it is a valid LOINC and

also provide a set of acceptable LOINCs – The tool can provide an “ERROR with exceptions”; meaning that

the validation flagged it but it may be OK based on the inspection of an “Expert Reviewer”

• Code is .net/C# (Desktop application)– We’ll leverage ideas, not code base

• Configuration for certain data elements– Addressing, Patient IDs, local codes?, etc.– Dynamically create messages from message templates– Inline with the validation suite WG design

Challenge of Testing Terminology I(Receiver System)

• Problem Statement:– A specific terminology is specified for use in the IG (e.g., LOINC; OBX.3.1

(code), OBX.3.2 (text), and OBX.3.3 (LN-coding system)– The LIS Test Harness sends the EHR a test message containing a LOINC

code• How do we test?• How do we test the universe of possible codes?• Considerations:

– Meaningful Use Requirements (See next Slide)– EHR likely translate/map code to internal code and display local text

representation on display– EHR could use standardized codes internally– Inspection Testing

• Automated testing is limited

Meaningful Use Requirements• LOINC is a named standard for lab test results in the §170.306.g Reportable Lab Results Stage

1 ONC certification criterion.• In the §170.302 (h) Incorporate laboratory test results criterion it states:

– Receive results. Electronically receive clinical laboratory test results in a structured format and display such results in human readable format.

– Display test report information. Electronically display all the information for a test report specified at 42 CFR 493.1291(c)(1) through (7).

– Incorporate results. Electronically attribute, associate, or link a laboratory test result to a laboratory order or patient record.

• The criteria use the following verbiage when referring to LOINC: – “§170.306 (g) Reportable lab results. Electronically record, modify, retrieve, and submit reportable

clinical lab results in accordance with the standard (and applicable implementation specifications) specified in §170.205(c) and, at a minimum, the version of the standard specified in §170.207(c).”

– §170.207(c) states: “Standard. Logical Observation Identifiers Names and Codes (LOINC®) version 2.27, when such codes were received within an electronic transaction from a laboratory (incorporated by reference in §170.299).”

– In the ONC Final Rule Preamble for 170.304.i Exchange Clinical Information and Patient Summary Record ONC stated the following:• “For the purposes of electronically exchanging a patient summary record, we expect the patient summary record

to include health information that is coded, where applicable, in accordance with adopted vocabulary standards. Therefore, unless otherwise required in the context of a meaningful use objective and measure, an eligible professional (or eligible hospital) would be permitted to map or crosswalk local/proprietary codes to the adopted vocabulary standards prior to transmitting a patient summary record.”

– MU Stage-2 requirement not finalized

Processing of Terminology

• Receiver requirements for processing terminology:– The receiver shall persist (store) the original

standardized code and the original standardized code text as received in exact representation.

– The receiver may perform a translation/mapping to locally defined representations.

Assessment of Terminology• Proposed procedure for assessing the receiver for incorporation of terminology:

– Where applicable to meet CLIA requirements the receiver shall display on the EHR GUI the equivalent representation of the received coded lab results. The receiver shall display at least one of the following:• Original standardized code text• Original standardized code (will the actual code ever be displayed; is it adequate or good

practice to display just the code?)• Local code text

– The receiver shall be capable of demonstrating the persistent of the original standardized code and the original standardized code text. Acceptable methods for attestation:• Administrative access to database• Inspector approved method

– If applicable the receiver shall be capable of demonstrating the linkage of the original standardized code the locally translated/mapped code. Acceptable methods for attestation:• Administrative access to database• Browse capabilities of configuration files• Inspector approved method

Equivalent Representation• The exact original code• The exact original code text• For translated/mapped local code text an equivalent representation as

determined by clinical terminology expert. The following rules and guidance are given to promote consistency in assessment:– Rule: A terminology shall never be made more specific in the

translation/mapping.– Rule: A terminology shall never be made more specific in the display of

standardized terminology.– Guidance: A limited number of synonyms are provided to assist the clinical

terminology expert. Note: a predefined definitive set is not possible—there are too many possible equivalent local representations.

– Guidance: The inspector must consider the context in which the code is used as this may impact the translation/mapping.

Questions• What should be the assessment if a more detailed

terminology term is received that is mapped to a less specific term?

• This mapping will lead to a loss of information if the original code is not persisted.

• Is it valid to display a representation that has less specificity? • Is it valid to display a representation that has less specificity as

long as the original data is persisted in the system? • Is it acceptable for a loss of information to occur when data is

rendered as a report or forwarded on to another system (e.g., public health)? That is, we sent a specific code and then a general code is forwarded on to public health. Is it acceptable for either the general or specific LOINC code to be forwarded?

LOINC-RELMA List for HemoglobinLOINC # Component Property Time Aspect System Scale Ex. UCUM Units Ex. Units Class Long Common Name

Order/ObsType11559-2 Oxyhemoglobin/Hemoglobin.total MFr Pt Bld Qn % % CHEM Fractional oxyhemoglobin in Blood

Observation 119949-7 Oxyhemoglobin/Hemoglobin.total MFr 8H^max BldA Qn % % PULM Fractional oxyhemoglobin in 8

hour maximum Arterial blood 219951-3 Oxyhemoglobin/Hemoglobin.total MFr 8H^min BldA Qn % % PULM Fractional oxyhemoglobin in 8

hour minimum Arterial blood 219953-9 Oxyhemoglobin/Hemoglobin.total MFr Pt Bld.preductal Qn % % PULM Fractional

oxyhemoglobin in Blood Preductal 219955-4 Oxyhemoglobin/Hemoglobin.total MFr Pt Bld.postductal Qn % % PULM Fractional

oxyhemoglobin in Blood Postductal 22714-4 Oxyhemoglobin/Hemoglobin.total MFr Pt BldA Qn % % CHEM Fractional oxyhemoglobin in Arterial

blood Observation 12715-1 Oxyhemoglobin/Hemoglobin.total MFr Pt BldC Qn % % CHEM Fractional oxyhemoglobin in

Capillary blood Observation 12716-9 Oxyhemoglobin/Hemoglobin.total MFr Pt BldV Qn % % CHEM Fractional oxyhemoglobin in Venous

blood Observation 12717-7 Oxyhemoglobin/Hemoglobin.total MFr Pt Plas Qn % % CHEM Fractional oxyhemoglobin in Plasma

Observation 130369-3 Oxyhemoglobin/Hemoglobin.total MFr Pt BldCoV Qn % % of tot CHEM Fractional

oxyhemoglobin in Venous cord blood Observation 130370-1 Oxyhemoglobin/Hemoglobin.total MFr Pt BldCoA Qn % % of tot CHEM Fractional

oxyhemoglobin in Arterial cord blood Observation 134969-6 Oxyhemoglobin/Hemoglobin.total MFr Pt BldMV Qn % % of tot CHEM Fractional

oxyhemoglobin in Mixed venous bloodObservation 1 No Method listed in RELMA for any of these tests

Testing Options/Policy-OLDLOINC # Component Property Time Aspect System Scale Ex. UCUM Units Ex. Units Class Long Common Name Order/ObsType11559-2Oxyhemoglobin/Hemoglobin.total MFr Pt Bld Qn % % CHEM Fractional oxyhemoglobin in Blood Observation 119949-7Oxyhemoglobin/Hemoglobin.total MFr 8H^max BldA Qn % % PULM Fractional oxyhemoglobin in 8 hour maximum Arterial blood 219951-3Oxyhemoglobin/Hemoglobin.total MFr 8H^min BldA Qn % % PULM Fractional oxyhemoglobin in 8 hour minimum Arterial blood 2

OBR|1|111325^EHR^2.16.840.1.113883.19.3.2.3^ISO|1132896^Lab^2.16.840.1.113883.19.3.1.6^ISO|718-7^Hemoglobin^LN|||20070701152505|||||||||100^Hippocrates^Harold||||||20070701162505||CH|F|NA&Not Applicable&LB

OBX|1|NM|11559-2^Hemoglobin.Total^LN||^12.4|g/dL^grams per deciliter^UCUM|12.0 to 16.0||||F|||20070701152505|||||||||Effective Labs, Inc^^^^^DRSD&2.16.840.1.113883.19.1.11&ISO^XX^^^6543|3434 Test Loop^^Ann Arbor^MI^48103^^B

• Depending on what the order is, what are possible test results? Are there more than one valid test result base on local conventions, etc.?

• Potential Test Case Policy: In the data sheet (test case) we can allow any of these to be selected. This assumes that nothing else in the message needs to be changed. It would be an easy way to provide the capability to tests all of these LOINC codes with minimal effort. The Long Common Name is the differentiator.

• One of the determining factors likely will be whether or not the same result value can be used for all of the various versions of Oxyhemoglobin/Hemoglobin.total.

Issues/Comments:• Other data elements in the HL7 message might also need to be test-specific. For example, if data for the SPM-4: Specimen Type

and SPM-8: Specimen Source are included in the message, then the same message probably could not be used for all of the versions of Oxyhemoglobin/Hemoglobin.total. OBX-7: Reference Range also might be a limiting factor if venous blood and arterial blood result values have normal ranges that don’t overlap.

• The LOINC code is the anchor that discretely and accurately identifies the lab test that was ordered/performed. No matter what a hospital or physician chooses to call the test, the LOINC code is the one constant that tells everyone which it test was. For our purposes, we’ll need to be sure that all of the data for the data elements in the HL7 message are appropriate for the test indicated by the LOINC code.

Note on the use of LOINC• The list of tests has been organized by order panel or, for

micro related tests, by target organism for easier readability.

• The LOINC terms included here are considered examples only – though they may be the more common LOINC terms, each laboratory needs to be sure to map their own tests to the most appropriate LOINC, even if it is NOT on this list.

• This is NOT an exclusive list of LOINC terms – ANY valid LOINC should be accepted in data exchange projects based on this specification.

Other Issues I• EHR – Implementation Choices for Terminology

– Use standardize coding internally– Map to local codes– Need to account for options in test procedure

• How to we test for complete coverage of recommended terminology?– Create all messages for all possible (recommended) terms (for important code system,

e.g., LOINC)—This is the preferred approach– Alternative approach: Create a subset and inspect/verify tables for coverage and accuracy– The LAB is not restricted to sending the recommended list of LOINC codes and the EHR

should not fail when it does not recognize a LOINC code. • How can this be tested; should it be tested?

– We should test that the EHR recognizes an invalid LONIC code and rejects the message.• Given that LONIC changes often what mechanisms will a system use to identify invalid codes (based

on a certain format?)• We could send messages where code and text don’t match as another negative test

– Selection of approach may be made on a case-by-case analysis – It may be necessary to provide coverage for all recommended LOINC codes. However, for

other terminology it may not be a priority (e.g., state) or may not be feasible (SNOWMED).

Other Issues II• Use of UCUM. Can/should UCUM be mapped

locally and displayed as local representation?• What terminology can be mapped?• Is it in scope to test the actual results of given

in the lab results (abnormal values was mentioned)

CLIA Requirements42 CFR 493.1291(c) The test report must indicate the following:

1. For positive patient identification, either the patient's name and identification number, or a unique patient identifier and identification number

2. The name and address of the laboratory location where the test was performed

3. The test report date4. The test performed5. Specimen source, when appropriate6. The test result and, if applicable, the units of measurement or interpretation,

or both7. Any information regarding the condition and disposition of specimens that do

not meet the laboratory's criteria for acceptability

CLIA Requirements Mapped to Data Elements42 CFR 493.1291(c) The test report must indicate the following:

1. For positive patient identification, either the patient's name and identification number, or a unique patient identifier and identification number– PID-3 : Unique patient identification number – PID-5 : Patient Name

2. The name and address of the laboratory location where the test was performed- OBX-23/24/25: Lab Identification Fields

3. The test report date- OBX-19: Date/Time Analysis

4. The test performed- OBX-3: LOINC codes for Observation Identifier

5. Specimen source, when appropriate– SPM-4: Specimen Type6. The test result and, if applicable, the units of measurement or interpretation, or both

– OBX-5: Observation Value– OBX-6: Units– OBX-7: Reference Range– OBX-8: Abnormal Flag– OBX-11: Observation Result Status

7. Any information regarding the condition and disposition of specimens that do not meet the laboratory's criteria for acceptability– SPM-21: Specimen Reject Reason– SPM-22: Specimen Quality

Required CLIA Report Elements

– PID-3 : Unique patient identification number – PID-5 : Patient Name– OBX-3: LOINC codes for Observation Identifier– OBX-5: Observation Value– OBX-6: Units– OBX-7: Reference Range– OBX-8: Abnormal Flag– OBX-11: Observation Result Status– OBX-19: Date/Time Analysis– OBX-23/24/25: Lab Identification Fields– SPM-4: Specimen Type– SPM-21: Specimen Reject Reason– SPM-22: Specimen Quality

Are all elements required to be displayed on screen (including the components and subcomponents of these fields)?

Action Item List I• Select message to handle core lab results

– Identify 20 or so common lab results (In progress)– Obtain/Adapt/Create test messages to cover the core set of lab results

(In progress)• Identify/List all pertinent data elements (In progress)

– Create spreadsheet of all data elements with usage of R, RE, and C (rows)– Columns will identify:

• Juror Document (How to assess the element)• Identify the elements required for CLIA testing• Identify static, configurable, or indifference data elements

• Identify/create value sets (In progress)– Incorporate the value sets in PHINVADS– Develop download mechanisms and transformation of values to support

the NIST tooling format

Action Item List II• Review LRI implementation Guide and create a list of all

conformance requirements (Not Started)– Create matrix based on data elements– Link all conformance requirements to data elements when possible– Create “higher” level list of conformance requirements

• Determine the policy for assessing receiver side terminology (Done: need to write policy statement)– Inspection test requirements and procedure– Automated test requirements and procedure

• Complete development of LIS Test Plan Skeleton• Complete development of EHR Test Plan Skeleton

Action Item List III• Identify and document the test dimensions (Not Started)

– Coverage of Lab Results– Scenarios (e.g., Preliminary, Final, Corrected)– Reporting formats– Negative testing– Minimally and maximally populated

• Contact CLIA and CAP inspectors to get their lab inspection process (Need contacts)

• Determine a process for verifying test cases (Not Started)• Implement process for verifying test cases (Not Started)• Research ELINCS Test Tool (DONE)

– Determine what we can leverage– Process flow, source code, test messages

Action Item List IV• Identify all the public health reportable lab results (In progress)• Identify the data elements that differ from the public health IG and the S & I

LRI IG (Not Started)• Determine a policy for validating LRI messages using EHR PH lab results

messages (Not Started)• Develop spreadsheets for managing test cases/data (In progress)

– Adapt tooling to process and incorporate data– Phase 1 nearly complete– Phase 2 will include the multiple dimensions (Data, Profile, Juror)

• Create the HL7 standard message profiles (Starting soon)– MWB (then produce XML message template)– Need to make updates to the message profile based on changes made in version

2.7 and 2.7.1– Write XSLT to modify XML message profile

Action Item List V• Identify the CLIA conformance requirements and compare to

the requirements in the implementation guide (In progress)– Mark in spreadsheet – (DONE)– Make sure conformance requirements and IG match– Write conformance requirements in IG where necessary to match

CLIA requirements• Prototype tool (In progress)

– Requirements and design (In progress)– Development (In progress)– Incorporate test cases (Not Started)– Testing (Not Started)

Validation Methodology

HL7 V2.5.1 Message Profile (LRI IG

assertions)

Test Case Specific Assertions (Validation

Context Files)

Vocabulary (Stored in PHINVADS than

translated to V2.8 Table Library Format)

Validation Engine and Juror Document Generator

Tooling Update• LRI Implementation Guide is in ballot process

– Open for comments until October 17th, 2011• We will begin developing the HL7 standard conformance profile

– MWB– Need to make updates to the conformance profile based on changes made in

version 2.7 and 2.7.1• Data Management of test cases/data will be with spreadsheet

– Spreadsheet is processed to build messages and to create validation context files– Validation context files encapsulates test case related assessment– Leverage/adapt existing NIST Test messages to S&I Framework LRI IG

• Early version of prototype tool developed– Limited functionality– Handles message validation based on message profiles and validation context files– Message Editing