1

Winning Strategies for a SustainableLab Informatics Platform

Ulysses J. Balis, M.D.Director, Division of Pathology Informatics

Department of PathologyUniversity of Michigan Health System

ulysses@med.umich.edu

Overview

• Some general observations• Case studies with lessons learned• Sustainable, winning strategies to consider• Some closing remarks• Open discussion and questions

2

478.2 TFLOPS 1.026 PFLOPS

35 OPS

Data stewardship approach: circa 1972

ThenFrom where have we traveled…

Now

3

Information Technology Evolution

• Boils down to three sectors of capability / capacity– Computational throughput

• Transactional load / scaling– Storage– Network connectivity, Infrastructure and redundancy

• Virtualization– n+1 architectures– Seamless failover– Distributed architectures– SAAS (software as a service / remote hosting models)

and…• Appliances

– Interface engines– Middleware solutions and engines– Whole slide scanners– Tracking technology

• Labeling / marking with symbologies• Reading of such symbologies

– CRM tools– Logistics tools

These are, or soon will be, commoditized

Case Study 1• Large Integrated Academic Health Center with aggressive 18 month

implementation cycle for new LIS• Bing-bang go-live for new LIS• One of two database instances fails at beginning of Go-live date• Upon Go-live, accessioning of a single routine specimen was taking

7-12 minutes• Under-developed/untested downtime procedure & planning in the

eventuality of failed crossover• By day two, post-go-live, over 4000 specimens were time-expired

and were necessarily discarded• Hospital CEO, CIO and CFO escalated growing crisis to vendor

leadership; emergency team flown in by end of day two• Resolution of problems over next several weeks; overall process

deemed as “profoundly unpleasant” by all involved

4

Case Study 1: Post-mortem• No load testing performed

– Ultimately, it turned out that the system, as initially configured, was underpowered for the nominal transactional load and intended user concurrency. A further diminution of 50% functionality, with the loss of the second database instance, was therefore catastrophic.

• Team lead tasked with the coordination of training and documentation of such did not communicate the facts that:– Medical Technology staff largely ignored the multiple scheduled training

sessions – Professional staff largely ignored the multiple scheduled training

sessions– Consequently, there was no escalation of a looming crisis where most

of the lab would be un/under-prepared to effectively utilize the new system

– Lack of user adroitness further impeded the accessioning process, creating an insurmountable upstream bottleneck; the system was unusable

Case Study 1: Lessons Learned• No undelivered communications: team leads must communicate both good

and bad news, fostered by a culture that does not place blame.• Load testing must be performed; ideally, inserting a factor of safety for high-

utilization periods and future growth• A program of documentation of competency would have detected the non-

compliance with LIS proficiency training and allowed for appropriate intervention

• Go-live dates are not immutable– Although undesirable from both a logistics and financial perspective, certain

classes of eventualities justify date postponement, especially when patient care is a concern

Note that I.T. is not the fundamental predicate for failure here; rather it was a failure of leadership combined with over-valued concern for preservation of an unrealistic implementation schedule. In fact, many “I.T. failures” are not fundamentally I.T. at all.

5

Case Study 2• An academic center that employs remote hosting for its

LIS operations loses connectivity with the off-site server cluster, owing to road construction outside the immediate hospital campus, causing the fiber optic network connection to be severed.

• The backup internet fiber connection, via a separate commercial network vendor was similarly severed as it too exited the building via the same demarc and underground plenum

• Restoration required 24 hours, leading to dependence on paper-based, downtime procedures for that interval and significant rework in manually entering data, upon restoration of connectivity to the LIS.

Case Study 2: Post-Mortem

• Internet connectivity was provisioned for the enterprise at a time preceding the emergence of SAAS and remote hosting models

• Single points of failure, from an I.T. perspective, were not considered.

• With the advent of SAAS, this was an incident waiting to happen

6

Case Study 2: Lessons Learned• Single points of failure are everywhere, and single-threaded/single instance

I.T. solutions are particularly susceptible to this failure mode• What is needed is a policy of continuous surveillance for existence or entry

of such points, as infrastructure evolves• Common areas to review:

– Internet access/building demarc– Single instances of critical servers/applications with not hot-standby redundant

instances– Room level (and not rack-level network switching); in fact, rack-level switching

should be fully redundant– Single power feeds & lack of backup power supplies for mission critical core

services– On the lab floor, suitable redundancy of specialized computational nodes (e.g.

console for automation lines / core labs track control, etc.)• Often, a single extra PC will address this need

– Single points of failure also occur with stewardship of applications; need to ensure that more than one or two individuals in an enterprise understand the support and technical underpinnings of every critical service.

Case Study 3(by way of Raymond Aller: The ABC’s of LIS)

• Data center operator hears a deafening screeching sound from one of the legacy system disk stacks.

• Unloads the errant “cakeplatter” and loads the most recent backup instance.

• Upon restart, another defining screech is heard.• Operator proceeds to repeat this approach with

the additional backup platters, leading to comprehensive application data loss.

7

Case Study 3: Post-Mortem

• Operators were inadequately trained to perform the tasks with which they were charged

• There was no system of periodically documented competencies, as there is with clinical laboratory medicine, to validate that core competencies were intact

• As a result, this was another accident waiting to happen

Case Study 3: Lessons Learned• Global I.T. stewardship is only as reliable as the service

layer that supports the hardware infrastructure layer; the two are inextricably linked.

• Periodic competency assessment is not an optional process

• This area has become an increasing challenge for pathology departments in the era of I.T. consolidation into central I.T. departments that my not possess adequate domain expertise for appropriately sophisticated stewardship

• Pathology regaining some measure of local control/ autonomy is highly desirable.

8

Winning Strategy 1• Is not technology at all; rather it is the affirmation of

importance for recognizing the extant process culture and recognizing the need for careful change management in the setting of anticipated IT deployment, including:– Impeccable transparency with no undelivered communications– True empowerment of the intended lab personnel with respect to

intended technology deployment– Willingness to realign staff who are unable to adopt to a

continuous culture of change and improvement• Essential to recognize that IT is a tool and not a means

in itself. Unmet Need drives IT implementation and innovation; IT should never itself be the driving agent.

Winning Strategy 2

• Regain appropriate oversight and local stewardship from enterprise I.T. initiatives that have subsumed everything, and in so doing, lost the ability to:– Provide timely, domain specific support– Provide timely support and strategically-

selected architectures– Lost the core value of customer support

9

Winning Strategy 3:Federation and Service-Oriented Architectures

• Recognize that our growing compendium of one-off interfaces and our silo-based repository strategy is unsustainable

• Systematically transition to interoperable interfaces which are reusable and easily deployed/maintained

• Leverage this goal at the time points that afford greatest influence:– RFP provisioning and gap analysis– Contract negotiations

What is Federation?(at the hospital level)

UserLIS RIS OR E.D. other

Participatory SQL servers

Web-basedJust-in-timeaggregation Single SQL (or ODBC) query

Concurrently vectored to eachParticipatory Single Source of Truth

Consequences of shifting to a SQL-based SSOT model:•Data only represented once in overall enterprise model•Reduction in number of interfaces requiring support•Potential to transfer classes information other than text•Reduction in support responsibilities of central hospital IT.

10

Federation(at the Pathology Department level)

LISUser

Chem Coag Micro Mol. other

Participatory SQL servers

Just-in-timeaggregation

The above model support best-of-breed solutions and modular implementations.

This will be essential to support personalized medicine reporting models, where clinical data will be required to complete the data-interpretive analysis steps

Service-Oriented Architecture

Transformation of our Transactional Model to a Relationship model (federation and beyond)

• Federation addresses single source of truth issues and just in time data aggregation

• It does not address distributed knowledge semantic models, that are predicted and required by relational networks

• The solution is reciprocal federation

11

Reciprocal Federation(at the Pathology Department level)

LISUser

LIS RIS OR E.D. other

Participatory SQL servers

Just-in-timeaggregation

A reverse feed from other clinical repositories back to the LIS / AP-LIS allows for simplified retrieval of clinical information at the time of decision, by laboratory staff.

At present a non-realized / under-realized functionality.

Interop 2009 Project Design• Partnership of Industry, government and academia

– Initially announced in March of 2008– At present:

• Five major LIS vendors• One data interpretive services vendor• National Cancer Institute caBIG biospecimen repository working group

– Dr. Ian Fore• University of Michigan

– Create a grid-services solution for demonstration at LITS-2009 that would demonstrate the following:

• Seamless interoperability of exchange of lab results between commercial LISs• Interactive format for LITs attendees, similar to DICOM / IHE “connecathons” of the

prior two decades– Effort for implementation not to exceed 40 man-hours per site– All tools based on open-source architectures and software– All tools and standards to be released to the public domain– Core functionality to serve as the edifice for subsequent expansion of the grid

services– Fully functional in time for LITS-2009

12

Federant1

Federant2

Federant4

Federant3

INTEROP CloudINTEROP Cloud

ExternalCloud

ServicesLIS

InitiatingQuery

1.Original Query. Patient hash provided to cloud

3. Cloud Initiates an internal service request, forwarding aggregated raw patient data for Interpretive Service Class action from the remote server

4. Remote Server Returns Interpretation message to cloud

via private internal services-level connection

5. All patient results from all responding sites aggregated with personalized interpretive report and returned to requesting LIS

2. Retrieval of hash, subsequent query at each LIS site to obtain same-patient data across federation

labinfotech summitINTEROP-2009

13

Federant1

Federant2

Federant4

Federant3

INTEROP CloudINTEROP Cloud

ExternalCloud

ServicesLIS

InitiatingQuery

1.Original Query. Patient hash provided to cloud

3. Cloud Initiates an internal service request, forwarding aggregated raw patient data for Interpretive Service Class action from the remote server

4. Remote Server Returns Interpretation message to cloud

via private internal services-level connection

5. All patient results from all responding sites aggregated with personalized interpretive report and returned to requesting LIS

2. Retrieval of hash, subsequent query at each LIS site to obtain same-patient data across federation

labinfotech summitINTEROP-2009

Federant1

Federant2

Federant4

Federant3

INTEROP CloudINTEROP Cloud

ExternalCloud

ServicesLIS

InitiatingQuery

1.Original Query. Patient hash provided to cloud

3. Cloud Initiates an internal service request, forwarding aggregated raw patient data for Interpretive Service Class action from the remote server

4. Remote Server Returns Interpretation message to cloud

via private internal services-level connection

5. All patient results from all responding sites aggregated with personalized interpretive report and returned to requesting LIS

2. Retrieval of hash, subsequent query at each LIS site to obtain same-patient data across federation

labinfotech summitINTEROP-2009

14

Federant1

Federant2

Federant4

Federant3

INTEROP CloudINTEROP Cloud

ExternalCloud

ServicesLIS

InitiatingQuery

1.Original Query. Patient hash provided to cloud

3. Cloud Initiates an internal service request, forwarding aggregated raw patient data for Interpretive Service Class action from the remote server

4. Remote Server Returns Interpretation message to cloud

via private internal services-level connection

5. All patient results from all responding sites aggregated with personalized interpretive report and returned to requesting LIS

2. Retrieval of hash, subsequent query at each LIS site to obtain same-patient data across federation

labinfotech summitINTEROP-2009

Federant1

Federant2

Federant4

Federant3

INTEROP CloudINTEROP Cloud

ExternalCloud

ServicesLIS

InitiatingQuery

1.Original Query. Patient hash provided to cloud

3. Cloud Initiates an internal service request, forwarding aggregated raw patient data for Interpretive Service Class action from the remote server

4. Remote Server Returns Interpretation message to cloud

via private internal services-level connection

5. All patient results from all responding sites aggregated with personalized interpretive report and returned to requesting LIS

2. Retrieval of hash, subsequent query at each LIS site to obtain same-patient data across federation

labinfotech summitINTEROP-2009

15

Core Interop Cloud Server Priorto Attachment of Clients.

16

Core Interop Cloud Server WithAttached Clients.

17

Winning Strategy 4:Evolution of Storage to Distributed Virtualization

• Prior Technology Layer: Storage Area Networks (SANs) and RAID– Although-fault tolerant, such configurations represented a single point of

failure, should the server chassis fail• Enterprise Virtualization

– Akin to SANS/RAID technology distributed over two or more locations (nodes)

– One or more whole nodes may fail without untoward impact on the overall availability of data functionality

• Potentially Disruptive– Calls into question the entire justification for monolithic datacenters;

rather multiple nodes can be weaved into the “fabric” of the building infrastructure

• Multiple on-site nodes• Multiple off-site nodes

– Assume that failures will routinely occur and simple factor them into the overall support model

18

Winning Strategy 5:Virtual Computational Environments

with Seamless Failover

• With Virtualized Storage also comes virtualized computation– Notion that the application thread can seamlessly

“jump” from one server to a redundant instance in real time, without the need for manual configurationalcrossover.

– Not the same as clustered computation or multi-core processing

– An entire datacenter node could vanish and users would not experience a service interruption, as the process thread jumps to an alternate datacenter

– Technology available today (VmWare and others)

Winning Strategy 5:Seamless Integration of Logistics Modules and CRM

with Extant LIS platforms

• CRM and Logistics are rapidly becoming expected services for any outreach or extended enterprise service level agreement

• Option to integrate new vendor-supplied modules or stand-alone federated modules to existing new and legacy systems

• Invaluable in addressing segments of the value stream outside traditional oversight of the lab– Patient bedside and procedure room identification

and tracking of specimens, far sooner than the specimen receiving entry point.

19

Winning Strategy 6:Electronic Documentation Systems

• Policies and Procedures– Cross-reference between CAP checklist questions

and an evidentiary packet that substantiates compliance

• Online Handbook– Single source of truth for both the department and the

enterprise at large• Online CME/CEU Tracking and content

repositories• Online Lean tools and A3 repository• Online Maintenance logs• Online Competencies

Winning Strategy 7:Logical Completion of the OngoingAP Workflow Automation Process

• Recognize that AP is the current greatest source of threat to patient safety within the overall specialties

• The public is increasingly aware of this• Integration of tracking and Lean tools into

the AP LIS layer itself will simplify this transformation

20

Winning Strategy 8:Incremental deployment of digital pathology

solutions in settings where markets are ready for it

• No need to implement a “wall-to-wall” solution to take advantage of this technology

• Adoption curve will likely follow niche-based areas, where the value proposition exceeds current consultative models supported by courier delivery of slide and block materials

• Total cost of ownership already viable with lease/rent models for scanners

Larger Context of the Path Leading to theDigital Pathology Department:not a question of if but when…• The emergence of patient safety and

operational gains made possible by the collective technologies that comprise Digital Pathology are significant.

• While not all the changes are immediately imminent, there is little question that the transformations pathology will experience will be sooner than many consider, and similarly, more substantive

• This process should not be feared but rather, embraced

“Enjoy the Vogon Poetry…” --Douglas Adamsfrom Hitchhiker’s Guide to the Galaxy

21

With the current expanding plurality of whole-slide scanners, it is merely a matter of time until Anatomic Pathology transforms into becoming a (nearly) all-digital modality.

With the advent of all-digital image management, a number of problems will arise:•Cost of storage•Difficulty with the stewardship of digital repositories•The need to develop tools and techniques to take advantage of having the majority of data in digital format.

The solution to the above is the use of rationally-designed workflow and IT approaches that are specific for the needs of the modality. We should not simply reproduce the Digital Radiology Continuum (that would be a surefire recipe for failure)

Whole slide images

X ray image from Dah http://farm3.static.flickr.com/2114/2141439225_4b996372df.jpg?v=0

•12 gigapixels (for one image)• each case can be 2-20 images• 1Gb even after image compression• A different challenge than faced Radiology 20 years ago

22

Current World View of Pathology Imagery Repositories

• Model 1: Relational Database– Image Metadata associated with case-level data– Entire Schema required to carry out discovery– Text-based– Image data is a passive component of the query

• Model 2: Metadata-tagged Images– Image Metadata associated with each image– Image becomes a self-contained dataset available

for discovery– Text-based– Image data is a passive component of the query

Entry in masteraccessiontable

Associated caseand image descriptors

Associated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptorsAssociated caseand image descriptors

Thesis Statement• The availability of digital whole slide data

sets represent an enormous opportunity to carry out new forms of numerical and data-driven query, in modes not based on textual, ontological or lexical matching.– Search image repositories with whole

images or image regions of interest– Carry our search in real-time via use of

scalable computational architectures

Extraction from Imagerepositories based upon

spatial information

Analysis of datain the digital domain

…001011010111010111..

Resultant Surface Map orgallery of matching images

or

23

Winning Strategy 9:Leverage interoperability for improved clinical

communication and extended market scope

• Interoperability will allow for aggregation of data sets that will be of immense value to personalized medicine and complex integrated reporting, where geographically distributed diverse data types are required

• Leveraging of RFPs and new contracts will drive the LIS market to embrace standards; in fact, the market has already recognized the intrinsic power of interoperability and will likely help drive the transformation itelf

Closing Remarks

• I.T. should be viewed as a powerful tool but not an end-all of itself

• We as a specialty, MUST secure a critical mass of I.T. stewardship back from the enterprise at large, where effective oversight has often suffered

• The future of advanced I.T. deployment in Pathology and Lab Medicine is very compelling: be optimistic and be aggressive!

24

Acknowledgements

• Bruce Friedman, MD; Professor Emeritus, University of Michigan

• Kathy Davis, MT (ASCP), System Manager, Division of Pathology Informatics, University of Michigan