Transitioning a 4 TB Health Care

Security Auditing System to MongoDBMichael Poremba

Director, Data Architecture

Practice Fusion

IntroductionsGetting started

+ 20 years software engineering

+ Data architect / application architect

+ High-volume OLTP relational databases

+ Application performance and scalability

+ Domain experience:Health care; financial services; IT management; content management and distribution;

targeted advertising; telecom billing; manufacturing; insurance

Michael Poremba @ Practice Fusion

+ Cloud-based electronic health records (EHR)

+ Over 100,000 health care providers in US

+ Over 90,000,000 patient medical records

+ OLTP database: Week day peak ~ 40,000 transactions per second

+ 4 TB security auditing records ~ 50% of OLTP database storage

Practice Fusion

+ HIPAA: Health Insurance Portability and Accountability Act of 1996

+ Who did what to which patient’s medical record when?

+ Regulatory requirement—audit log must be kept and reviewed

+ Law enforcement and evidence in legal discovery

+ Save the audit log forever

+ Primary use cases:

Audit report in EHR: Security audit log viewer

Physician data analytics: Clinical quality measures (CQM)

HIPAA Security Audit Log

HIPAA Security Auditing on MongoDB

Project anatomy & lessons learned

Security Auditing – Legacy Architecture

Public

Load

Balancer

App 1

App 2

App n

.

.

.

EHR

(OLTP DB)

ActivityFeed

ActivityFeedParameter

4..8

CQM

(reporting)

ETL

Audit

Report

+ Latency on SAN increased

+ Response time slowed for writes

+ Database connections held longer

+ Connection pool expanded

+ User interface locked up—waiting

+ Users tried to log in again

+ Login is heaviest user operation

+ [Repeat]

The Log Jam

Found at: http://anchorhardwoods.com/wp-content/uploads/2011/08/log-jam.jpg

Audit Service – New Architecture

Public

Load

Balancer

App 1

App 2

App n

.

.

.

MongoDB

Audit Log

Audit

ServiceAMQ

Queue

Listener

Audit

Report

CQM

(reporting)

ETL

+ Isolate auditing system from EHR OLTP database

+ Extract audit IO off of EHR SAN

+ New service interface for audit events

+ Scale out audit service

+ Scale out data store for auditing

Benefits of New Architecture

Project Objectives

+ New infrastructure for MongoDB

and AMQ

+ Modernize audit service API

+ Modernize audit report UI

+ Convert ~200 audit write operations

to new service API

+ Data warehouse ETL from MongoDB

+ Migrate 4 billion exiting audit records

New Security Auditing SystemColetteprogram management

Ernestservices expert

Bhaviktest engineering

Michaeldata architecture

Jeffcluster architecture

JayMongoDB expert

BrettAMQ expert

Bryaninfrastructure coordination

Carlosdata warehouse ETL

+ Transaction volume: Sustain 1,000 new documents per second

+ Data volume: Scale to 10’s of billions of audit event records

+ High availability and disaster recovery—higher SLA than EHR

+ Quick UI response time for interactive audit report

+ Tamper prevention and detection

No updates or deletes permitted on audit log

Security alerts when audit log is altered

+ Leverage industry standards for health care security audit logging

~300 distinct auditable user actions

Required and varying data elements

Security Auditing – Application Requirements

AuditEvent

ParticipantObject

AuditSystem

User

0..n1..1 1..2

Health Care Industry Standards for Audit Logging

+ ISO 27789:2013: Health

Informatics – Audit trails for

electronic health records

+ ASTM E2147-01(2013):

Standard Specification for Audit

Disclosure Logs for Use in

Health Information Systems

+ FHIR SecurityEvent – resource

definition for auditing

{

"_id" : <BinaryData(4)>, // The audit event GUID

"docHash" : <String; Required>, // Tamper detection

"audOrgGuid" : <BinaryData(4); Required>, // Shard key

"crtdDttmUtc" : <Date; Required>, // Datetime record was inserted

"evnt" : {// Required subdocument

"dttmUtc" : <Date; Required>, // Date/time that event occurred

"typ" : <String; Required>, // Event record type; ~ 300 types

"ptDataTyp" : <String; Required>, // Standard set of patient data types

"actn" : <String; Required>, // Standard set of actions

"sys" : <String; Required> // Source system for audit event

},

"usr" : { // Required subdocument

"usrId" : <String; Required>, // Human-readable ID

"usrGuid" : <BinaryData(4); Required>, // Machine-readable ID

"dispNm" : <String; Required>, // Required; Display name for user

"orgId" : <String; Required>,

"orgNm" : <String; Required>

},

"altUsr" : { // Optional subdocument for second user

... // Subdocument contains same properties as "usr"

},

"pt" : { // Optional subdocument

"ptId" : <String; Required>, // Human-readable ID for patient

"ptPracGuid" : <BinaryData(4); Required>, // Machine-readable ID for patient

"dispNm" : <String; Required>, // Display name for patient

"orgId" : <String; Required>,

"orgNm" : <String; Required>

},

"body" : { // Optional subdocument

... // Flattened list of attributes, specific to audit event subtype

}

}

JSON Document Schema for Audit Events

AuditEvent

ParticipantObject

AuditSystem

User

0..n1..1 1..2

Schema Design – Lessons Learned

+ Prop nms strd per doc Long names add up for large collections (ours: 1 TB)

Consider using abbreviated property names

Up-vote this feature request:

https://jira.mongodb.org/browse/SERVER-863

+ Know your application read/write patterns

+ Application responsible for data integrity

+ Be aware of data type behaviors Indexed string search is case sensitive

Several binary data types for UUID—use type 4

(default type is specific to database driver)Found at: http://www.milesfinchinnovation.com/blog/wp-

content/uploads/2013/02/iStock_000019474446Medium.jpg

Schema Design – Lessons Learned

Leverage native data types:

+ Date

+ Boolean

+ Numeric "1" + "1" "11"

"11" + "1" "111"

+ UUID "8c290139-f4e3-49c1-9ba2-a883defc6a15"

"8C290139-F4E3-49C1-9BA2-A883DEFC6A15"

"8c29-0139-f4e3-49c1-9ba2-a883-defc-6a15"

"8c290139f4e349c19ba2a883defc6a15"

"{8c290139-f4e3-49c1-9ba2-a883defc6a15}"

"{8C290139-F4E3-49C1-9BA2-A883DEFC6A15}"

Found at: http://www.industryweek.com/innovation/innovation-one-size-fits-one

ActivityFeed

Audit EventType

ActivityFeed

Parameter

Action TypePatient

Data Type

(~300)

(~4 billion)

(~30 billion)

(10) (18)

UserPatient

(~100,000)(~90 million)

Practice

(~50,000)

Legacy Auditing System – Relational Schema

Issues around data normalization

+ New requirements introduced

+ Filter criteria and sort criteria

stored in five different tables

+ Audit events must be read into

memory for filtering and sorting

Join and expand data set by practice

Sort and filter expanded data set

+ Response time suffers for large

practices with many audit events

Schema Design – Lessons Learned

ActivityFeed

Audit EventType

ActivityFeed

Parameter

Action TypePatient

Data Type

UserPatient

Practice

Denormalize with care:

{

"_id" : <BinaryData(4)>,

"docHash" : <String; Required>,

"audOrgGuid" : <BinaryData(4); Required>,

"crtdDttmUtc" : <Date; Required>,

"evnt" : {

"dttmUtc" : <Date; Required>,

"typ" : <String; Required>,

"ptDataTyp" : <String; Required>,

"actn" : <String; Required>,

"sys" : <String; Required>

},

"usr" : {

"usrId" : <String; Required>,

"usrGuid" : <BinaryData(4); Required>,

"dispNm" : <String; Required>,

"orgId" : <String; Required>,

"orgNm" : <String; Required>

},

"pt" : {

"ptId" : <String; Required>,

"ptPracGuid" : <BinaryData(4); Required>,

"dispNm" : <String; Required>,

"orgId" : <String; Required>,

"orgNm" : <String; Required>

},

"body" : { ... }

}

+ Millions of events per owning organization

+ Quick UI Response Time for Interactive Audit Reports

+ Audit report UI allows events to be sorted/filtered five different ways

+ UI allows paging through audit event

+ Create a secondary index for each sort method

Index Design

+ Organization, event date DESCdb.auditEvent.ensureIndex ( {"audOrgGuid": 1, "evnt.dttmUtc": -1} );

+ Organization, patient, event date DESCdb.auditEvent.ensureIndex ( {"audOrgGuid": 1, "pt.ptId": 1, "evnt.dttmUtc": -1 } );

+ Organization, user, event date DESCdb.auditEvent.ensureIndex ( {"audOrgGuid": 1, "usr.usrId": 1, "evnt.dttmUtc": -1 } );

+ Organization, patient data type, event date DESCdb.auditEvent.ensureIndex ( {"audOrgGuid": 1, "evnt.ptDataTyp": 1, "evnt.dttmUtc": -1

} );

+ Organization, user action type, event date DESCdb.auditEvent.ensureIndex ( {"audOrgGuid": 1, "evnt.actn": 1, "evnt.dttmUtc": -1} );

+ Document created date DESCdb.auditEvent.ensureIndex ( {"crtdDttmUtc": -1 } );

Index Definitions

+ Filter by practice GUID

+ Sort by event created date time, descending order

+ Limit to 20 documents

db.auditEvent.find( {"audOrgGuid": BinData(4,"ABrlAG57Rx6gY3zyHzFK3Q==")} )

.sort( {"evnt.dttmUtc" : -1} ).limit(20).explain();

{

"clusteredType" : "ParallelSort",

"shards" : {

"RepSet02/MNGODDB03-SHRD02:27018, MNGODDB04-SHRD02:27018" : [

{

"cursor" : "BtreeCursor auditEvent_audOrgGuid_dttmUtc",

...

} ] }

...

"numshards" : 1,

...

Query Plan

Indexing Strategy – Lessons Learned

+ As with relational databases,

indexes are essential for efficient

queries

+ Learn how to use .explain()

to read query plans

+ Avoid collection scans:"cursor" : "BasicCursor"

+ For compound indexes, query

sort order must match index sort

orderFound at: http://www.ebay.com/itm/13-pc-Hex-Shank-Titanium-Drill-Bit-Set-Quick-Change-

Bits-/350526103504?pt=LH_DefaultDomain_0&hash=item519cfbdfd0

Principle of least privilege

+ MongoDB cluster not accessible from public Internet

+ Security enabled on cluster

+ Application users granted minimum permissions required

Signed audit events

+ Audit events signed with hash of audit event contents

+ Recompute hash on reads—test the data against hash value

+ Send security alert when hash does not match

Oplog monitoring

+ Use mongo-connector Python scripts to monitor oplog

+ Watch for .update() and .delete() operations on collection

+ Send security alert when data changes are detected

Tamper Prevention and Detection

Found at:http://legacymedia.localworld.co.uk/275663/Article/images/17639732/4416792.jpg

Security – Lessons Learned

+ Minimize network access to

MongoDB cluster

+ Enable authentication

+ Leverage role-based

authorization

+ Use SSL (MongoDB Enterprise)

+ Disable REST interface and

HTTP status interface

Found at: http://www.harborfreight.com/3-1-2-half-inch-circular-padlock-98972.html

+ Shard the database to scale out

+ Begin with small number of shards (2 or 3)

+ Group all audit events from the same medical practice

Every audit event is “owned” by some practice

Audit report UI always queries events by medical practice

+ Composite shard key on { PracticeGuid, _id }db.runCommand({

shardcollection : "AuditLog.auditEvent",

key: {audOrgGuid: 1,

_id: 1}});

Transaction Volume: 1,000 New Documents per Second

Found at:http://s3.amazonaws.com/Reconsales/800/0bfe72e0-9b06-42ac-9644-5727a3ca9c79.jpg

Sharding the Database – Lessons Learned

+ At the onset of development

determine whether to shard

+ Specify shard key in queries Allows mongos to route query

Minimize distributed “scatter/gather” queries

Queries spanning chunks likely span shards

+ Choose a key that allows even

balancing Balancing is performed in 32 MB chunks

Design shard key to ensure chunks will not

exceed 32 MB

Found at: http://www.airbrushaction.com/content/sites/default/files/tipstricks-images/4_27.png

High Availability and Disaster Recovery – Replica Sets

+ If audit log is down, then 100,000

health care providers are idle

+ Audit logging subsystem must be

more reliable than customer EHR

+ Node failover must be automatic

+ Protect against network and data

center failure scenarios

Found at: http://www.huntsmart.com/App_Themes/hs.com/ProductImages/250/DNSBC.jpg

Disaster Recovery DCPrimary DC DC2 AZ2

Sharded Cluster Replicated Across Multiple Data Centers

config

mongos shard 2

arbitermongos

amq

arbiter

amq

DC3 AZ1

shard 2

DC2 AZ1

shard 2

mongos shard 3

arbitermongos

arbiter

shard 3shard 3

mongos shard 1

arbitermongos

arbiter

shard 1shard 1

config config

amq amq

Performance and Stress Testing – Lessons Learned

+ Acquire or build load testing tools

+ Test using a realistic, unbiased data set

+ Test database cluster to ensure write

throughput

+ Ensure read & write performance meets

load requirements

+ Find the performance ceiling

+ Find and resolve bottlenecks

+ Tune IO and memory

Found at: http://www.webdesign.org/img_articles/21892/broken_chain.jpg

Data Migration – Lessons Learned

Data Migration

+ Parallelize data migration process

+ Identify and remove bottlenecks

+ Scale out MongoDB cluster to handle

heavy write load

+ Determine whether best to add

indexes before or after migration

+ It takes a while to extract, transform,

and load billions of documentsFound at: http://www.dennissy.com/wp-content/uploads/2010/07/house_moving_malaysia.jpg

Choosing the Appropriate Data Store

MongoDB over relational?

+ Scale out for transaction volume

and data volume

+ Highly varying document

structure

+ Developer productivityEasy map between application and data store

+ Offload read activity in optimized

format different from data writes(a.k.a. CQRS pattern)

Found at: http://www.meonuk.com/hammers-mauls

Choosing the Appropriate Data Store

Relational over MongoDB?

+ Complex normalized data model

+ Diverse read patterns requiring

joins

+ Ad hoc reporting and analysis

+ Data integrity difficult to manage

in application layerFound at:

http://3.bp.blogspot.com/_QUmmdgc7l6A/TTPUyRWFNPI/AAAAAAAAAO8/KV_i2c2lrRk/s1600/saws+various.jpg

MongoDB @ Practice Fusion

Upcoming MongoDB projects

+ Read cache for patient medical

records

+ Online patient intake process

+ Ad campaign segmentation

+ Scale-out data store for

patient clinical observationsFound at: http://jbirdmedia.org/vessels/images/uploads/framing-new-const-lg.jpg

Q&A

Michael Poremba

mporemba@practicefusion.com