Frederic S Resnic MD MScHenry Ssemaganda MD MScSusan Robbins BSComparative Effectiveness Research InstituteLahey Clinic Medical Center

Prospective Active Surveillance The DELTA System

User Training Session

October 22 2018

1

2

bull Introduction to Automated Surveillance (Prospective Active Surveillance)

bull Overview of DELTA System

bull CathPCI Pilot Example

(Break)

bull DELTA User Support

bull DELTA Demo Interactive Session

Agenda

3

Introduction to DELTA

Automated Safety Surveillance

Frederic S Resnic MD MSc

Challenges of Existing Post-Market Medical Device Surveillance Approaches

1 Adverse event reporting unable to

estimate event rates (lsquodenominatorrsquo

issue) non-representative

(reporting behavior) resource

intensive

2 Post-Approval Studies generally

underpowered without

contemporary comparator groups

for low frequency safety events

3 Current approaches lack systematic

evaluation of real-world

performance information to provide

timely information to providers

manufacturers and regulators

Complementary role forProspective Near Real-Time Active Safety Surveillance

bull The term ldquoActive Surveillancerdquo implies differing monitoring strategies to different stakeholders

bull CDRH Non-research efforts that are part of routine post-market vigilance programs for specific (typically high-risk) medical devices

bull Industry Exploratory non-research post-market device performance monitoring

bull Academic Prospective pre-specified risk-adjusted time-limited analysis of RWE

bull Key attributes of medical device Active Surveillance

bull Timely Primary goal is to assess device performance in time frame that permits appropriate action (by manufacturer clinicians and regulators)

bull Repeated Assessments Minimize delay in identifying safety signal or absence of signal

bull Methodologically Appropriate Appropriate risk adjustment comparators and signal thresholds Minimize Type II error (false alarms) without missing opportunities to detect real issues

Defining Active Surveillance

bull Automated Surveillance extends Active Surveillance by leveraging software tools to support the pre-specified repeated evaluations of an accruing dataset

bull Automated Surveillance enables

bull Near real-time prospective evaluation of accruing experience

bull Simultaneous monitoring of multiple devices and outcomes

bull Application of robust data security protocols for remote data monitoring

Defining ldquoAutomatedrdquo Surveillance

Retrospective Single Study

Device Performance

Evaluation

Real Time Routine

Continuous Device

Performance MonitoringRetrospective

SequentialStudy

Accruing Data Sequential

Study

Accruing Data Near Real-Time

Sequential Study

Accruing Data Near Real-Time

Sequential Programmed

Active

Surveillance

Automated

Surveillance

Inexorable progression toward real-time high quality RWE availability as EHR RegistryCRN

and Informatics technologies mature

JAMIA 2006

JAMA 2010Circ Card Qual 2012

7

Device Active Surveillance 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

2

bull Introduction to Automated Surveillance (Prospective Active Surveillance)

bull Overview of DELTA System

bull CathPCI Pilot Example

(Break)

bull DELTA User Support

bull DELTA Demo Interactive Session

Agenda

3

Introduction to DELTA

Automated Safety Surveillance

Frederic S Resnic MD MSc

Challenges of Existing Post-Market Medical Device Surveillance Approaches

1 Adverse event reporting unable to

estimate event rates (lsquodenominatorrsquo

issue) non-representative

(reporting behavior) resource

intensive

2 Post-Approval Studies generally

underpowered without

contemporary comparator groups

for low frequency safety events

3 Current approaches lack systematic

evaluation of real-world

performance information to provide

timely information to providers

manufacturers and regulators

Complementary role forProspective Near Real-Time Active Safety Surveillance

bull The term ldquoActive Surveillancerdquo implies differing monitoring strategies to different stakeholders

bull CDRH Non-research efforts that are part of routine post-market vigilance programs for specific (typically high-risk) medical devices

bull Industry Exploratory non-research post-market device performance monitoring

bull Academic Prospective pre-specified risk-adjusted time-limited analysis of RWE

bull Key attributes of medical device Active Surveillance

bull Timely Primary goal is to assess device performance in time frame that permits appropriate action (by manufacturer clinicians and regulators)

bull Repeated Assessments Minimize delay in identifying safety signal or absence of signal

bull Methodologically Appropriate Appropriate risk adjustment comparators and signal thresholds Minimize Type II error (false alarms) without missing opportunities to detect real issues

Defining Active Surveillance

bull Automated Surveillance extends Active Surveillance by leveraging software tools to support the pre-specified repeated evaluations of an accruing dataset

bull Automated Surveillance enables

bull Near real-time prospective evaluation of accruing experience

bull Simultaneous monitoring of multiple devices and outcomes

bull Application of robust data security protocols for remote data monitoring

Defining ldquoAutomatedrdquo Surveillance

Retrospective Single Study

Device Performance

Evaluation

Real Time Routine

Continuous Device

Performance MonitoringRetrospective

SequentialStudy

Accruing Data Sequential

Study

Accruing Data Near Real-Time

Sequential Study

Accruing Data Near Real-Time

Sequential Programmed

Active

Surveillance

Automated

Surveillance

Inexorable progression toward real-time high quality RWE availability as EHR RegistryCRN

and Informatics technologies mature

JAMIA 2006

JAMA 2010Circ Card Qual 2012

7

Device Active Surveillance 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

3

Introduction to DELTA

Automated Safety Surveillance

Frederic S Resnic MD MSc

Challenges of Existing Post-Market Medical Device Surveillance Approaches

1 Adverse event reporting unable to

estimate event rates (lsquodenominatorrsquo

issue) non-representative

(reporting behavior) resource

intensive

2 Post-Approval Studies generally

underpowered without

contemporary comparator groups

for low frequency safety events

3 Current approaches lack systematic

evaluation of real-world

performance information to provide

timely information to providers

manufacturers and regulators

Complementary role forProspective Near Real-Time Active Safety Surveillance

bull The term ldquoActive Surveillancerdquo implies differing monitoring strategies to different stakeholders

bull CDRH Non-research efforts that are part of routine post-market vigilance programs for specific (typically high-risk) medical devices

bull Industry Exploratory non-research post-market device performance monitoring

bull Academic Prospective pre-specified risk-adjusted time-limited analysis of RWE

bull Key attributes of medical device Active Surveillance

bull Timely Primary goal is to assess device performance in time frame that permits appropriate action (by manufacturer clinicians and regulators)

bull Repeated Assessments Minimize delay in identifying safety signal or absence of signal

bull Methodologically Appropriate Appropriate risk adjustment comparators and signal thresholds Minimize Type II error (false alarms) without missing opportunities to detect real issues

Defining Active Surveillance

bull Automated Surveillance extends Active Surveillance by leveraging software tools to support the pre-specified repeated evaluations of an accruing dataset

bull Automated Surveillance enables

bull Near real-time prospective evaluation of accruing experience

bull Simultaneous monitoring of multiple devices and outcomes

bull Application of robust data security protocols for remote data monitoring

Defining ldquoAutomatedrdquo Surveillance

Retrospective Single Study

Device Performance

Evaluation

Real Time Routine

Continuous Device

Performance MonitoringRetrospective

SequentialStudy

Accruing Data Sequential

Study

Accruing Data Near Real-Time

Sequential Study

Accruing Data Near Real-Time

Sequential Programmed

Active

Surveillance

Automated

Surveillance

Inexorable progression toward real-time high quality RWE availability as EHR RegistryCRN

and Informatics technologies mature

JAMIA 2006

JAMA 2010Circ Card Qual 2012

7

Device Active Surveillance 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

Challenges of Existing Post-Market Medical Device Surveillance Approaches

1 Adverse event reporting unable to

estimate event rates (lsquodenominatorrsquo

issue) non-representative

(reporting behavior) resource

intensive

2 Post-Approval Studies generally

underpowered without

contemporary comparator groups

for low frequency safety events

3 Current approaches lack systematic

evaluation of real-world

performance information to provide

timely information to providers

manufacturers and regulators

Complementary role forProspective Near Real-Time Active Safety Surveillance

bull The term ldquoActive Surveillancerdquo implies differing monitoring strategies to different stakeholders

bull CDRH Non-research efforts that are part of routine post-market vigilance programs for specific (typically high-risk) medical devices

bull Industry Exploratory non-research post-market device performance monitoring

bull Academic Prospective pre-specified risk-adjusted time-limited analysis of RWE

bull Key attributes of medical device Active Surveillance

bull Timely Primary goal is to assess device performance in time frame that permits appropriate action (by manufacturer clinicians and regulators)

bull Repeated Assessments Minimize delay in identifying safety signal or absence of signal

bull Methodologically Appropriate Appropriate risk adjustment comparators and signal thresholds Minimize Type II error (false alarms) without missing opportunities to detect real issues

Defining Active Surveillance

bull Automated Surveillance extends Active Surveillance by leveraging software tools to support the pre-specified repeated evaluations of an accruing dataset

bull Automated Surveillance enables

bull Near real-time prospective evaluation of accruing experience

bull Simultaneous monitoring of multiple devices and outcomes

bull Application of robust data security protocols for remote data monitoring

Defining ldquoAutomatedrdquo Surveillance

Retrospective Single Study

Device Performance

Evaluation

Real Time Routine

Continuous Device

Performance MonitoringRetrospective

SequentialStudy

Accruing Data Sequential

Study

Accruing Data Near Real-Time

Sequential Study

Accruing Data Near Real-Time

Sequential Programmed

Active

Surveillance

Automated

Surveillance

Inexorable progression toward real-time high quality RWE availability as EHR RegistryCRN

and Informatics technologies mature

JAMIA 2006

JAMA 2010Circ Card Qual 2012

7

Device Active Surveillance 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

bull The term ldquoActive Surveillancerdquo implies differing monitoring strategies to different stakeholders

bull CDRH Non-research efforts that are part of routine post-market vigilance programs for specific (typically high-risk) medical devices

bull Industry Exploratory non-research post-market device performance monitoring

bull Academic Prospective pre-specified risk-adjusted time-limited analysis of RWE

bull Key attributes of medical device Active Surveillance

bull Timely Primary goal is to assess device performance in time frame that permits appropriate action (by manufacturer clinicians and regulators)

bull Repeated Assessments Minimize delay in identifying safety signal or absence of signal

bull Methodologically Appropriate Appropriate risk adjustment comparators and signal thresholds Minimize Type II error (false alarms) without missing opportunities to detect real issues

Defining Active Surveillance

bull Automated Surveillance extends Active Surveillance by leveraging software tools to support the pre-specified repeated evaluations of an accruing dataset

bull Automated Surveillance enables

bull Near real-time prospective evaluation of accruing experience

bull Simultaneous monitoring of multiple devices and outcomes

bull Application of robust data security protocols for remote data monitoring

Defining ldquoAutomatedrdquo Surveillance

Retrospective Single Study

Device Performance

Evaluation

Real Time Routine

Continuous Device

Performance MonitoringRetrospective

SequentialStudy

Accruing Data Sequential

Study

Accruing Data Near Real-Time

Sequential Study

Accruing Data Near Real-Time

Sequential Programmed

Active

Surveillance

Automated

Surveillance

Inexorable progression toward real-time high quality RWE availability as EHR RegistryCRN

and Informatics technologies mature

JAMIA 2006

JAMA 2010Circ Card Qual 2012

7

Device Active Surveillance 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

bull Automated Surveillance extends Active Surveillance by leveraging software tools to support the pre-specified repeated evaluations of an accruing dataset

bull Automated Surveillance enables

bull Near real-time prospective evaluation of accruing experience

bull Simultaneous monitoring of multiple devices and outcomes

bull Application of robust data security protocols for remote data monitoring

Defining ldquoAutomatedrdquo Surveillance

Retrospective Single Study

Device Performance

Evaluation

Real Time Routine

Continuous Device

Performance MonitoringRetrospective

SequentialStudy

Accruing Data Sequential

Study

Accruing Data Near Real-Time

Sequential Study

Accruing Data Near Real-Time

Sequential Programmed

Active

Surveillance

Automated

Surveillance

Inexorable progression toward real-time high quality RWE availability as EHR RegistryCRN

and Informatics technologies mature

JAMIA 2006

JAMA 2010Circ Card Qual 2012

7

Device Active Surveillance 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

JAMIA 2006

JAMA 2010Circ Card Qual 2012

7

Device Active Surveillance 10 Years of Learning Lessons

NEJM 2017

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

CathPCI DELTA Study Background

bull Vascular Closure Devices (VCD) approved for use following cardiac catheterization procedures

bull VCD use variety of mechanicalpharmacologic designs and delivery mechanisms to secure device and accelerate clotting

bull PMA studies generally small and limited high quality post-market safety data regarding VCD

bull Mynx VCD suspected of having higher failure rates than alternative VCD

bull Observations from 5-hospital prospective safety surveillance (Kumar et al - Circ Card Qual Outcomes 2014)

bull Post-publication analysis of national registry study (Tavris

et al - J Invasive Card 2012)

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

CathPCI DELTA Study Background

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

CathPCI DELTA Study Hypothesis

1 Active prospective surveillance of national cardiovascular registry will demonstrate

Mynx VCD is associated with increased rates of post-procedure vascular complications andor bleeding compared with propensity matched patients receiving alternative VCD

2 Additionally Mynx safety signal will be heightened in high risk patient subgroups women agegt70yrs and diabetic patients

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

CathPCI DELTA Study Methods Overview

bull Prospective observational study using Propensity Matching (11) to compare similar populations of patients receiving Mynx following PCI to patients receiving other ldquoactiverdquo VCD

bull Primary endpoint Any vascular complication ndash composite of access site hematoma requiring treatment access site bleeding requiring intervention or RPH

bull Pre-specified protocol and statistical plan including adjustment for multiple comparisons and planned interim data reviews [Avoid ldquoData miningrdquo]

bull Pre-specified high risk patient subsets (DM female agege70yr) sensitivity analyses and ldquofalsification hypothesisrdquo analysis

bull Study oversight committee recommended post-hoc analyses including new falsification hypothesis and signal verification study

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

13

CathPCI DELTA Study Patient Flow Diagram

Source Resnic FS Majithia A et al NEJM Jan 2017

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

Methods Review What is Propensity Matching

ALL PCI Patients Receiving Mynx ALL PCI Patients Receiving Other VCD

Matched Mynx PatientsUnmatched

Mynx Patients Matched Non-Mynx Patients

PS

=

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

15

CathPCI DELTA Study Primary AnalysisMynx VCD was associated with a 59 increased risk of vascular complications

as compared with alternative VCD Signal apparent within 9 months

Source Resnic FS Majithia A et al NEJM Jan 2017

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

Pre-specified subgroup analysis was performed for high risk subgroups including women the agegt70 and patients with diabetes

Female Patients Age gt 70yr

Diabetic Patients

CathPCI DELTA Study Results High Risk Populations

Source Resnic FS Majithia A et al NEJM 2017

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

17

CathPCI DELTA Falsification Hypothesis

Source Resnic FS et al NEJM Jan 2017 (Supplementary Appendix)

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

bull Multiple options each with inherent limitations

bull Risk-adjusted performance compared with OPC ndashquestions of validity of OPC

bull Risk-adjusted performance compared with historical performance of alternative therapeutic option - issues around learning impacts and secular treatment trends

bull Direct device-device (or product group) comparisons (propensity matched or adjusted) ndash significant stakeholder sensitivity

bull Guiding Principle Select comparator to maximize validity within limitations of stakeholder sensitivitytolerance

Methodologic Challenge Selecting ComparatorsV

ali

dit

y

Se

ns

itiv

ity

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

Methodologic Challenge Selecting Methods

bull Multiple options for analyzing serial outcomes data using surveillance framework

bull Propensity matched analysis ndash robust addresses confounding most interpretable Weaknesses truly novel technology multiple comparisons treatment selection biases sample loss

bull Propensity adjusted analysis ndash robust addresses confounding maximizes sample Weaknesses interpretability multiple comparisons

bull Sequential Testing (CSUM SPRT etc) ndash indefinite surveillance address multiple comparison generalizable Weaknesses interpretability risk adjustment

bull Guiding Principle Maximize interpretability while minimizing risk of confounding

Ge

ne

rali

za

bilit

y

Inte

rpre

tab

ilit

y

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

bull Selecting appropriate thresholds for triggering additional explorationaction is critical Must be part of written protocol endorsed by all prior to start of data analysis

bull Recommended Select clinically meaningful difference in performance

bull Can differ based on outcome studied Small differences in mortality with larger differences tolerated with less serious outcomes

bull Select alpha (false-positive rate) based on seriousness of outcome Acceptable to be different from 005

bull Example ICD Lead failure absolute difference ge 10 than comparator lead at 1-year α=005 Death alerts at ge 5 absolute difference α=010

Methodologic Challenge Alert Thresholds

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

Advantages of DELTA Surveillance

1 Automated Continuous Prospective Active Surveillance suitable for monitoring of large clinical datasets using straightforward analytic methods

2 Supports multiple simultaneous active surveillance analyses

3 Pre-specified analytic plan DELTA configured at outset of analysis to perform studies in accordance with SAPprotocol

4 Propensity matching is uniquely suited to comparative safety analyses Statistically robust very easy to interpret and explain and is conducive to post-hoc analysis for signal exploration

5 Pragmatic and Scalable Approach Validated in central data model and distributed models

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

25

Overview of DELTA

Frederic S Resnic MD MSc

and Susan Robbins

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

26

Overview of DELTA System

DELTA Open-Source Version 361

bull DELTA user interface with integrated DELTAlyticsanalytic engine utilizing MySQL as back-end data repository

bull Installation packages available on BitBucket

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

Overview of DELTA System

JUST TO MENTIONhellipSite Admin Setup

bull Advanced Featuresbull DELTAlytics statistical package interface setup ndash part of Installation

bull Site Organizations

bull Users PermissionsRoles (all Demo Users are Organization Admins)

bull Projects Groups

bull Source Data Connections and Table Relationships (Rely on your data managers)

bull JDBC databases allowed ndash MSSQL MySQL

bull Requires Database source file ndash even a Flat File must be imported to db

bull Alerts and Notifications

Note Topic detail not covered today

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

28

DELTA Study Life-cycle Stages

bull Model Builder ndash data field definitions and utilization characteristics

bull Study Configuration ndash analysis method parameters and data set extraction

bull Results ndash table and graph visualization

Overview of DELTA System

Note Topic detail not covered today

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

29

Requires Completed Data SourceTables and Table Relationships

bull Atomic Fields

bull Field Creation ndash virtual (primarysecondary)

bull Handling Missing

bull Data Acquisition (Staging) and Extraction (Flat Table)

bull Descriptive Statistics (StagingFlat Table)

Note Topic detail not covered today

Overview ndash Data Model Builder

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

30

Overview ndash Atomic and Virtual Fields

Model Fields and Descriptive Stats

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

31

Overview ndash Study Configuration

Requires Successful Model Processing

bull Model Selection

bull Identification of Study Dates Case and Sequencer Fields

bull Analysis Method Selection

bull Completion of Method Parameters

bull Study Submission and Analytic Processing

Note Topic detail not covered today

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

32

Overview ndashDELTA Methods

DELTA Statistical Methods

bull Logistic Regression

(including LR Make and LR Apply)

bull Propensity Analysis

(including PA Create PA Match and PA Outcomes)

bull Survival

(including standard and propensity score matched survival) Note Topic detail not covered today

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

33

Overview ndash Propensity Analysis Method

DELTA Propensity Analysis Method

bull Matched propensity parameters

bull Outcomes ndash single or multiple

bull Covariate selection for match formula

bull Match characteristics

bull Options - Alpha Spending and Automated Variable Selection

bull Method configuration divided as separate analysis steps

bull Create match formula ndash formula stored

bull Match using formulandash case set stored

bull Outcomes ndash process outcomes of existing match set

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

34

Overview - Study Results

Results Visualization

bull Dashboard

bull Method specific Charts and Tables

bull Export to CSV PDF PNG JPEG

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

35

Overview - Study Results

List of PA chart and table visualizations

bull Risk Adjusted ObservedExpected Chart (Periodic)

bull Event Rate Comparison Chart

bull Risk Adjusted ObservedExpected Chart (Area)

bull Event Rate Comparison Chart Data

bull Observed Expected Chart Data

bull Descriptive Statistics ndash Field stats

bull Covariate Matched Statistics ndash Field stats Prior After Unmatched

bull Logistic Regression Formula Data

bull Outcomes Data

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

36

CathPCI DELTA Pilot Project Example

Configuration - Susan Robbins

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

37

Steps to Configure DELTA Mynx VCD Pilot Analyses

1 Evaluate Protocol required fields including devices and outcomes

2 Evaluate Protocol method parameters

3 Prepare source database tables define DELTA connections

4 Select Atomic fields assign classkind and process Acquistion

5 Create virtual fields assign classkind

6 Review Descriptive Stats Impute and enter lsquoMissingrsquo

7 Process All fields

8 Create and Configure Studies run Analyses

CathPCI DELTA Pilot

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

bull Outcomes In-hospital significant vascular and bleeding

bull Primary Safety Any Vascular Complication ndash composite of access site bleeding requiring treatment large hematoma (gt6cm) retroperitoneal hemorrhage vascular comp requiring intervention

bull Secondary Safety Access site bleeding alone requirement for post-procedural blood transfusion

bull Pre-specified Analyses per written SAP

bull High risk populations Women diabetics agege70 years

bull Alternative risk adjustment Multivariate logistic risk adjustment

bull Falsification Hypothesis Thrombectomy catheter adverse events (anticipated no difference expected between devices)

bull Interim Reviews pre-specified (SAP) for review by oversight committee after 12 months and at study completion

CathPCI DELTA Pilot

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

39

Atomic and Virtual Fields ndash Outcome example

bull Any Vascular Complication defined with PCI Registry Fields

PostOtherVascComplication ndash 8035

PostBleedHematoma gt5-10cm gt 20cm ndash 8060

PostRetroBleed ndash 8070

PostBleedAccessSite ndash 8055

bull DELTA virtual field formula from Atomic Dichotomous fields

(PostOtherVascComp]=1) OR (PostBleedHematoma=1) OR (PostRetroBleed=1) OR (PostBleedAccessSite=1)

CathPCI DELTA Pilot

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

40

Partial List ndash Field FormulasField Description mysql_syntax

Calculated BMI = Weight(kg)(Height(m)Height (m)) (Weight(HeightHeight))

Is BMI gt=21 and lt25 IF(t_BM1gt=210 AND t_BM1lt250 10)

Is BMI gt=30 IF(t_BMIgt=300 1 0)

Is BMI lt 210 IF(t_BMIgt210 10)

Is Agegt 70 IF(Age gt=70 1 0)

CAD Presentation (5000) = STEM or Equivalent IF(CADPresentation_TEXT=STEMI or equivalent 1 0)

PCI Status 7020 IF(PCIStatus_TEXT=Emergency 10)

IIF Stemi OR NonSTEMI IF(t_lsPresentationNSTEMI= 1 OR t_lsPresentationSTEMI= 110)

1 Immediate PCI for STEM IF(PCindication=1 10)

MAX pre-procedure Lesion Severity Max(StenosisPriorTreat)

Minimal PreProcedure TIMI Flow Rate min(PreProcTIMI)

IIF (CIRCStenosisgt50 10) IF(CIRCStenosis gt=50 1 0)

IIF (MidLADDz= 1 or ProxLADDz= 110) IF(t_MidLAD_Dz= 1 OR t_HasProxladDz= 110)

IIF (NumVessei_Dzgt33NumVessei_Dz]) IF(f_NumVessel_Dz gt33f_NumVessei_Dz)

Is any thrombectomy device class = Aspiration IF(ICDeviceClass=AspirationTF)

Is any Vascular Closure Device Class= Active but non Mynx IF(ClosureDevClass=Active AND t_lsDEVFamily_Mynxlt=gt 110)

Is any Vascular Closure Device Class= Active IF(ClosureDevClass=ActiveT F)

Is any Closure Device Family Mynx IF(ClosureDevFamily=MynxTF)

Is any MedUsed = Aspirin (any) IF(MedUsed=AspirinTF)

Has Any ln-Stent Thrombosis IF(InThrombosis= 1TF)

Has any previously treated lesion IF(PrevTreatedlesion= 1TF)

Presence of any graft lesion IF(LesionGraft= 1 2 3TF)

MACE= 8000 PostPCI_MI8015 PostPCI_CVA 9000 CABG Death IF(CABG=1 OR PostCVA=1 OR PostMI=1 OR t_lsDeath=110)

CathPCI DELTA Pilot

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

CathPCI DELTA Pilot

bull Propensity Match Analysis 11 match of patients treated with Mynx as compared to patients treated with alternative active VCD within 6 months of Mynx case

bull Non-parsimonious propensity model (likelihood of receiving Mynx) fixed caliper width=001 greedy match

bull Quarterly analysis with adjustment for multiple comparisons (OrsquoBrien-Fleming) for each endpoint

bull Alert if 95 CI of differences between proportions (Wilson method) do not cross zero

bull Missing Data Simple imputation (unknown ldquomissingnessrdquo in advance)

bull Post-Hoc Analyses On basis of interim and final reviews Scientific Committee recommended sensitivity analyses and analysis of independent validation dataset (10113-92015)

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

42

Study Method Parameters

CathPCI DELTA Pilot

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

43

Study Filters and Subgroups

CathPCI DELTA Pilot

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

44

Study Results Dashboard

CathPCI DELTA Pilot

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

45Source Resnic FS et al

CathPCI DELTA Pilot

Study Result Chart

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

46

Study Subgroup Results Dashboard

CathPCI DELTA Pilot

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

47

Study Subgroup Results Chart

CathPCI DELTA Pilot

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

48

BREAK

(10 minutes)

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

49

DELTA User Support

Susan Robbins

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

50

DELTA Resources

bull Live Demo Site

httpsCERIdemolaheyorgDELTA3

bull Download site

httpsbitbucketorgCERI-Laheydelta3downloads

bull On-line documentation

User Manual httpsceri-laheyatlassiannetwikispacesDUMoverview

Statistical Methods httpsceri-laheyatlassiannetwikispacesDSMSoverview

Questions

51

DELTA Interactive DEMO

and Questions

Fred Resnic Henry Ssemaganda

Susan Robbins

httpsCERIdemolaheyorgDELTA3

Learning Objectives

bull Understand basic fundamentals of DELTA

bull Menus features and functions

bull Understand Data Model

bull Atomic and virtual fields missing substitutions data acquisition and extraction descriptive statistics and flat table

bull Understand Study Configuration

bull Basic Propensity Analysis parameters

bull Understand Results Visualization

bull Dashboard and ChartsTables

53

1 Model review

2 Atomic fields assign classkind

3 Create virtual fields assign classkind

4 Create and Configure Study

5 Review Demo Results

You can try cloning a model editing it and processinghellip

And try cloning a study assign new model change a parameter and submithellipwersquoll see how you do

DELTA Demo - Topics

54

DELTA Demo ndash Login

bull Login with UsernamePassword on card

bull Brief intro to Projects Feedback Stat Program User Dropdown buttons

bull DELTA Menu

Select ldquoData Modelsrdquo

55

DELTA Demo ndash Models

Sample Model Grid

bull Review Step 1 Manage (Add Edit Clone Delete)

bull Step 2 Build Model Structure

Select (ldquoDemo Modelrdquo) and click Step 2

bull Model Builder

bull Review Step 1A 1B 1C (SourceAtomicRelationships)

56

DELTA Demo ndash Model Source Data

bull Step 1A ndash Select well structured dataset

Normalized with Primary Key

Can manage 1many with Step 1C lsquoRelationshipsrsquo

Sample Data is a Flat Table in database

bull Step 1B ndash Atomic Field Selection

(click +) to display fields check mark indicates those added to the model

Note Topic detail not covered today

57

DELTA Demo ndash Model Field Grid

bull Atomic ndash from Source(Type read from Source ndash will prompt if not

valid in DELTA)

bull Virtual ndash Primary are based on Source Column

bull Virtual ndash Secondary are based on Primary Virtual (stacked)

bull Missing ndash enter formulavalue on Atomic AND flag in ldquoMissingrdquo column

(Verify formulas before processing)

bull Field Characteristics

bull ClassKind relate to USE of data by DELTA

58

DELTA Demo ndash Model Data

Sample Model Grid

bull Review Step 2 and 3 Acquisition Virtual Fields Extraction

ldquoSTATUS Processing of Descriptive Stats finishedrdquo

Select Step 3 Data Extraction gt All Statistics

bull Review Statistics table

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

53

1 Model review

2 Atomic fields assign classkind

3 Create virtual fields assign classkind

4 Create and Configure Study

5 Review Demo Results

You can try cloning a model editing it and processinghellip

And try cloning a study assign new model change a parameter and submithellipwersquoll see how you do

DELTA Demo - Topics

54

DELTA Demo ndash Login

bull Login with UsernamePassword on card

bull Brief intro to Projects Feedback Stat Program User Dropdown buttons

bull DELTA Menu

Select ldquoData Modelsrdquo

55

DELTA Demo ndash Models

Sample Model Grid

bull Review Step 1 Manage (Add Edit Clone Delete)

bull Step 2 Build Model Structure

Select (ldquoDemo Modelrdquo) and click Step 2

bull Model Builder

bull Review Step 1A 1B 1C (SourceAtomicRelationships)

56

DELTA Demo ndash Model Source Data

bull Step 1A ndash Select well structured dataset

Normalized with Primary Key

Can manage 1many with Step 1C lsquoRelationshipsrsquo

Sample Data is a Flat Table in database

bull Step 1B ndash Atomic Field Selection

(click +) to display fields check mark indicates those added to the model

Note Topic detail not covered today

57

DELTA Demo ndash Model Field Grid

bull Atomic ndash from Source(Type read from Source ndash will prompt if not

valid in DELTA)

bull Virtual ndash Primary are based on Source Column

bull Virtual ndash Secondary are based on Primary Virtual (stacked)

bull Missing ndash enter formulavalue on Atomic AND flag in ldquoMissingrdquo column

(Verify formulas before processing)

bull Field Characteristics

bull ClassKind relate to USE of data by DELTA

58

DELTA Demo ndash Model Data

Sample Model Grid

bull Review Step 2 and 3 Acquisition Virtual Fields Extraction

ldquoSTATUS Processing of Descriptive Stats finishedrdquo

Select Step 3 Data Extraction gt All Statistics

bull Review Statistics table

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

54

DELTA Demo ndash Login

bull Login with UsernamePassword on card

bull Brief intro to Projects Feedback Stat Program User Dropdown buttons

bull DELTA Menu

Select ldquoData Modelsrdquo

55

DELTA Demo ndash Models

Sample Model Grid

bull Review Step 1 Manage (Add Edit Clone Delete)

bull Step 2 Build Model Structure

Select (ldquoDemo Modelrdquo) and click Step 2

bull Model Builder

bull Review Step 1A 1B 1C (SourceAtomicRelationships)

56

DELTA Demo ndash Model Source Data

bull Step 1A ndash Select well structured dataset

Normalized with Primary Key

Can manage 1many with Step 1C lsquoRelationshipsrsquo

Sample Data is a Flat Table in database

bull Step 1B ndash Atomic Field Selection

(click +) to display fields check mark indicates those added to the model

Note Topic detail not covered today

57

DELTA Demo ndash Model Field Grid

bull Atomic ndash from Source(Type read from Source ndash will prompt if not

valid in DELTA)

bull Virtual ndash Primary are based on Source Column

bull Virtual ndash Secondary are based on Primary Virtual (stacked)

bull Missing ndash enter formulavalue on Atomic AND flag in ldquoMissingrdquo column

(Verify formulas before processing)

bull Field Characteristics

bull ClassKind relate to USE of data by DELTA

58

DELTA Demo ndash Model Data

Sample Model Grid

bull Review Step 2 and 3 Acquisition Virtual Fields Extraction

ldquoSTATUS Processing of Descriptive Stats finishedrdquo

Select Step 3 Data Extraction gt All Statistics

bull Review Statistics table

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

55

DELTA Demo ndash Models

Sample Model Grid

bull Review Step 1 Manage (Add Edit Clone Delete)

bull Step 2 Build Model Structure

Select (ldquoDemo Modelrdquo) and click Step 2

bull Model Builder

bull Review Step 1A 1B 1C (SourceAtomicRelationships)

56

DELTA Demo ndash Model Source Data

bull Step 1A ndash Select well structured dataset

Normalized with Primary Key

Can manage 1many with Step 1C lsquoRelationshipsrsquo

Sample Data is a Flat Table in database

bull Step 1B ndash Atomic Field Selection

(click +) to display fields check mark indicates those added to the model

Note Topic detail not covered today

57

DELTA Demo ndash Model Field Grid

bull Atomic ndash from Source(Type read from Source ndash will prompt if not

valid in DELTA)

bull Virtual ndash Primary are based on Source Column

bull Virtual ndash Secondary are based on Primary Virtual (stacked)

bull Missing ndash enter formulavalue on Atomic AND flag in ldquoMissingrdquo column

(Verify formulas before processing)

bull Field Characteristics

bull ClassKind relate to USE of data by DELTA

58

DELTA Demo ndash Model Data

Sample Model Grid

bull Review Step 2 and 3 Acquisition Virtual Fields Extraction

ldquoSTATUS Processing of Descriptive Stats finishedrdquo

Select Step 3 Data Extraction gt All Statistics

bull Review Statistics table

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

56

DELTA Demo ndash Model Source Data

bull Step 1A ndash Select well structured dataset

Normalized with Primary Key

Can manage 1many with Step 1C lsquoRelationshipsrsquo

Sample Data is a Flat Table in database

bull Step 1B ndash Atomic Field Selection

(click +) to display fields check mark indicates those added to the model

Note Topic detail not covered today

57

DELTA Demo ndash Model Field Grid

bull Atomic ndash from Source(Type read from Source ndash will prompt if not

valid in DELTA)

bull Virtual ndash Primary are based on Source Column

bull Virtual ndash Secondary are based on Primary Virtual (stacked)

bull Missing ndash enter formulavalue on Atomic AND flag in ldquoMissingrdquo column

(Verify formulas before processing)

bull Field Characteristics

bull ClassKind relate to USE of data by DELTA

58

DELTA Demo ndash Model Data

Sample Model Grid

bull Review Step 2 and 3 Acquisition Virtual Fields Extraction

ldquoSTATUS Processing of Descriptive Stats finishedrdquo

Select Step 3 Data Extraction gt All Statistics

bull Review Statistics table

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

57

DELTA Demo ndash Model Field Grid

bull Atomic ndash from Source(Type read from Source ndash will prompt if not

valid in DELTA)

bull Virtual ndash Primary are based on Source Column

bull Virtual ndash Secondary are based on Primary Virtual (stacked)

bull Missing ndash enter formulavalue on Atomic AND flag in ldquoMissingrdquo column

(Verify formulas before processing)

bull Field Characteristics

bull ClassKind relate to USE of data by DELTA

58

DELTA Demo ndash Model Data

Sample Model Grid

bull Review Step 2 and 3 Acquisition Virtual Fields Extraction

ldquoSTATUS Processing of Descriptive Stats finishedrdquo

Select Step 3 Data Extraction gt All Statistics

bull Review Statistics table

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

58

DELTA Demo ndash Model Data

Sample Model Grid

bull Review Step 2 and 3 Acquisition Virtual Fields Extraction

ldquoSTATUS Processing of Descriptive Stats finishedrdquo

Select Step 3 Data Extraction gt All Statistics

bull Review Statistics table

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

59

DELTA Demo ndash Study Configuration

bull Review Step 1 Add Study

bull Review Step 2 Edit Study

bull Review Study Grid fields

Select (ldquoPA Demo Study 2rdquo) and click ldquoStep 2rdquo

Then ldquoEdit parametersrdquo

Select ldquoStudiesrdquo

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

60

Note Can schedule to rerun without intervention

bull Study Grid Fields

DELTA Demo ndash Study Configuration

Parameters

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

61

DELTA Demo ndash Study Configuration

bull Sample Study PA Method Parameters

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

62

DELTA Demo ndash Study Results

bull Sample Result (PA Demo Study 2 P1)

bull Review lsquoResult Dashboardrsquo

bull Review lsquoResult Chartsrsquo

Select (ldquoPA Demo Study 2 P1rdquo) and click ldquoDashboardrdquo andor ldquoResult Chartsrdquo

Select ldquoResultsrdquo

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

Thank You

Lahey Clinic Medical Center

Susan Robbins BSHenry Ssemaganda MD MPH

Marek Mizeracki MSc

Vanderbilt University VAMC

Michael Matheny MD MSc MPHDax Westerman MSc

For more information please contact Frederic S Resnic MD MScLahey Hospital and Medical Center41 Mall RoadBurlington MA 01805781-744-2778

fredericresniclaheyorg

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

64

DELTA Administration Overview

bull Organizations

bull Users

bull Permissions and Roles

bull ProjectsGroups

bull Database connections

bull Statistical Method Interface

65

DELTA Administration

65

DELTA Administration