Upload
azize
View
62
Download
4
Tags:
Embed Size (px)
DESCRIPTION
June 2012 Update June 14, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench. With Funding Support provided by National Institute of Standards and Technology. Agenda for Today. Progress on folder organization, naming conventions, and ISA roll-ups (Gary) - PowerPoint PPT Presentation
Citation preview
June 2012 UpdateJune 14, 2012
Andrew J. Buckler, MSPrincipal Investigator,
QI-Bench
WITH FUNDING SUPPORT
PROVIDED BY NATIONAL
INSTITUTE OF STANDARDS AND
TECHNOLOGY
Agenda for Today• Progress on folder organization, naming
conventions, and ISA roll-ups (Gary)• Demonstration of multiple time point batch
analysis capability (Mike)• Progress on automated statistical analysis (Kjell)
22222222
First Generation: 3A Challenge• Created newly named data set upon
request so that all dicom files had an extension of dcm.
• These files follow our 1st generation standard for naming file layout and results.
• Meeting program needs; but from system point of view, essentially no automation.
33
Infrastructure progress for next generation:
• Created naming conventions and file formats for roll up files.
• Created scripts for generating roll up files for location and change data.
44
Example: Reference Data Sets for QIBA compliance
• Our most fully 2nd generation compliant reference data sets.
• Totality of data from DICOM to analysis is included.
• Study data, in ISA-TAB (like) structure is available for analysis.
55
MULTI-TIMEPOINT BATCH ANALYSIS DEMO
6
Demonstration of automated analysis using R controlled by Iterate.
7
Iterate workflow for running an R script
777777
Test/Re-test Study• Test/Re-test is one of the building-block study designs for
understanding a measurement system• Helps us to understand the variation we see in the system
under the “no change” condition
• Fundamental components of the analysis– Scatterplot of test values versus retest values– Mean versus difference of test/re-test values (i.e. Bland-Altman)– Intraclass correlation coefficient (ICC):
• variation due to the subjects (class) relative to the unexplainable variation.– Minimum detectable change (MDC):
• minimum change considered to be statistically significant
888888
Test/Re-test Example• CT Volumetry data
– 32 subjects in “coffee break” design• 2 subjects excluded as uncomputable
– Analysis of volume calculations
• Results can be generated for any study that follows the Test/Re-test format
999999
101010101010
Volume Difference
value1 value22 7.554111 8.0970284 771.499954 654.2557966 22099.090961 22461.1688568 45558.404866 45411.06484310 8.713604 7.79649012 7824.517733 8166.32472514 31671.844931 29196.96870316 12.363709 13.83378518 19.949875 19.91985420 11.560880 6.16771422 37485.766057 32506.89271224 2.451236 2.45123926 33261.732159 12771.46856028 7720.802020 11.45661330 18012.668569 18377.25751232 8725.320453 8880.53900034 7.780063 8.64938936 16.691373 56622.55576738 8.241816 6070.07127240 16.342152 10.60224442 13.167784 12.48320944 6593.866812 6193.35183746 6.043523 6724.99050748 5.826424 4.81851450 6683.346163 7479.94868052 15.816401 18.15936354 7432.544980 7393.72354356 6.786277 6.78628158 26144.753408 31253.57102360 717.220963 745.647667
111111111111
121212121212
131313131313
Volume Summary
Volume Calculation N Mean Standard Deviation Min Median Max1 ProportionalChange 30 3.51 38.66 -99.70 0 99.942 VolumeDifference 30 1339.28 11401.51 -20490.26 0 56605.86
141414141414
Intraclass Correlation
Intraclass Correlation for Test-retest: 0.68 Minimum Detectable Change : 20835.38
Other Building-Block Studies(In Process)
• Inter- and intra-reader variability
• Studies of bias and variation when truth is known via phantoms
• Studies for understanding the variation due to standard, uncontrolled factors (e.g. site, machine, etc.)
151515151515
Formal Deployment: QIBA/RIC instance• We are well on the way to having a
“third party” installation of Execute.• These reference data are largely in
our 2nd generation format for naming and layout.
• Whereas CT volumetry has been (and continues to be) the focus of the test bed, the first use of the RSNA instance is DCE-MRI.
1616
1717
Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing
standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them
without huge costs– Public wants to trust the decisions that they contribute to
• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data
• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders
1818
Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased
statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of
generalizability. • We make formal specification accessible to diverse groups of experts that are
not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into
representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or
requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of
collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.
1919
QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)
• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik)
• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu)
• Collaborators / Colleagues / Idea Contributors– Georgetown (Baris Suzek)– FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha)– Northwestern (Pat Mongkolwat)– UCLA (Grace Kim)– VUmc (Otto Hoekstra)
• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, …
• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)
2020