Monthly Program UpdateJanuary 12, 2012
Andrew J. Buckler, MSPrincipal Investigator
WITH FUNDING SUPPORT
PROVIDED BY NATIONAL
INSTITUTE OF STANDARDS AND
TECHNOLOGY
Agenda• Monthly snapshot in Jira
– (including status of installation at NIST)• QIBA 3A project snapshot• Theoretical development• Architecture and SW stack
22
3
BSD-2 licenseDomain is www.qi-bench.org.
Landing page provides • Access to
prototypes, • Repositories for
download and development,
• Acknowledgements,
• Jira issue tracking, and
• Documentation
3
Go to the site and workWith the apps and
Jira
QIBA 3A PROJECT SNAPSHOT(recalling that this is a testbed for us)
444
Basic structure of the challenges
5
Pilot
Pivotal
Investigation 1
Train Test Pilot
Pivotal
Investigation
Train Test Pilot
Pivotal
Investigation
Train Test Pilot
Pivotal
Investigation n
Train Test
P r i m a r y
S e c o n d a r y• Defined set of data• Defined challenge• Defined test set policy
555
First one: • Presently in pilot phase,
• Using StudyDescription method• Used batch scripting with reference method to
aid data curation• 10-12 participants (about 20 QI-Bench users)• First participant data received• Analysis plan using N-way ANOVA in R started
• Pivotal phase starting with batch assisted curation• Will be transitioning to database schema for
metadata (gradually away from spreadsheet)
Pooled Bias
Pooled Variability
Variability across shapesVariability across densities
Variability across slice thicknesses
0
5
10
Method AMethod BMethod CMethod DMethod EMethod FGroup
666
1. Relative performance is computed according to descriptive statistics
2. We determine a group value for each of the descriptive statistics, e.g., as the mean plus 1 stdev (or as wide as we think wise).
3. Results presented using radar plots
Bias Variability Repeatabilitycross-x
reproducibilitycross-y
reproducibilityNew Method 7.00 3.00 10.00 11.00 12.00Group 7.16 6.64 6.96 7.81 9.00
Bias
Variability
Repeatabilitycross-x reproducibility
cross-y reproducibility
0.00
10.00
20.00
New MethodGroup
In this example, the new proposed method does not perform well enough to be considered a valid method since it falls outside the group values.
777
In this example, the new proposed method is seen to perform within group values and may even help pave the way for an improved claim.
Bias VariabilityRepeatability
cross-x reproducibility
cross-y reproducibility
New Method 3.00 3.00 4.00 6.00 8.00Group 7.16 6.64 6.96 7.81 9.00
Bias
Variability
Repeatabilitycross-x reproducibility
cross-y reproducibility
0.00
5.00
10.00
New MethodGroup
888
THEORETICAL DEVELOPMENTprogress re: utilization of logical and statistical inference at each of two levels, technical performance of assay methods, and qualification of biomarker in specific clinical context
999
Another way to look at what needs to happen
10101010
Formulate
Statistical Analysis Results (Relation
strength)
Annotation and Image Markup,
Non-imaging Clinical Data
Primary Data: Images and other
Raw Data
Reference Data SetsQIBO
Specify
RDF Triple Store
CT Volumetry CT
obtained_by
Tumor growth
measure_of
TherapeuticEfficacy
used_for
Analyze
Y=β0..n+β1(QIB)+β2T+ eij
Execute
Feedbac k
Feed
bac
k
Specify: Establish a logical specification and setup terms for mathematical analysis
1111
• Functionality:• Establish means to semantically labeling imaging
biomarker data with emphasis on representing both the clinical context in which an imaging biomarker is used as well as the specifics of the imaging protocol used to acquire the images.
• Set up the logistic regression model:• Precisely specify dependant variable• Account for covariates• Enumerate independent variables and
error terms (sources of variability)• Establish database for collection of terms.
• Method:• Provide GUI to traverse the QIBO concepts
according to their relationships and create statements represented as RDF triples and stored in an RDF store.
• Each set of RDF triples will be stored as a “profile.”
• Relationship strength initialized based on prior estimates (if available) QIBO
Specify
RDF Triple Store
CT Volu
metryCT
obtained_by
Tumor growth
measure_of
Therapeutic
Efficacy
used_for
Ontologies supporting Specify
1212
• Extend the QIBO to link to existing established ontologies
1. leverage BFO upper ontology to align different ontologies
2. convert portions of BRIDG and LSDAM to ontology models in OWL
• Automated conversion would done in two steps:
1. convert current Sparx Enterprise Architect XMI EMF UML format
2. export resulting EMF UML into a RDF/OWL representation using TopBraid Composer
Formulate: advanced query framework made possible by Specify
1313
• allow users to select the profiles (or set of RDF triplets) created in Specify, execute a query and retrieve the results in various forms.
• assemble/transform the set of RDF triples to SPARQL queries:
1. form an uninterrupted chain linking the instance of the input class from the ontology to the desired output class
2. formulate/invoke necessary SPARQL queries against the web services deployed in SADI framework.
• interface with the query engine and will have offline (asynchronous) query execution capability.
• results to be exportable as serialized objects (RDF/XML and CSV)
Formulate
Statistical
Analysis
Results (Relati
on strengt
h)Annotation and
Image Markup, Non-imagin
g Clinical
DataPrimary Data: Images
and other Raw Data
Reference Data Sets
RDF Triple Store
CT Volu
metryCT
obtained_by
Tumor growth
measure_of
Therapeutic
Efficacy
used_for
Data Services supporting Formulate
1414
• wrap existing data services such as NBIA, caArray, caTissue, AIM and PODS using Semantic Automated Discovery and Integration (SADI)• this is enabled by metadata available
through the UML representations of the models exposed by these services and CDE annotations available for them through caDSR.
• describe service I/O semantically using the extended version of QIBO
• service registry of SADI will help the automated composition of computer-interpretable queries by the query engine. • example: “there is a service that
returns Biological Subjects that has undergone certain Biological Interventions”
Analyze: Use annotation and image markup to support statistical inference
1515
• Support Clinical Performance assessment (i.e., in addition to current Technical Performance) • Outcome studies• Integrated genomic/proteomic
correlation studies• Group studies for biomarker qualification
• (set up a basic multiple regression analysis, e.g.) Intent to treat analysis of the primary outcome via covariance model of the general form (QIBt)=β0..n+β1(QIB0)+β2T+ eij where QIBt and QIB0 are the QIB at a time after treatment and at randomization respectively, T is a treatment group indicator, and β0..n, β1, and β2 are model parameters. β2 represents the effect of treatment and its estimate is the difference between group means on the log scale, after adjustment for any imbalance between the groups in log QIB. The error terms in the model, eij, are assumed mutually independent and normally distributed. Depending on the nature of the QIB, the log transformation may be used instead of the direct value. Likewise calculations may be performed using z scores with corresponding conversion with raw values.
Quanti tative Ima ging Specification Lan guage
Batch Ana lysis Service
Reference Data Set Man ag er
UPICT Protocols, QIBA Profiles, literature papers and other sources
QIBO-
BatchMakeScripts
Reference Data Sets, Annotations, and Analysis Results
(red edgesrepresent
biostatisticalgeneralizability)
Source of clinical study results
Clinical Body of Evidence (formatted to enable SDTM and/or other standardized registrations
4. Output
3. Batch analysis scripts
UPICT Protocols, QIBA Profiles, entered with
Ruby on Rails web service
QIBO
Examples of output at biomarker (above the assay level)
1616
From Jack 1999. Note: W-score is the relative score of the measured HC volume corrected for intracranial volume and compared to age and sex adjusted normals.
Hypothetical model of dynamic biomarkers of the Alzheimer's pathological cascadeJack et al., 2010
161616
To inform thresholding
To substantiate surrogacy (or its weaker form of “activity”)
ARCHITECTURE AND SW STACKSo what is a cohesive architecture that maximizes leverage of best thinking, existing touchpoints, and stays current over time?
171717
STDM standard of CDISC into repositories like FDA’s
Janus.
MVT portion of AVT, re-useable
library of R scripts.
MIDAS, BatchMake, Condor Grid;
built using Zend on
PHP.
caB2B, NBIA,
PODS data elements, DICOM
query tools.
QIBO, AIM,RadLex/ Snomed/ NCIt; built
using Ruby on Rails.
•Specify context for use and assay methods.
•Use consensus terms in doing so.
Specify
•Assemble applicable reference data sets.
•Include both imaging and non-imaging clinical data.
Formulate •Compose and iterate
batch analyses on reference data.
•Accumulate quantitative read-outs for analysis.
Execute
•Characterize the method relative to intended use.
•Apply the existing tools and/or extend them.
Analyze •Compile evidence for regulatory filings.
•Use standards in transfer to regulatory agencies.
Package
1818
MVT: Reasonable framework, but many gaps
1919
There are multiple possibilities to deploy it as a web application, some of which we’ve considered:
1. Re-implement the existing implementation to use GWT in place of Swing, inclusive of both the XIPHost as well as MVT components, retaining the WG23 concept.
2. Re-implement only those parts necessary to perform the needed MVT functions using GWT with enough data handling to do so but without doing everything necessary to retain the full XIPHost capability.
3. Leverage the GUI design concept but otherwise implement without starting from the Swing code.
In all cases, there is the secondary design alternative of introducing a RESTful web service layer explicitly or not. (By the way, just for fun, I performed a conversion of the current Swing code to Ajax using AjaxSwing. I got most of AVT working over the web with minimal effort, but this isn’t a serious alternative because AjaxSwing has a license fee. I did it because I wanted to see how easy such a path would be. It’s an interesting capability! But irrelevant in the end.)
Pros: optimized for DICOM, works with workstationsCons: hard to create web apps, not optimized for semantic web
2020
HW
XIPApplication
InventorApplication Modules
WG 23 System Services PLUG
WG 23 System Services SOCKET
GRIDCLIENT
SERVICES
DICOMSERVICES(DCMTK)
OTHERSERVICES
VTK ITK AIMTK other
OS
NCIA
XIPIDE
RadLex
AIM
NCI
ProtégéEVS
XIP
MIDDLEWARE
DICOM
DICOM Services
IVI MiddlewarecaGrid
CaBIG
caDSR, EVS, RadLex, AIM ontology, etc
Client accessService access
Grid Data Service
Grid Analytical Service AIM Data
Service
XIP App
ServiceHost
WG23
DICOM Image
Sources
2020
Alternative architectural form…
21
SW Stack• J2SE (J2EE compliant)• MySQL• caGrid• Globus• Application:
• JBoss• caCore 212121
With pros and cons “opposite” that of the XIP based architecture
Functionality view annotated with architecture
2222
HW
XIPApplication
InventorApplication Modules
WG 23 System Services PLUG
WG 23 System Services SOCKET
GRIDCLIENT
SERVICES
DICOMSERVICES(DCMTK)
OTHERSERVICES
VTK ITK AIMTK other
OS
NCIA
XIPIDE
RadLex
AIM
NCI
ProtégéEVS
XIP
MIDDLEWARE
DICOM
DICOM Services
IVI MiddlewarecaGrid
CaBIG
caDSR, EVS, RadLex, AIM ontology, etc
Client accessService access
Grid Data Servi
ce
Grid Analytical Service
AIM
Data
Service
XIP App
Service
Host
WG23
DICOM
Image
Sources
2222
MIDAS Core
Apache File SystemPostGreSQL
Publication DB
MIDAS Data Server
MIDAS e-journal
MIDAS Compute Server
MIDAS Visualization
MIDAS Client
MIDAS C++ API
MIDAS Web API
When annotation and markup has already been done
Reference data
setsAnnotati
on
and marku
p
AIM-enabled (e.g., ClearCanvas) workstation
RIS worklist items
DICOM Q/R
First step to rationalizing architecture: mash them together and see what falls out
23232323
NCIA
RadLex
AIM
NCI
XIP
MIDDLEWARE
DICOM
DICOM Services
IVI MiddlewareSADI framework (e.g., wrapped caGrid)
CaBIG
caDSR, EVS, RadLex, AIM ontology, etc
Client accessService access
Grid Data Service
Grid Analytical Service AIM Data
Service
HW
XIPApplication
InventorApplication Modules
WG 23 System Services PLUG
WG 23 System Services SOCKET
GRIDCLIENT
SERVICES
DICOMSERVICES(DCMTK)
OTHERSERVICES
VTK ITK AIMTK other
OS
XIPIDE
ProtégéEVS
XIP App
ServiceHost
WG23
DICOM Image
Sources
This is an ongoing discussion. More to come!
2424
Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing
standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them
without huge costs– Public wants to trust the decisions that they contribute to
• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data
• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders
2525
Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased
statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of
generalizability. • We make formal specification accessible to diverse groups of experts that are
not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into
representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or
requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of
collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.
2626
QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)
• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik, Tiffany Ting Liu)
• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, Guillaume Radde)
• Collaborators / Colleagues / Idea Contributors– FDA (Nick Petrick, Marios Gavrielides)– UCLA (Grace Kim)– UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad)– VUmc (Otto Hoekstra)– Northwestern (Pat Mongkolwat)– Georgetown (Baris Suzek)
• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens (Maria Athelogou), Claron Technologies (Ingmar Bitter)
• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)
2727