Upload
stasia
View
45
Download
0
Embed Size (px)
DESCRIPTION
QIBA CT Volumetrics Group 1B: (Patient Image Datasets). Charge. To agree on questions to be answered by these Reference Datasets To identify requirements for those datasets, based on questions to be answered To identify existing datasets that can be leveraged to provide desired datasets. - PowerPoint PPT Presentation
Citation preview
QIBA CT VolumetricsQIBA CT Volumetrics Group 1B: Group 1B:
(Patient Image Datasets) (Patient Image Datasets)
ChargeCharge
1.1. To agree on questions to be answered To agree on questions to be answered by these Reference Datasetsby these Reference Datasets
2.2. To identify requirements for those To identify requirements for those datasets, based on questions to be datasets, based on questions to be answeredanswered
3.3. To identify existing datasets that can be To identify existing datasets that can be leveraged to provide desired datasetsleveraged to provide desired datasets
Questions (overview)Questions (overview)1.1. What level of What level of accuracy and precisionaccuracy and precision can be achieved can be achieved
in measuring tumor volumes in patient datasets?in measuring tumor volumes in patient datasets?2.2. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can
be achieved when measuring tumors in be achieved when measuring tumors in phantomphantom datasets?datasets?
3.3. What is the What is the minimum detectable level of changeminimum detectable level of change that that can be achieved when measuring tumors in can be achieved when measuring tumors in patient patient datasets under a “No Change” conditiondatasets under a “No Change” condition??
4.4. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can be achieved in measuring tumors in be achieved in measuring tumors in patient datasets patient datasets with “Unknown Change” conditionwith “Unknown Change” condition??
5.5. What is the effect of slice thickness on estimating What is the effect of slice thickness on estimating changechange in tumors using in tumors using patientpatient datasets? datasets?
Initially Agreed to First Pursue:Initially Agreed to First Pursue:1.1. What level of What level of accuracy and precisionaccuracy and precision can be achieved can be achieved
in measuring tumor volumes in patient datasets?in measuring tumor volumes in patient datasets?2.2. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can
be achieved when measuring tumors in be achieved when measuring tumors in phantomphantom datasets?datasets?
3.3. What is the What is the minimum detectable level of changeminimum detectable level of change that that can be achieved when measuring tumors in can be achieved when measuring tumors in patient patient datasets under a “No Change” conditiondatasets under a “No Change” condition??
4.4. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can be achieved in measuring tumors in be achieved in measuring tumors in patient datasets patient datasets with “Unknown Change” conditionwith “Unknown Change” condition??
5.5. What is the effect of slice thickness on estimating What is the effect of slice thickness on estimating changechange in tumors using in tumors using patientpatient datasets? datasets?
Next StepsNext Steps
Define requirements and Design Define requirements and Design Experiments for these two questionsExperiments for these two questions
Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)
in measuring tumor volumes in measuring tumor volumes 1.1. Specific AimsSpecific Aims
(a)(a) Investigate both bias and variance of both readers and Investigate both bias and variance of both readers and algorithm-assisted readers in measuring volumes, algorithm-assisted readers in measuring volumes, diameters and bi-directional diameters of lesionsdiameters and bi-directional diameters of lesions
(b)(b) Investigate inter-observer variability in each taskInvestigate inter-observer variability in each task(c)(c) Investigate Intra-observer variability in each task ???Investigate Intra-observer variability in each task ???(NOTE: here observer should be interpreted broadly – as (NOTE: here observer should be interpreted broadly – as
reader measuring manually for diameters as well as reader measuring manually for diameters as well as algorithm-assisted reader measuring contours).algorithm-assisted reader measuring contours).
Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)
in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS LIDC datasetLIDC dataset lesions with “known size” will be used (size lesions with “known size” will be used (size
based on contours from 4 LIDC readers)based on contours from 4 LIDC readers) Each lesion will have boundariesEach lesion will have boundaries Only a single time point is needed (no followup Only a single time point is needed (no followup
needed, no diagnosis needed)needed, no diagnosis needed) Need to identify which nodules to use – Need to identify which nodules to use –
criteria?criteria?
Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)
in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS Like NIST Biochange 2008, lesions are Like NIST Biochange 2008, lesions are
identified and coordinates provided to readers.identified and coordinates provided to readers. Reader tasksReader tasks
Manually mark 2 Diameters (w/o LIDC marks)Manually mark 2 Diameters (w/o LIDC marks)• Longest diameter and perpendicular diameterLongest diameter and perpendicular diameter
Also semiautomated contour of lesionAlso semiautomated contour of lesion• From contour, determine volume, longest diameter and From contour, determine volume, longest diameter and
diameter perpendicular to longest diameterdiameter perpendicular to longest diameter Have readers perform some cases > 1 (intra reader Have readers perform some cases > 1 (intra reader
variation)variation)
Data CollectedData CollectedNodule #Nodule # coordscoords LIDC LIDC
volvolLIDC LIDC Longest Longest DiamDiam
LIDC LIDC Perp Perp DiamDiam
R1 R1 volvol
R1 R1 LD LD from from contcontourour
R1 R1 PD PD from from contocontourur
R1 R1 LDLD
R1 R1 PDPD
R2 R2 volvol
Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)
in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS AnalysesAnalyses
LIDC would be considered “truth”LIDC would be considered “truth” Estimate bias of each readerEstimate bias of each reader
• VolumeVolume• Diam (manual and Assisted)Diam (manual and Assisted)• Product of diameters (LD x PD) (Manual and assisted) Product of diameters (LD x PD) (Manual and assisted)
Estimate inter-reader variabilityEstimate inter-reader variability• Reader vs. “gold standard”Reader vs. “gold standard”• Reader vs. reader Reader vs. reader
Intra-reader variabilityIntra-reader variability
Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)
in measuring tumor volumes in measuring tumor volumes 1.1. QuestionsQuestions
1.1. How many cases?How many cases?2.2. How many readers?How many readers?3.3. Case composition (all spherical? Some Case composition (all spherical? Some
spiculated? Range of sizes?)spiculated? Range of sizes?)4.4. Enough of each subgroup to perform a stat. Enough of each subgroup to perform a stat.
sign analysis?sign analysis?1.1. (reader bias on spherical nodules is XX; (reader bias on spherical nodules is XX; 2.2. (reader bias on spiculated nodules is YY)(reader bias on spiculated nodules is YY)
5.5. Do we want to do both size and shape Do we want to do both size and shape subgroup analyses?subgroup analyses?
QuestionsQuestions
Question 2 – What is the min detectable level of Question 2 – What is the min detectable level of change in patient datasets under a “No Change” change in patient datasets under a “No Change”
condition?condition?
1.1. Specific AimsSpecific Aims(a)(a) For patient datasets acquired over a very short time interval For patient datasets acquired over a very short time interval
(presumably the “no change” condition”) investigate variance of (presumably the “no change” condition”) investigate variance of both readers and algorithm-assisted readers in measuring both readers and algorithm-assisted readers in measuring changechange in volume, diameter and bi-directional diameters of lesions (here, in volume, diameter and bi-directional diameters of lesions (here, the expected value of the the expected value of the changechange should be zero) should be zero)
(b)(b) Investigate several change metrics such as:Investigate several change metrics such as:(a)(a) Absolute value of changeAbsolute value of change(b)(b) fractional change in volume/diameterfractional change in volume/diameter(c)(c) Categorical variables (progression, regression, etc.)??Categorical variables (progression, regression, etc.)??
(c)(c) Investigate inter-observer variability in each taskInvestigate inter-observer variability in each task(d)(d) Investigate Intra-observer variability in each task ???Investigate Intra-observer variability in each task ???(NOTE: again, observer should be interpreted broadly – as reader (NOTE: again, observer should be interpreted broadly – as reader
measuring manually for diameters as well as algorithm-assisted measuring manually for diameters as well as algorithm-assisted reader measuring contours).reader measuring contours).
Existing ResourcesExisting Resources
RIDER – MSK Coffee Break Experiment RIDER – MSK Coffee Break Experiment (No Change Condition)(No Change Condition)
32 NSCLC patients32 NSCLC patients Imaged twice on the same scanner w/in 15 Imaged twice on the same scanner w/in 15
minutesminutes Thin section (1.25 mm) imagesThin section (1.25 mm) images Manual linear measurements performed by 3 Manual linear measurements performed by 3
readers; volume obtained from algorithm.readers; volume obtained from algorithm.
QuestionsQuestions
1.1. METHODS and MATERIALSMETHODS and MATERIALS MSK Coffee break experiment patient MSK Coffee break experiment patient
datasetsdatasets Repeat scan of lesions over short time interval Repeat scan of lesions over short time interval
--- “No change” condition--- “No change” condition True size is unknownTrue size is unknown
QuestionsQuestions1.1. METHODS and MATERIALSMETHODS and MATERIALS Like NIST Biochange 2008, lesions are identified and Like NIST Biochange 2008, lesions are identified and
coordinates provided to readers.coordinates provided to readers. Reader tasksReader tasks
For each lesion, manually mark 2 Diameters For each lesion, manually mark 2 Diameters • Longest diameter and perpendicular diameterLongest diameter and perpendicular diameter
Also each reader obtains a semiautomated contour of entire Also each reader obtains a semiautomated contour of entire volume of lesionvolume of lesion
• From contour, determine volume, longest diameter and From contour, determine volume, longest diameter and diameter perpendicular to longest diameterdiameter perpendicular to longest diameter
• Keep actual contour coordinates/mask (TBD)Keep actual contour coordinates/mask (TBD) Randomize order of cases so reader does not simultaneously Randomize order of cases so reader does not simultaneously
review both scans of a patient (does not see both scans of same review both scans of a patient (does not see both scans of same lesion); has to contour them independently.lesion); has to contour them independently.
Data CollectedData CollectedQIBA QIBA Patient ID Patient ID ##
Scan#Scan# Nodule Nodule ##
coordscoords R1 R1 volvol
R1 R1 LD LD from from contourcontour
R1 R1 PD PD from from contourcontour
R1 R1 LDLDmanualmanual
R1 R1 PDPDmanualmanual
R2 R2 volvol
11 11 11 (320,267, (320,267, 52)52)
2525
11 22 11 (310, (310, 248, 48)248, 48)
2222
11 11 22 (221,335, (221,335, 13)13)
3434
11 22 22 (215, (215, 340, 15)340, 15)
2828
Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)
in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS AnalysesAnalyses
Estimate variance of different change metrics forEstimate variance of different change metrics for• VolumeVolume• Diam (manual and Assisted)Diam (manual and Assisted)• Product of diameters (LD x PD) (Manual and assisted) Product of diameters (LD x PD) (Manual and assisted)
Estimate inter-reader variabilityEstimate inter-reader variability
Intra-reader variability ?? (can we do that with data we Intra-reader variability ?? (can we do that with data we would have? Or would we have to specifically introduce would have? Or would we have to specifically introduce the exact same cases multiple times?)the exact same cases multiple times?)
Questions to be Pursued in a Questions to be Pursued in a Future PhaseFuture Phase
QuestionsQuestions2. 2. What level of reproducibility in estimating What level of reproducibility in estimating changechange can be achieved can be achieved
in measuring tumors in in measuring tumors in phantomphantom datasets? datasets?(a)(a) RECIST change vs. volume changeRECIST change vs. volume change(b)(b) Investigate just variance (not bias)?Investigate just variance (not bias)?(c)(c) inter-observer variabilityinter-observer variability(d)(d) Intra-observer variabilityIntra-observer variability(e)(e) Change metric – absolute value? fractional change in volume/diameter? Change metric – absolute value? fractional change in volume/diameter?
categorical variable?categorical variable?
For this question:For this question: Variety of lesions with known sizeVariety of lesions with known size Compare “size” of different lesions somehow (TBD)Compare “size” of different lesions somehow (TBD)
Use different existing lesions and treat them as though they were the Use different existing lesions and treat them as though they were the same lesion at different time points.same lesion at different time points.
Physically alter lesions over time and scan at both time points (Bob Ford’s Physically alter lesions over time and scan at both time points (Bob Ford’s water balloon experiment)water balloon experiment)
QuestionsQuestions4. 4. What level of reproducibility in estimating What level of reproducibility in estimating changechange can be achieved in can be achieved in
measuring tumors in measuring tumors in patientpatient datasets datasets with “Unknown Change” conditionwith “Unknown Change” condition??(a)(a) RECIST change vs. volume changeRECIST change vs. volume change(b)(b) Investigate just variance?Investigate just variance?(c)(c) inter-observer variabilityinter-observer variability(d)(d) Intra-observer variability (would be good to have more than 2 time points; may not Intra-observer variability (would be good to have more than 2 time points; may not
exist in RIDER).exist in RIDER).(e)(e) Change metric – absolute value? fractional change in volume/diameter? categorical Change metric – absolute value? fractional change in volume/diameter? categorical
variable?variable?(f)(f) Look at effects of lesion size for a specific (thin) slice thicknessLook at effects of lesion size for a specific (thin) slice thickness
For this question:For this question: Variety of lesions (true size may be unknown)Variety of lesions (true size may be unknown) Patient datasets with same lesions at different time points.Patient datasets with same lesions at different time points. Lesions may or may not have changed size – change is unknown.Lesions may or may not have changed size – change is unknown. RIDER datasets – unannotated as of yet. We have identified about 20 RIDER datasets – unannotated as of yet. We have identified about 20
lesions of various sizes. Possible cases from RadPharmlesions of various sizes. Possible cases from RadPharm..
QuestionsQuestions5. 5. What is the effect of What is the effect of slice thicknessslice thickness on on estimating changeestimating change in tumors using in tumors using patientpatient
datasets?datasets?(a)(a) RECIST change vs. volume changeRECIST change vs. volume change(b)(b) Investigate just variance?Investigate just variance?(c)(c) inter-observer variabilityinter-observer variability(d)(d) Intra-observer variabilityIntra-observer variability(e)(e) Change metric – absolute value? fractional change in volume/diameter? categorical variable?Change metric – absolute value? fractional change in volume/diameter? categorical variable?(f)(f) Look at interactions between slice thickness and lesion size (and other lesion characteristics Look at interactions between slice thickness and lesion size (and other lesion characteristics
such as shape, margin, etc.)such as shape, margin, etc.)
For this question:For this question: Also Variety of lesions (true size may be unknown)Also Variety of lesions (true size may be unknown) Patient datasets with same lesions at different time points, all done originally with Patient datasets with same lesions at different time points, all done originally with
thin slices; create a thick slice series (average together adjacent images).thin slices; create a thick slice series (average together adjacent images). Lesions may or may not have changed size – change is unknown.Lesions may or may not have changed size – change is unknown. Estimate change on thin and thick series and compareEstimate change on thin and thick series and compare Thin Slice RIDER datasets that can be fused together. Other cases from Thin Slice RIDER datasets that can be fused together. Other cases from
RadPharm?RadPharm?