METHODOLOGY OpenAccess … · 2017. 8. 28. · icaltargetvolume(CTV),optimaldosepaintinginsidethe GTV and more effective sparing of OARs. Several studies have focused on exploring

Chen et al. Radiation Oncology 2013, 8:293http://www.ro-journal.com/content/8/1/293

METHODOLOGY Open Access

Multiple comparisons permutation test forimage based data mining in radiotherapyChun Chen*, Marnix Witte, Wilma Heemsbergen and Marcel van Herk

Abstract

Comparing incidental dose distributions (i.e. images) of patients with different outcomes is a straightforward way toexplore dose-response hypotheses in radiotherapy. In this paper, we introduced a permutation test that comparesimages, such as dose distributions from radiotherapy, while tackling the multiple comparisons problem. A test statisticTmax was proposed that summarizes the differences between the images into a single value and a permutationprocedure was employed to compute the adjusted p-value. We demonstrated the method in two retrospectivestudies: a prostate study that relates 3D dose distributions to failure, and an esophagus study that relates 2D surfacedose distributions of the esophagus to acute esophagus toxicity. As a result, we were able to identify suspicious regionsthat are significantly associated with failure (prostate study) or toxicity (esophagus study). Permutation testing allowsdirect comparison of images from different patient categories and is a useful tool for data mining in radiotherapy.

IntroductionWhen planning a radiotherapy treatment, a compromiseis made between coverage of the target and exposure ofOrgans At Risk (OAR). While the dose to the designatedtarget is generally uniform and homogeneous betweenpatients, the dose to surrounding structures can be highlyvariable, depending on patient geometries, tumor loca-tions, and treatment techniques. Such heterogeneous inci-dental dose distributions in patients might “accidentally”lead to different treatment outcomes regarding tumorcontrol (e.g. if subclinical disease is important) or nor-mal tissue toxicity. Therefore, applying data mining tech-niques to incidental dose distributions gives the possibilityto explore dose patterns that are associated with clinicaloutcomes.The purpose of introducing data mining in radiotherapy

is to explore hypotheses for dose-response relationships.In cancer radiotherapy treatment, variations in stem cells,tumor microscopic disease and radiosensitivity distribu-tions can be expected to affect dose-response relation-ships. Unfortunately, many of them are unknown. Datamining on incidental dose may yield suspicious anatom-ical features from which –based on biological or clinical

*Correspondence: [email protected] of Radiation Oncology, The Netherlands Cancer Institute,Plesmanlaan 121, 1066CX, Amsterdam, The Netherlands

considerations– hypotheses for dose-response relation-ships can be formulated. If validated, such dose-responserelationships would eventually provide evidence for bettertreatment planning, such as refined knowledge of the clin-ical target volume (CTV), optimal dose painting inside theGTV and more effective sparing of OARs.Several studies have focused on exploring dose-

response relationships from a different perspective thanthe conventional dose volume histogram (DVH) basedmethod [1-3]. These methods include either exploringthe characteristics of dose distributions (e.g. eccentric-ity, homogeneity), or applying an advanced classifier (e.g.neural network). However, these methods are not easilyapplicable in the situation where the hypothesized regionis a priori not known. Directly comparing dose distribu-tions is then a straightforward method of exploring dose-response relationships. Since the dose at each voxel iscompared without prior anatomical or geometrical basedhypothesis, a voxel-by-voxel based testing is suitable forhypothesis generation, i.e., to localize suspicious regions.In a prostate study [4], the 3D dose prescribed to prostatepatients were registered to an anatomy grid and testedvoxel-by-voxel (t-test) for relations with failure. Resultsindicate that a cluster of voxels outside the prostate yielda p-value of less than 0.05. However, obtaining a p-valueat every voxel is not yet the complete result. Since a largenumber of voxels were tested simultaneously, it is likely

© 2013 Chen et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly cited.

Chen et al. Radiation Oncology 2013, 8:293 Page 2 of 10http://www.ro-journal.com/content/8/1/293

that the null hypothesis was incorrectly rejected at somevoxels (type I errors), and this is the so called multiplecomparisons problem.The aim of this paper is to introduce a multiple com-

parisons permutation test to compare images betweenpatients in radiotherapy studies.We begin with describingthe methodology. Afterwards, we demonstrate the valid-ity of this method with simulations. Finally, we give twoexamples of applying permutation test in radiotherapy:one study that relates dose to failure for prostate can-cer patients [4] and another study that relates dose toacute esophagus toxicity for non-small cell lung cancer(NSCLC) patients.

Materials andmethodsA permutation test involves five steps: 1) register imagesfrom different patients, 2) form a null hypothesis, 3) definea scalar test statistic, 4) generate random samples by per-muting the true labels of the patients and extract thetest statistic from each random sample, 5) calculate theadjusted p-value from the distribution of the test statistic.Thus, instead of a p-value for every voxel, this test givesa single p-value to describe the difference between twoimaging datasets.Suppose we observe a sample of patients with two out-

comes: non-event (N) and event (E). These patients areconsidered to be representative for the entire popula-tion. To compare the dose distributions between the twogroups, the first step is to register the dose distributionsof all patients into the same grid, through an image regis-tration method [5,6]. The null hypothesis then states thatthere be no difference in dose distributions between theN and E labeled groups. In the following part, we intro-duce a test statistic Tmax, and describe the permutationprocedure to compute the adjusted p-value.

Test statisticIn order to compare the dose distribution for a sample i(randomly drawn from the study population) that includestwo outcome groups N and E, the most straightforwardway is to compute their average dose difference at eachvoxel, resulting in a dose difference map. To account formultiple comparisons, we can choose the maximum valueof such a dose difference map as a single number teststatistic. However, the maximum will not be consistentover all random samples (e.g. i = 1, . . . , 1000), becauseit is highly sensitive to the variation or standard devi-ation (SD) of the dose difference at each voxel over allrandom samples. For instance, if voxel 1 has an averagedose difference of 10 Gy in sample i but the SD of thedose difference is 10 Gy over all samples, while voxel 2has an average dose difference of 8 Gy in sample i andthe SD of the dose difference is 1 Gy over all samples, themaximum dose difference for sample i would be 10 Gy as

derived from voxel 1. In fact, it is more likely that in sam-ple i, the large 10 Gy dose difference for voxel 1 is due tochance, because it has a large variation over all samples.To account for this effect, the average dose difference di,kfor sample i, (i = 1, . . . ,Np) between the E and N groupsat voxel k, (k = 1, . . . ,Nv) should be normalized into Ti,kaccording to an estimate of its SD:

di,k = μE,i,k − μN,i,k (1)

Ti,k = di,kσk

(2)

where μE,i,k and μN,i,k are the average dose at voxel k forgroup E and N in sample i, and σk is the standard devi-ation of di,k over Np samples. σk is computed over therandom samples generated from the permutation proce-dure, as described in the following part. As a result, weobtain a normalized dose difference map (or Ti,k map) forsample i. The test statistic Tmax,i is then selected as themaximum value of the Ti,k map. Unlike a voxel-by-voxelbased test, Tmax,i gives a single number that summarizesthe discrepancy of the dose distributions between the twolabel groups, rather than the discrepancy of a particularvoxel. Therefore, Tmax accounts for multiple comparisons.Clearly, Tmax is not the only option for extracting a singlevalue test statistic from the Tk map. Other test statis-tics like the x percentile, are also eligible [7]. However,Tmax is often chosen for its strong control over Type 1errors [8].

Permutation testWe then introduce a permutation procedure that gen-erates random samples under the null hypothesis, suchthat the distribution of Tmax,i is determined. A permu-tation test relies on the rearrangement of the outcomelabels. Under the null hypothesis that there is no signif-icant dose difference between group E and N, labels Eand N are exchangeable. Thus randomly permuting thelabels of the observed sample gives a new randomizedsample. For example, in a sample of 5 patients with 3 Nand 2 E labels, there are 10 possible label sets, as shownin Table 1. If the first case is the true labelling that weobserved, the other 9 labellings are assigned by permu-tation. In practice, the numbers of both E and N labelsare large in the sample, leading to an extremely large andunfeasible number of possible permutations. However, ithas been shown in [9] that when the number of patients inthe observed sample is large, 1000 permutations can effec-tively approximate the distribution for the null hypothesis.The permutation test contains the following steps: Forevery permuted sample i, a di,k is computed for every voxelk. After 1000 permutations, the standard deviation σk of{di,k , i = 1, . . . , 1000} at voxel k is computed and the nor-malized dose difference maps Ti,k and T̃k are generated


Table 1 The 10 possible label combinations of 3 Ns and 2 Es

1. N N N E E 6. N E E N N

2. N N E N E 7. E N N N E

3. N N E E N 8. E N N E N

4. N E N N E 9. E N E N N

5. N E N E N 10. E E N N N

If the first case is the true labels that we observed, the other 9 labels are assignedby permutation.

for every permuted sample i and the observed sample,respectively. Subsequently, the maximum Tmax,i of Ti,k isextracted for every permuted sample i and the maximumT̃max of T̃k is extracted from the observed sample. Thus,after 1000 permutations, we obtain a distribution of Tmax,iunder the null hypothesis. Finally, the adjusted p-valueis computed as the proportion of the permutation sam-ples that yield a higher Tmax value than the T̃max observedwith the true labels. If the adjusted p-value is smaller thansignificance level α (e.g. 0.05), we reject the null hypoth-esis. Furthermore, the (1 − α) percentile of the Tmax,idistribution gives a threshold value T∗. In the observednormalized dose difference map T̃k , voxels higher than T∗show significant dose difference between E and N groups.The mathematical formulation of the permutation test ispresented in more detail in Appendix A.

SimulationTo demonstrate the permutation test, we conducted simu-lations on two groups of artificially generated dose images(with 128×128 pixels). Each group had 50 images. As seenin Figure 1(a) and 1(b), group E had a homogeneous aver-age dose of 60 Gy, while for group N, a block (50 × 50) ofaverage 58 Gy was located inside the dose image. At eachpixel, we simulated the additive dose variation as a normaldistribution N ∼ (0, σ). Figure 1(c) illustrates the simu-lated values for σ . Large variations were simulated in theupper part (σmax = 10 Gy), while small variations werein the lower part (σmin = 1 Gy). After adding noise tothe average dose, a smooth Gaussian filter was applied toeach patient’s dose image. The purpose of generating spa-tially correlated variation and applying Gaussian filter is tomimic actual spatially correlated incidental dose in plan-ning. We first illustrate how the the multiple comparisonpermutation test works step-by-step. Afterwards, we con-duct a sensitivity analysis to show how the result changesaccording to the average dose difference and the samplesize, compared to the voxel-by-voxel based t-test.First, a voxel-by-voxel based t-test was applied to these

two groups. Pixels obtaining a p < 0.05 are illustrated inFigure 2(a), showing a big central region as well as manyisolated spots. These isolated spots obtained a low p-valueby chance. For the permutation test, Figures 2(b)–2(d)show the average dose difference map for the observed

sample, the SD of the average dose difference from 1000permutation random samples, and the consequent nor-malized average dose difference map for the observedsample, respectively. The distribution of Tmax,i from 1000random samples is illustrated in Figure 2(e). The T̃maxfrom the observed sample is much larger than any of theTmax,i, thus the adjusted p-value is zero, i.e. there is ahighly significant dose difference between the two groups.The 95% of the Tmax,i distribution gives a threshold T∗, byapplying it to the T̃k map, we obtained the second regionillustrated in Figure 2(a). This region is smaller than thatfrom a voxel-by-voxel based test. Additionally, the iso-lated spots, which would probably falsely reject the nullhypothesis by the voxel-by-voxel based test, are excluded.Therefore, compared to the voxel-by-voxel based test, per-mutation test provides a statistically stronger result, interms of an adjusted p-value and a more accurate regionwith dose differences.Furthermore, Figure 3 shows the simulation results by

increasing the average dose difference, from 0.5 Gy, 2Gy (example above) to 10 Gy, given the other parame-ters fixed as the above example. The larger the averagedose difference is, the more regions were detected forboth multiple comparisons test and t-test, especially atthe top of the square region, where the dose variance ishigher (Figure 2(c)). Multiple comparison test becomesmore conservative than the t-test, when the average dosedifference is small. However, t-test always ends up withmore isolated false positive spots (type-I error).Similarly, Figure 4 shows the simulation results by

increasing the number of patients, from n = 5, n =50 to n = 100. Results show that multiple comparisontest becomes more conservative than the t-test, especiallywhen the sample size is small. However, in any cases, t-test ends up with more isolated false positive spots (type-Ierror).

ApplicationsStudy I: prostateWe applied the permutation test on data used by [4]. Theaim of this study was to relate dose distributions with fail-ure in prostate cancer patients. We selected a group of 67patients with a relatively higher risk for extraprostatic dis-ease, estimated according to T-stage, iPSA and Gleasonscore or differentiation grade [10]. These patients weretreated in Erasmus Medical Center, The Netherlands, andthey were included in the Dutch Phase III trial (CKVO96-10) with dose randomized between 68 Gy and 78 Gy[11]. The Ethical Committee of each institution approvedthe protocol. Patients mainly had tumors of stage T3b andwere treated to the delineated prostate and the seminalvesicles. The extra boost of 10 Gy had a 5 mm margin tothe CTV (except towards the rectum, where a 0 mm mar-gin was applied). For the 68 Gy PTV, a 10 mm margin


40

45

50

55

60

65

70

(a)40

45

50

55

60

65

70

(b)

2

3

4

5

6

7

8

9

(c)

(d) (e) (f)

(g)Figure 1 Simulated dose images (128 × 128 pixels) in two groups (E and N). (a) and (b) are the average dose map for E and N groups.(c) is the simulated SD of variations. Each group was simulated with 50 images. (d-e) and (f-g) are two examples of the dose map for group E and N,respectively.

was applied. In this study, the failure was biochemical(PSA nadir plus 2) [12] or clinical (locoregional or distantprogression or start of salvage hormone therapy), deter-mined at a fixed 4 year endpoint. As a result, 37 failurepatients and 30 non-failure patients were eligible for anal-ysis. Delineations from the planning CT and the planneddose distributions were collected for each patient. Firstly,dose distributions of all patients were registered intoa dose grid as described in [4]. In short, voxels correspondif their direction with respect to the center of mass (CM) isthe same, and their distance to the surface in this directionis the same. For voxels inside the prostate, corresponding

voxels have the same fractional distance between theCM and the surface. The registration identifies anatomi-cal points at locations relative to the delineated prostatesurface. The choice for this registration procedure is animportant part of the dose-effect hypothesis, and wasbased on the suspicion that extracapsular extension mighthave affected outcome. The resulting grid has a dimensionof 31 × 35 × 34, resulting in Nv = 36890 dose voxels. Thenull hypothesis is that there is no dose distribution differ-ence between the failure and the non-failure patients. Themultiple comparison permutation test was applied to theregistered dose maps.


(a)0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

(b)

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

(c)

0

5

10

15

20

(d)

5 10 15 20 2520

0.05

0.1

0.15

0.2

0.25

0.3

(e)Figure 2 Permutation test procedure with simulations. (a) The identified region with p < 0.05, through a voxel-by-voxel based t-test (graycontour) and the permutation test (black contour). (b) The average dose difference map d̃k (k = 1, . . . , 128 × 128) for the observed sample. (c) TheSD σk of the dose difference, computed from 1000 permutated random samples. (d) The normalized dose difference map T̃k for the observedsample, which is (b) divided by (c). (e) The distribution of Tmax,i from i = 1, . . . , 1000 permutation samples, where T̃max is the maximum value infigure (d) and T∗ is the 95 percentile of Tmax,i .

Study II: esophagusWe applied the permutation test on a esophagus toxicitystudy. The aim of this study is to relate dose distributionson the esophagus surface with acute esophagus toxic-ity (AET) in NSCLC patients. We selected 185 NSCLCpatients treated in Netherlands Cancer Institute (NKI)from 2008 to 2010 with concurrent chemotherapy com-bined with IMRT. The RT dose was 66 Gy in 24 fractions.The concurrent chemotherapy included a daily low dosecisplatin [13]. AET was scored according to the Common

Toxicity Criteria 3.0. Toxicity was scored weekly frombaseline until 3 weeks after RT. Afterwards, patients werechecked every 2 months or more frequently if necessary.Of the 185 patients, 76 had no or grade 1 AET; 67 patientsdeveloped grade 2 and 42 patients had grade 3; Grade 4or 5 AET did not occur. The delineated esophagus fromthe planning CT and the planned dose distributions wereretrospectively collected for each patient, allowing a 2Desophagus surface dose map (ESDM) to be computed. Foreach patient, dose was sampled on every slice of the CT

(a) (b) (c)Figure 3 The identified region with p < 0.05, through a voxel-by-voxel based t-test (gray contour) and the permutation test (blackcontour), given the average dose difference of (a) 0.5 Gy, (b) 2 Gy and (c) 10 Gy.


(a) (b) (c)Figure 4 The identified region with p < 0.05, through a voxel-by-voxel based t-test (gray contour) and the permutation test (blackcontour), given the number of patients of (a) n = 5, (b) n = 50 and (c) n = 100.

scan (3 mm thickness) at 36 fixed orientations along thedelineated esophagus contour to the center, from 0 to 360degrees with 10 degrees increment. The 0 degree anglewas always chosen in the Right-Left direction. (In ourexperience the esophagus is always star shaped, i.e. the fullcontour can always be seen from a single centreline.) Thesame sampling procedure was then done through all themslices where the esophagus was delineated. As a result, theESDM contains m × 36 pixels for every patient (m variesfrom patient to patient). ESDMs of all patients were reg-istered such that the pixel with the highest dose is in thecenter of the 2D dose map, alowing translations along androtations around the length of the esophagus. The choicefor this mapping was based on the assumption that thelength and the circumference of the high dose region onthe esophagus surface is associated with AET, irrespectiveof its anatomical location. Permutation tests were appliedto find differences between grade 0–1 and grade 2–3, andbetween grade 0–2 and grade 3 AET.

ResultsResults of study IThe distribution of Tmax,i, (i = 1, . . . , 1000) from the1000 random permutations under the null hypothesis isshown in Figure 5. The proportion of Tmax,i that arelarger than the observed value (T̃max = 3.81) givesan adjusted p-value of 0.02, i.e., there is a significantdose difference between the non-failure and the fail-ure patients for this patient group at α = 0.05 sig-nificance level. Furthermore, the 95 percentile of theTmax,i distribution gives the threshold T∗ = 3.56. InFigure 6, voxels above this threshold in the observed T̃kmap are marked, overlaid on the CT scan of the stan-dard anatomy grid. This region is situated in the obtu-rator region and suggests nodal involvement, but doesnot correspond to the presumed extracapsular exten-sion. The significant region is much smaller than thatobtained through a voxel-by-voxel based t test [4]. Theadjusted value p = 0.02 was significant for the test

statistic Tmax. Compared to the previous voxel-by-voxelbased testingmethod, the permutation test gives a statisti-cally stronger conclusion that there is indeed a differencein the dose distribution outside the prostate betweenfailure and non-failure patients.

Results of study IIAn example of generating an ESDM is illustrated inFigure 7. The average dose maps of the registered ESDMsfor all 185 patients as well as each toxicity grade subgroupare illustrated in Figure 8. The differences in the isodoselines for patients with grade 0–1 until grade 3 AET implythat for patients of increasing AET grade, their averageesophagus surface that received high dose increases: Allisodose lines expand along the length of the esophagus forpatients with more complications. Specifically, the 50 Gyand 60 Gy isodose lines are expanding through both thelength and the circumference.

1 1.5 2 2.5 3 3.5 4 4.5 4.70

0.05

0.1

0.15

0.2

0.25

0.28

Figure 5 The histogram of Tmax,i obtained from the 1000random permutations. The adjusted p-value (P = 0.02) is computedas the black area larger than the observed T̃max (solid line). The 95percentile T∗ (dashed line) determines the region with significantdose difference as shown in Figure 6.


(a) (b) (c)Figure 6 The region (marked in red) with significant dose difference (adjusted p-value< 0.05) from the permutation test in the prostatestudy, together with the region (marked in yellow) with p-value< 0.05 in a voxel-by-voxel based t test. (a) coronal view; (b) axial view;(c) sagittal view.

The Tmax distribution, adjusted p-value and T∗ thresh-old were computed in the same way as in study 1.The adjusted p-value for the dose distribution betweenpatients with AET grade 0–1 and grade 2–3 is p < 0.001,showing a significant dose difference at α = 0.05 signif-icance level. The region with significant dose differenceis illustrated in Figure 9(a). Note that the region refers toa certain shape of dose distribution (as the highest doseswere mapped to the same point), rather than a specificanatomical area on the esophagus. Considering the esoph-agus as a tube, this region includes a full band covering thehigh dose area (35 to 65 Gy) and a sub-band covering themoderate dose region (10 to 30 Gy). Similarly, the adjustedp-value for the dose distribution of patients between AETgrade 0–2 and grade 3 is p = 0.002. The region withsignificant dose difference is illustrated in Figure 9(b), itappears to be part of the significant region in Figure 9(a),

showing a high dose (50 to 60 Gy) area with extra lengthand circumference coverage.

DiscussionIn this paper, we introduced multiple comparison per-mutation testing for voxel based data mining in radio-therapy and we demonstrated the test in two studies.For both studies we were able to locate regions wheredose significantly associates with the outcome. In theprostate study, we were able to provide strong sta-tistical evidence for a dose difference between non-failure and failure patients, confirming a differencelocated in the obturator region that could be suspi-cious for subclinical disease. In the esophagus study,both regions to predict grade ≥ 2 and grade 3are consistent with the V50 dose volume histogram(DVH) parameter as derived in study [13]. Grade 2

R L RP A

S

I

originalESDM

Planning CT

Dose

36 orientations

m slices

registeredESDM

Figure 7 An example of generating an esophagus surface dose map (ESDM). The pixel with the highest dose (in white) is registered to thecenter location in the registered ESDM through translation and rotation of the original ESDM.


Figure 8 The average dose map of the locally registered ESDMs. (a) all patients; (b) patients with AET Grade 0–1; (c) patients with AET Grade 2;(d) patients with AET Grade 3. The iso-dose lines of 60 Gy, 50 Gy, 30 Gy and 10 Gy are marked in black, white, red and yellow.

seems to be caused by high dose (≥ 50 Gy) and thelength/circumference coverage of low dose, while thelength/circumference coverage of high dose (around 50Gy) plays a role in severe AET of grade 3. This result sug-gests that using the length and circumference parametersmay be a more sensitive method to predict AET comparedto DSHs.A broadly recognized method to address the problem

of multiple comparisons is the Bonferroni correction [14]:if n independent hypotheses are tested, each individualhypothesis is tested at the 1/n times of the original statisti-cal significance level when tested for only one hypothesis.However, this correction is not straightforwardly applica-ble to voxel maps, since there can be millions of voxelsthat are highly correlated in space. Hypothesis testing onimages was first conducted through parametric randomfield methods [15]: t-tests are conducted at every voxeland the distributional results for continuous random fieldsare used to identify regions that are significant. Contraryto these parametric methods, non-parametric permuta-tion tests on voxel maps were introduced by [8,16]. Twotest statistics are often used: a single maxima thresholdand the supra-threshold cluster. A single maxima thresh-old is the Tmax as we used in our study. In [16], Tmaxwas applied in a permutation test to localize the regionof visual cortex sensitive to motion on 3D PET imaging[17] and to analyze the order effects in working memoryusing fMRI [18]. Contrarily, a supra-threshold cluster testassesses the size of the connected supra-threshold regionsfor significance. As a result, the power to localize regionswas reduced. Since the goal of data mining in radiotherapywas to localize suspicious regions, we recommend usingTmax as test statistic.

The incidental dose essentially comes from the varia-tions of dose planning for some un-targeted organs, andit’s a good thing to explore. Whether or not we are ableto detect a significant dose difference depends on twoaspects: 1) the average incidental dose between 2 groups

(a) (b)Figure 9 The region (in white contour) with significant dosedifference (a) between AET Grade 0–1 and 2–3, and (b) betweenAET Grade 0–2 and 3, overlaid on the average dose map of thetotal 185 patients.


and 2) the variance of the incidental dose. Statistically,a higher average incidental dose difference and a lowervariance of each group facilitates yielding a true pos-itive result. On the other hand, registration error is abad thing. Inaccuracies in the registration, or an inappro-priate choice for the registration method, could preventthe method to identify an existing dose effect relation,thus reducing the power of the statistical test. While itseems less likely that some particular registration pro-cedure or inaccuracy therein generates a false positiveresult from dose variations which only consist of noise,any dose effect relation which is subsequently derivedshould be verified on an independent data set. Depend-ing on the specific anatomical properties and the expecteddose-effect parameters, the requirements for the accuracyof the registration vary. For instance, if the data miningis conducted in regions with small structures (e.g. headand neck), a sophisticated registration procedure may berequired to find significant results. Contrarily, if we wantto explore a large volume of low gradient dose distribu-tions (e.g. lung), a loose registration may suffice. If we aimto explore dose distributions surrounding one structure,the registration accuracy is then focused on regions closeto this structure. Therefore, a registration strategy shouldbe chosen in advance based on the type of hypothesisthat we want to explore. Afterwards, significant regionscan be anatomically identified, and subjected to biologicaland clinical interpretation. Such consideration can thenguide further efforts to derive dose-response relationshipsthrough conventional modeling methods.Permutation testing is a useful tool to explore dose

patterns from incidental dose distributions. Instead ofanalyzing dose-response effect, we intend to use permu-tation testing as a preliminary step to identify suspiciousregions for hypothesis generation. Permutation testingtakes into account multiple comparisons by yielding anadjusted p-value and gives visually straightforward suspi-cious regions. Another advantage of such method is thatit is non-parametric. Thus, this test does not depend onthe assumption of Normal distribution, which is often nottrue in the case of incidental dose in the planning. Permu-tation testing is practically useful and important in radio-therapy, especially in the era where adaptive radiotherapyis on the agenda, but we still have only limited knowledgeabout tumor stem cells, microscopic disease, radiosen-sitivity, etc. Permutation testing helps us to maximallyexplore dose-response relationships from the incidentaldose in the clinical data.

ConclusionsWe introduced a permutation test that deals with hypoth-esis testing on images and illustrated this method in asynthetic dataset, and in clinical datasets from a prostateand an esophagus study. Compared to a voxel-by-voxel

based test, the permutation method reduces the rate offalse positives. Permutation testing is a useful tool to iden-tify hypotheses for dose-response relationships and tacklethe multiple comparisons problem.

ConsentWritten informed consent was obtained from the patientfor the publication of this report and any accompanyingimages.

Appendix A: themultiple comparisonspermutation testSuppose we observed a sample of patients with two out-comes: non-event (N) and event (E). Every patient hasa dose distribution of Nv voxels and they are all regis-tered to an identical grid. To compare the dose distri-bution between the two groups, the permutation test isconducted as follows:

(i) Compute the average dose difference d̃k betweenE and N groups in the observed sample:

d̃k = μ̃E,k − μ̃N,k , k = 1, . . . ,Nv , (A.1)

where μ̃E,k and μ̃N,k are the average dose value atvoxel k for group E and N, respectively.

(ii) Permute the labelling of the observed sample andcompute the average dose difference. Repeat thisprocess for Np times:

di,k = μE,i,k −μN,i,k , k = 1, . . . ,Nv, i = 1, . . . ,Np ,(A.2)

where μE,i,k and μN,i,k are the average dose valueat voxel k for group E and N in the ith permutedrandom sample.

(iii) Compute the standard deviation for every voxel kover all Np random samples:

σk =√√√√ 1

Np − 1

Np∑i=1

(di,k − d̄k), where d̄k = 1N

Np∑i=1

di,k .

(A.3)

(iv) Compute the locally normalized dose differencefor every voxel in every random sample as well asthe observed sample:

Ti,k = di,kσk

, (A.4)

T̃k = d̃kσk

, k = 1, . . . ,Nv . (A.5)


(v) Compute the test statistic Tmax for everyresampling as well as the true labeling sample:

Tmax,i = max(Ti,k) , (A.6)

T̃max = max(T̃k), k = 1, . . . ,Nv . (A.7)

(vi) Compute the adjusted p-value:

p = Pr[Tmax,i > T̃max

]. (A.8)

(vii) Compare the adjusted p-value with thesignificance level α. If p < α, reject the nullhypothesis, otherwise the null hypothesis can notbe rejected.

(viii) Compute the T∗ as (1 − α) percentile ofTmax, i = 1, . . . ,Np. Regions in T̃k that are aboveT∗ show significant difference between E and Ngroups.

Competing interestsThe authors declare that they have no competing interests.

Authors’ contributionsAll authors participated in the study and drafted the manuscript. All authorsread and approved the final manuscript.

AcknowledgementsThe authors would like to acknowledge the Dutch center for translationalmolecular medicine (CTMM) for supporting the project “Personalizedchemo-radiation of lung and head-and-neck cancer” (AIRFORCE).

Received: 6 August 2013 Accepted: 24 November 2013Published: 23 December 2013

References1. Buettner F, Gulliford SL, Webb S, Partridge M: Using dose-surface maps

to predict radiation-induced rectal bleeding: a neural networkapproach. Phys Med Biol 2009, 54:5139–5153.

2. Buettner F, Gulliford SL, Webb S, Sydes MR, Dearnaley DP, Partridge M:Assessing correlations between the spatial distribution of the doseto the rectal wall and late rectal toxicity after prostate radiotherapy:an analysis of data from the mrc rt01 trial (isrctn 47772397). PhysMed Biol 2009, 54:6535–6548.

3. El Naqa I, et al: Exploring feature-based approaches in pet images forpredicting cancer treatment outcomes. Pattern Recognit 2009,42:1162–1171.

4. Witte MG, Heemsbergen WD, Bohoslavsky R, Pos FJ, Al-Mamgani A,Lebesque JV, van Herk, M: Relating dose outside the prostate withfreedom from failure in the dutch trial 68 gy vs. 78 gy. Int J RadiatOncol Biol Phys 2010, 77:131–138.

5. Castadot P, Lee JA, Parraga A, Geets X, Macq B, Grégoire V: Comparisonof 12 deformable registration strategies in adaptive radiationtherapy for the treatment of head and neck tumors. Radiother Oncol2008, 89:1–12.

6. Goshtasby AA: 2-D and 3-D Image Registration: for Medical, Remote Sensing,and Industrial Applications. Hoboken: Wiley-Interscience; 2005. ISBN0471649546.

7. Chen C, Witte MG, Heemsbergen WD, van Herk M: Significance testingin dose difference maps. In Proc. of XVIth International Conference on theuse of Computers in Radiation Therapy. Amsterdam; 2010.

8. Holmes AP, Blair RC, Watson JD, Ford I: Nonparametric analysis ofstatistic images from functional mapping experiments. J Cereb BloodFlowMetab 1996, 16:7–22.

9. Edgington ES: Approximate randomnization tests. J Psychol 1969,72:143–149.

10. Partin AW, Yoo J, Carter HB, Pearson JD, Chan DW, Epstein JI, Walsh PC:The use of prostate specific antigen, clinical stage and gleason score

to predict pathological stage in men with localized prostate cancer.J Urol 1993, 150:110–114.

11. Peeters ST, Heemsbergen WD, van Putten, W L, Slot A, Tabak H, Mens JW,Lebesque JV, Koper PC: Acute and late complications afterradiotherapy for prostate cancer: results of a multicenterrandomized trial comparing 68 gy to 78 gy. Int J Radiat Oncol Biol Phys2005, 61:1019–1034.

12. Roach M, Hanks G, Thames H, Schellhammer P, Shipley WU, Sokol GH,Sandler H: Defining biochemical failure following radiotherapy withor without hormonal therapy in men with clinically localizedprostate cancer: recommendations of the rtog-astro phoenixconsensus conference. Int J Radiat Oncol Biol Phys 2006, 65:965–974.

13. Kwint M, Uyterlinde W, Nijkamp J, Chen C, de Bois J, Sonke JJ, van denHeuvel M, Knegjens J, van Herk M, Belderbos J: Acute esophagustoxicity in lung cancer patients after intensity modulated radiationtherapy and concurrent chemotherapy. Int J Radiat Oncol Biol Phys2012, 84(2):e223–e228.

14. Abdi H: Bonferroni and Sidak corrections for multiple comparisons.In Encyclopedia of Measurement and Statistics. Edited by Salkind NJ. Sage;2007.

15. Friston KJ, Holmes AP, Worsley KJ, Poline JP, Frith CD, Frackowiak RSJ:Statistical parametric maps in functional imaging: A general linearapproach. Human Brain Mapp 1994, 2:189–210.

16. Nichols TE, Holmes AP: Nonparametric permutation tests forfunctional neuroimaging: a primer with examples. Human BrainMapp 2002, 15:1–25.

17. Watson JDG, Myers R, Frackowiak RSJ, Hajnal JV, Woods RP, Mazziotta JC,Shipp S, Zeki S: Area V5 of the human brain: evidence from acombined study using positron emission tomography andmagneticresonance imaging. Cereb Cortex 1993, 3:79–94.

18. Marshuetz C, Smith EE, Jonides J, DeGutis J, Chenevert TL: Orderinformation in working memory: fMRI evidence for parietal andprefrontal mechanisms. J Cogn Neurosci 2000, 12(Suppl 2):130–144.

doi:10.1186/1748-717X-8-293Cite this article as: Chen et al.: Multiple comparisons permutation test forimage based data mining in radiotherapy. Radiation Oncology 2013 8:293.

Submit your next manuscript to BioMed Centraland take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at www.biomedcentral.com/submit

Documents

METHODOLOGY OpenAccess … · 2017. 8. 28. · icaltargetvolume(CTV),optimaldosepaintinginsidethe GTV and more effective sparing of OARs. Several studies have focused on exploring