bmjopen.bmj.com · For peer review only TRIPOD Checklist: Prediction Model Validation Section/Topic Item Checklist Item Page Title and abstract Title 1 Identify the study as developing

BMJ Open is committed to open peer review. As part of this commitment we make the peer review history of every article we publish publicly available. When an article is published we post the peer reviewers’ comments and the authors’ responses online. We also post the versions of the paper that were used during peer review. These are the versions that the peer review comments apply to. The versions of the paper that follow are the versions that were submitted during the peer review process. They are not the versions of record or the final published versions. They should not be cited or distributed as the published version of this manuscript. BMJ Open is an open access journal and the full, final, typeset and author-corrected version of record of the manuscript is available on our site with no access controls, subscription charges or pay-per-view fees (http://bmjopen.bmj.com). If you have any questions on BMJ Open’s open peer review process please email

[email protected]

on February 5, 2020 by guest. P

rotected by copyright.http://bm

jopen.bmj.com

/B

MJ O

pen: first published as 10.1136/bmjopen-2017-019996 on 24 M

ay 2018. Dow

nloaded from

http://bmjopen.bmj.com/

[email protected]


For peer review only

Value of prediction models for risk stratification of

pulmonary subsolid nodules: a retrospective cohort study

Journal: BMJ Open

Manuscript ID bmjopen-2017-019996

Article Type: Research

Date Submitted by the Author: 06-Oct-2017

Complete List of Authors: Kim, Hyungjin; Seoul National University Hospital Park, Chang Min; Department of Radiology Jeon, Sunkyung; Seoul National University Hospital Lee, Jong Hyuk; Seoul National University Hospital Ahn, Su Yeon; Seoul National University Hospital Yoo, Roh-Eul; Seoul National University Hospital Lim, Hyun-ju; National Cancer center Park, Juil; Seoul National University Hospital

Lim, Woo Hyeon; Seoul National University Hospital Hwang, Eui Jin; Seoul National University Hospital Lee, Sang Min; University of Ulsan College of Medicine Goo, Jin Mo; Seoul National University Hospital

<b>Primary Subject Heading</b>:

Oncology

Secondary Subject Heading: Diagnostics

Keywords: subsolid nodule, risk prediction model, Brock model, validation, lung cancer

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml

BMJ Open on F

ebruary 5, 2020 by guest. Protected by copyright.

http://bmjopen.bm

j.com/

BM

J Open: first published as 10.1136/bm

jopen-2017-019996 on 24 May 2018. D

ownloaded from



Value of prediction models for risk stratification of

pulmonary subsolid nodules: a retrospective cohort study

Hyungjin Kim1, Chang Min Park

1, 2, 3, Sunkyung Jeon

1, Jong Hyuk Lee

1, Su Yeon Ahn

1, Roh-

Eul Yoo1, Hyun-ju Lim

1, 4, Juil Park

1, Woo Hyeon Lim

1, Eui Jin Hwang

1, Sang Min Lee

5, Jin

Mo Goo1, 2, 3

1Department of Radiology, Seoul National University College of Medicine, Seoul, Korea

2Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul,

Korea

3Seoul National University Cancer Research Institute, Seoul, Korea

4Department of Radiology, National Cancer center, Goyang, Korea

5Department of Radiology and Research Institute of Radiology, University of Ulsan College

of Medicine, Seoul, Korea

* Corresponding author

Chang Min Park, MD, PhD. Department of Radiology, Seoul National University College

of Medicine, 101, Daehak-ro, Jongno-gu, Seoul 110-744, Korea. Tel: 82-2-2072-0367, Fax:

82-2-743-7418. E-mail: [email protected]

Word count: 3134

Page 1 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



ABSTRACT

Objectives: We aimed to validate the performances of two prediction models (Brock and Lee

models) for the differentiation of minimally-invasive and invasive pulmonary

adenocarcinomas (MIA/IPA) from preinvasive lesions among subsolid nodules (SSNs). The

purpose of our study was to evaluate the feasibility of the two models for the risk

stratification of persistent SSNs.

Design: A retrospective cohort study.

Setting: A tertiary university hospital in South Korea.

Participants: 410 patients with 430 SSNs who underwent surgical resection for the

pulmonary adenocarcinoma spectrum.

Primary and secondary outcome measures: Using clinical and radiological variables, the

predicted probability of MIA/IPA was calculated from pre-existing logistic regression models

(Brock and Lee models). Areas under the receiver operating characteristic curve (AUCs)

were calculated and compared between models. Performance metrics including sensitivity,

specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV)

were also obtained.

Results: For pure ground-glass nodules, the AUC of the Brock model in differentiating

MIA/IPA from preinvasive lesions was 0.680. Sensitivity, specificity, accuracy, PPV, and

NPV based on the optimal cutoff value were 62.9%, 68.2%, 65.1%, 73.6%, and 56.6%,

respectively. Sensitivity, specificity, accuracy, PPV, and NPV according to the Lee criteria

were 75.8%, 45.5%, 63.2%, 66.2%, and 57.1%, respectively. For part-solid nodules, the AUC

was 0.748 for the Brock model and 0.769 for the Lee model (P=0.623). Sensitivity, specificity,

Page 2 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



accuracy, PPV, and NPV were 81.5%, 55.6%, 79.3%, 95.3%, and 21.4%, respectively, for the

Brock model and 76.4%, 74.1%, 76.2%, 97.0%, and 22.2%, respectively, for the Lee model.

Conclusions: The performance of prediction models for SSNs in differentiating MIA/IPA

from preinvasive lesions was unsatisfactory. Thus, an alternative risk calculation model

should be developed for incidentally detected SSNs.

Strengths and limitations of this study

• This is the first study to externally validate the performance of pre-existing risk prediction

models for the incidentally detected pulmonary subsolid nodules.

• The main limitation of this study is that it only analyzed surgically resected lung nodules,

thus inducing selection bias.

Page 3 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



INTRODUCTION

Pulmonary subsolid nodules (SSNs) represent a histologic spectrum of

adenocarcinoma and its preinvasive precursors, including atypical adenomatous hyperplasia

(AAH), adenocarcinoma-in-situ (AIS), minimally invasive adenocarcinoma (MIA) and

invasive pulmonary adenocarcinoma (IPA).1 SSNs are common findings at chest computed

tomography (CT) which have been increasingly detected in CT screening studies.2 3

Indeed,

according to one prospective screening study, 4.2% of the participants had at least one pure

ground-glass nodule (pGGN) and 5.0% had at least one part-solid nodule (PSN) at baseline

rounds of screening.2

With this prevalence in mind, numerous studies have justifiably focused on the

differentiation of invasive adenocarcinomas from preinvasive lesions4-13

as invasive

adenocarcinoma requires surgical resection with conventional lobectomy and lymph node

dissection14

whereas preinvasive lesions can be followed-up conservatively with annual CT

surveillance or resected at a lesser extent (sublobar resection).15

Thus, the discrimination of

invasive adenocarcinoma has been a major topic of interest for many radiologists and

clinicians to date.

In a quest to obtain quantitative risk prediction tools for pulmonary nodules,

McWilliams et al.16

developed a prediction model (Brock model) utilizing various clinical

and radiological features. The Brock model demonstrated higher accuracy in determining the

likelihood of malignancy in pulmonary nodules compared to other existing models17

and was

also externally validated in three independent screening populations.18-20

Nevertheless, in the

context that a substantial percentage of persistent SSNs may belong to the adenocarcinoma

spectrum, the performance of the established model in differentiating invasive

Page 4 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



adenocarcinoma should also be validated in order to encourage the use of the model in

routine practice, as suggested by the British Thoracic Society (BTS).21

Lee et al.7 also developed a prediction model (Lee model) using simple size metrics

and morphologic features for the differentiation of invasive adenocarcinomas appearing as

SSNs. The model accuracy was reported to be excellent for the identification of invasive

adenocarcinomas. However, it has also not been tested or validated.

Therefore, we aimed to validate the performances of the two prediction models

(Brock and Lee models) for the differentiation of MIAs and IPAs from preinvasive lesions

among SSNs. The purpose of our study was to evaluate the feasibility of the two models in

the risk stratification of persistent SSNs.

Page 5 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



METHODS

This retrospective analysis was approved by the institutional review board of Seoul

National University Hospital (IRB No. 1705-116-855) and the requirement for written

informed consent was waived.

Study population

We retrospectively reviewed the electronic medical records of our hospital and found

1915 patients who had undergone surgical resection for lung cancer between 2011 and 2015.

Among the 1915 patients, we identified 1073 patients whose pathologic diagnoses belonged

to the pulmonary adenocarcinoma spectrum including AAH, AIS, MIA, and IPA.1 22

Thereafter, we reviewed the thin-section CT images of the patients to include only those with

SSNs (reviewers: J.S.K., J.H.L., S.Y.A., R.E.Y., H.L., and H.K.); 548 patients whose lung

cancers appeared as solid nodules on CT scans were excluded. We also excluded 76 patients

with nodules smaller than 5 mm or larger than 3 cm and 39 patients in whom data regarding

the family history of lung cancer were not available. Consequently, 410 patients were

included in this study. Among these patients, 18 patients had two nodules and one patient had

three nodules. Therefore, a total of 430 nodules from 410 patients were analyzed in the

present study (figure 1). There were 175 men and 235 women (mean age ± standard deviation,

60.9±9.6 years). As for the nodule type, there were 106 pGGNs and 324 PSNs. IPAs were

found in 305 nodules followed by MIA in 54 nodules, AIS in 52 nodules, and AAH in 19

nodules. Median nodule size was 15.6 mm (interquartile range, 11.7, 20.9 mm) (table 1).

Table 1 Demographic data of the entire study population

Characteristics Value

Page 6 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Age (year)a 60.9±9.6

Sex Male 175 (42.7)

Female 235 (57.3)

Pathology AAH 19 (4.4)

AIS 52 (12.1)

MIA 54 (12.6)

IPA 305 (70.9)

Nodule type Pure GGN 106 (24.7)

Part-solid nodule 324 (75.3)

Family history of lung cancer Yes 17 (4.0)

No 413 (96.0)

Emphysema Yes 44 (10.2)

No 386 (89.8)

Nodule size (mm)b 15.6 (11.7, 20.9)

Solid portion size (mm)b 5.7 (1.6, 11.2)

Solid proportion (%) 41.0 (11.4, 62.0)

Location Upper lobe 259 (60.2)

Other lobes 171 (39.8)

Lobulation Yes 143 (33.3)

No 287 (66.7)

Spiculation Yes 140 (32.6)

No 290 (67.4)

Nodule count per scanb 3 (1, 5)

Note.— Unless otherwise specified, data are numbers of patients (with percentages in

parentheses).

aData are mean ± standard deviation.

bData are median (with interquartile range in parentheses).

AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass

nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma

Page 7 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Data collection

Patient characteristics including demographic data were collected from the electronic

medical records of Seoul National University Hospital. Patient age, sex, pathologic diagnosis,

family history of lung cancer, and nodule location (lobe) were recorded. The thin-section CT

images were also reviewed to obtain radiologic information of nodules (nodule type, nodule

size, solid portion size, solid proportion, lobulation, spiculation, and nodule count per scan)

and the background lung parenchyma (presence of visually detected emphysema). These

features were used as input variables for logistic regression analysis at the Brock model16

and

Lee model.7 Nodule size and solid portion size were measured as the maximum transverse

diameter (mm) using an electronic caliper. Solid proportion (%) was calculated as the solid

portion size divided by the nodule size. Nodule count was defined as the total number of non-

calcified nodules at least 1 mm in diameter.16

Image review was conducted by three

radiologists (J.P., W.H.L., and H.K.) and each nodule was analyzed once by one of these

radiologists. Details regarding the CT scanning protocols are described in the online

supplementary material.

Measurement variability

We previously analyzed and reported the measurement variability of SSNs and solid

portion size using two same-day repeat CT scans.23

Measurement variability range for the

maximum transverse diameter of SSNs on lung window CT images was ±2.2 mm. For the

solid portion, it was ±3.7 mm. Inter-reader agreement (κ) of nodule type ranged from 0.80 to

0.96. Therefore, we did not re-evaluate the measurement variability or inter-reader agreement

of nodule type in this study.

Statistical analysis

Page 8 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



To investigate whether the variables incorporated in the established models (Brock

and Lee models) were significantly different between preinvasive (AAH and AIS) and

invasive lesions (MIA and IPA), we first performed univariate analysis. Categorical variables

were analyzed using the Pearson Chi-square test or Fisher’s exact test and continuous

variables were analyzed using the independent t-test or Mann-Whitney U test, as appropriate.

We then calculated the predicted probability from each logistic regression model. For

the Brock model, a full model with spiculation was used with input variables of age, sex,

family history of lung cancer, emphysema, nodule size, nodule type, nodule location, nodule

count per scan, and spiculation.16

Nodule size was subjected to power transformation prior to

entry as described previously.16

Age and nodule count were centered at a mean of 62 years

and 4, respectively.16

We recorded the predicted probability of each nodule, which was a

continuous value from 0 to 1 (0 to 100%). For the Lee model, two different methods were

used for analysis. For pGGNs, a single cutoff of nodule size (≥10 mm) was used to

discriminate invasive lesions as stated by Lee et al.7 In the case of PSNs, four variables

(nodule size, solid proportion, lobulation, and spiculation) were substituted into the

regression formula.7 Therefore, predicted probability was obtained only for PSNs in terms of

the Lee model. No preprocessing of variables was performed.

With the predicted probability obtained through each model, receiver operating

characteristic curve (ROC) analysis was performed to investigate the discriminative

performance of the prediction models in diagnosing invasive lesions. Areas under the ROC

curve (AUCs) were obtained and an optimal cutoff value based on the Youden index was

recorded. We calculated the sensitivity, specificity, accuracy, positive predictive value (PPV),

and negative predictive value (NPV) of each model with the optimal cutoff. For the Brock

model, two different cutoffs were applied to the calculation: 1) a threshold of 10% risk of

Page 9 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



malignancy as suggested by the BTS21

and 2) an optimal cutoff based on the Youden index.

AUCs were compared between the models based on DeLong’s method.24

As the

predicted probability of pGGNs was not available for the Lee model, AUC comparison was

conducted only for PSNs. Diagnostic accuracy was also compared between the models using

the McNemar test. All statistical analyses were performed using two commercial software

programs (MedCalc version 12.3.0, MedCalc Software, Mariakerke, Belgium; and SPSS 19.0,

IBM SPSS Statistics, Armonk, NY, USA). A P-value <0.05 was considered to indicate

statistical significance.

Page 10 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



RESULTS

Pathologic diagnoses of pGGNs and PSNs

Among 106 pGGNs, 44 were preinvasive and 62 were invasive lesions. As for the

324 PSNs, 27 were preinvasive and 297 were invasive lesions.

Comparisons between preinvasive and invasive lesions

For pGGNs, a family history of lung cancer was more frequently observed in patients

with invasive lesions (6/62) than in those with preinvasive lesions (0/44; P=0.040). Invasive

lesions (14.1±5.3 mm) were also significantly larger than preinvasive lesions (10.8±4.1 mm;

P=0.001). In addition, patients with invasive lesions had a smaller nodule count per scan

(invasive vs. preinvasive lesions: median, 2 vs. 4 nodules per scan; P=0.008). There were no

significant differences in age, sex, presence of emphysema, nodule location, lobulation, and

spiculation (table 2).

Table 2 Comparison of clinical and radiological characteristics between differing pathologic

diagnoses in patients with pure GGNs

Characteristics AAH and AIS

(n=44)

MIA and IPA

(n=62)

P-value

Age (year)a 56.2±10.3 57.8±10.4 0.434

Sex Male 18 (40.9) 31 (50.0) 0.430

Female 26 (59.1) 31 (50.0)

Family history of lung

cancer

Yes 0 (0) 6 (9.7) 0.040

No 44 (100) 56 (90.3)

Emphysema Yes 5 (11.4) 5 (8.1) 0.738

Page 11 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



No 39 (88.6) 57 (91.9)

Nodule size (mm)a 10.8±4.1 14.1±5.3 0.001

Location Upper lobe 26 (59.1) 33 (53.2) 0.560

Other lobes 18 (40.9) 29 (46.8)

Lobulation Yes 3 (6.8) 12 (19.4) 0.091

No 41 (93.2) 50 (80.6)

Spiculation Yes 1 (2.3) 7 (11.3) 0.136

No 43 (97.7) 55 (88.7)

Nodule count per scanb 4 (2, 7) 2 (1, 4) 0.008


parentheses).





For PSNs, nodule size, solid portion size, and solid proportion were significantly

larger in invasive lesions (invasive vs. preinvasive lesions: mean nodule size, 18.1±5.8 mm vs.

13.5±4.9 mm, P<0.001; median solid portion size, 8.4 mm vs. 4.7 mm, P<0.001; mean solid

proportion, 51.6±20.3% vs. 42.2±18.6%, P=0.018). Lobulation and spiculation were more

frequently observed in invasive lesions (invasive vs. preinvasive lesions: lobulation, 123/297

vs. 5/27, P=0.023; spiculation, 128/297 vs. 4/27, P=0.004). There were no significant

differences in age, sex, family history of lung cancer, presence of emphysema, nodule

location, and nodule count per scan (table 3).


diagnoses in patients with part-solid nodules

Page 12 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




parentheses).





SSN risk stratification using the Brock and Lee models


(n=27)

MIA and IPA

(n=297)

P-value

Age (year)a 59.6±9.4 62.6±8.9 0.125

Sex Male 7 (25.9) 127 (42.8) 0.104

Female 20 (74.1) 170 (57.2)


cancer

Yes 1 (3.7) 10 (3.4) 1.000

No 26 (96.3) 287 (96.6)

Emphysema Yes 2 (7.4) 32 (10.8) 0.753

No 25 (92.6) 265 (89.2)

Nodule size (mm)a 13.5±4.9 18.1±5.8 <0.001

Solid portion size

(mm)b

4.7 (3.5, 5.5) 8.4 (5.1, 13.6) <0.001

Solid proportion (%)a 42.2±18.6 51.6±20.3 0.018


Other lobes 15 (55.6) 109 (36.7)

Lobulation Yes 5 (18.5) 123 (41.4) 0.023

No 22 (81.5) 174 (58.6)

Spiculation Yes 4 (14.8) 128 (43.1) 0.004

No 23 (85.2) 169 (56.9)

Nodule count per scanb 3 (2,8) 3 (1,5) 0.207

Page 13 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



For pGGNs, the AUC of the Brock model’s predicted probability for differentiating

invasive lesions from preinvasive lesions was 0.680 [95% confidence interval (CI): 0.582,

0.767] (table 4). A cutoff of 10%, suggested by the BTS, yielded a sensitivity, specificity,

accuracy, PPV, and NPV of 32.3%, 90.9%, 56.6%, 83.3%, and 48.8%, respectively. Another

cutoff of 4.29%, an optimal threshold based on the Youden index, provided a sensitivity,

specificity, accuracy, PPV, and NPV of 62.9%, 68.2%, 65.1%, 73.6%, and 56.6%. A nodule

size cutoff (10 mm) suggested by Lee et al.7 was also applied to our study population. The

resultant sensitivity, specificity, accuracy, PPV, and NPV were 75.8%, 45.5%, 63.2%, 66.2%,

and 57.1%, respectively. There were no significant differences in diagnostic accuracy

between the Brock model and Lee criteria (Brock model cutoff 10% vs. nodule size cutoff

10mm, P=0.382; Brock model cutoff 4.29% vs. nodule size cutoff 10 mm, P=0.832).

Table 4 Performance of each prediction model in diagnosing minimally invasive

adenocarcinoma and invasive adenocarcinoma for pure ground-glass nodules

Note.— Data are in percentages except for AUC.

aA cutoff of 10% predicted probability was used as suggested by the British Thoracic Society.

bA cutoff of 4.29% predicted probability, the optimal threshold based on the Youden index,

was used.

cA cutoff of 10 mm nodule size was used.

AUC, area under the receiver operating characteristic curve; NPV, negative predictive value;

Sensitivity Specificity Accuracy PPV NPV AUC

Brock model1a 32.3 90.9 56.6 83.3 48.8 0.680

(0.582, 0.767) Brock model2b 62.9 68.2 65.1 73.6 56.6

Lee model

(Nodule sizec)

75.8 45.5 63.2 66.2 57.1 -

Page 14 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



PPV, positive predictive value

As for PSNs, the AUC of the Brock model was 0.748 (95% CI: 0.697, 0.795) for the

discrimination of invasive lesions from preinvasive lesions (table 5). A cutoff of 10% yielded

a sensitivity, specificity, accuracy, PPV, and NPV of 81.5%, 55.6%, 79.3%, 95.3%, and

21.4%. The optimal cutoff based on the Youden index was 10.11%, which was very close to

the suggested threshold by the BTS. Therefore, performance metrics were not calculated

separately. AUC of the Lee model was 0.769 (95% CI: 0.720, 0.814), and an optimal cutoff of

66.68% provided a sensitivity, specificity, accuracy, PPV, and NPV of 76.4%, 74.1%, 76.2%,

97.0%, and 22.2%, respectively. AUCs and diagnostic accuracies were not significantly

different between the two models (P=0.623 and 0.237, respectively).


adenocarcinoma and invasive adenocarcinoma for part-solid nodules

Note.— Data are in percentages except for AUC.

aA cutoff of 10% predicted probability was used, which was suggested by the British

Thoracic Society and was optimal at the same time.

bAn optimal cutoff of 66.68% was adopted based on the Youden index.




Brock modela 81.5 55.6 79.3 95.3 21.4 0.748

(0.697, 0.795)

Lee modelb 76.4 74.1 76.2 97.0 22.2 0.769

(0.720, 0.814)

Page 15 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



DISCUSSION

In this study, we revealed that AUCs for the differentiation of invasive lesions among

PSNs using the established risk prediction models ranged from 0.748 to 0.769 with no

significant differences between the two models. Diagnostic accuracies based on the optimal

cutoffs were 79.3% for the Brock model and 76.2% for the Lee model. For pGGNs, the

diagnostic accuracy was 56.6%-65.1% for the Brock model depending on the cutoff values

used and 63.2% for the Lee criteria.

McWilliams et al.16

originally developed a lung cancer prediction model (Brock

model) using participants enrolled in a lung cancer screening study and chemoprevention

trials. Thus, the Brock model initially targeted pulmonary nodules detected on first screening

CT. Incidentally detected nodules as well as surgical candidates were not the original target

lesions of this model. However, at present, the BTS recommends using the same diagnostic

approach for nodules detected incidentally as those detected through screening.21

BTS also

recommends using the Brock model for the risk calculation of both solid and subsolid

nodules.21

A cutoff of 10% predicted probability for malignancy is suggested in order to

differentiate high risk SSNs for the performance of biopsy or surgical resection.21

This

quantitative diagnostic approach is to discern malignant SSNs with an appropriate false

positive rate. However, it must be noted that most persistent SSNs belong to one of the four

categories of the adenocarcinoma spectrum: AAH, AIS, MIA and IPA. Therefore, the

potential of the risk prediction model in discriminating lesions with invasive components

(MIA and IPA) should also be tested using pathologic diagnosis as a reference standard.

Indeed, as clinical management strategies differ substantially between preinvasive and

invasive lesions, if it can be feasible to predict the invasiveness of SSNs, clinical planning

Page 16 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



whether to perform annual CT surveillance, limited resection, or conventional lobectomy can

be facilitated.

The performance of the prediction models for the risk stratification of SSNs was not

optimal according to our study results. For PSNs, AUCs of the two models ranged between

0.748 and 0.769 with diagnostic accuracies close to 80%. Past research on distinguishing

invasive adenocarcinomas appearing as SSNs have reported that logistic regression models

built with size metrics, morphologic features, or texture features showed AUCs ranging from

0.79 to 0.98.4-7 9 11 13

However, these models were not tested for an independent cohort or

validated externally. Therefore, their true performance may be lower than that reported in the

literature.

Another important finding of our study was that the PPV for the differentiation of

invasive lesions among PSNs was very high for both models, over 95%. In other words, the

probability of being an invasive lesion was over 95% for nodules predicted as being invasive

through these models. A concern, however, is the high false negative rate of these models.

PSNs predicted as preinvasive lesions, which have a low calculated risk, should be managed

according to their solid portion size.25

PSNs with solid components <6 mm require

surveillance CT annually, while those with solid components ≥6 mm should be biopsied or

surgically resected in consideration of invasive adenocarcinoma.

The diagnostic accuracies of both the Brock model and Lee criteria were even lower

for pGGNs. Among multiple clinical and radiologic characteristics investigated in our study,

only three variables (family history of lung cancer, nodule size, and nodule count per scan)

were significantly different between preinvasive and invasive lesions. This implies the need

for other useful features for the development of new better prediction models. Features such

as nodule volume, mass, or radiomic features may provide additional clues for their

Page 17 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



differentiation.4 26

In addition, changes in nodule characteristics at follow-up CT scans, such

as an increase in nodule size, attenuation, or new development of a solid portion, may also be

valuable for the discrimination.14

Alternatively, computational classification analysis

including deep learning algorithms, which do not require hand-crafted features and can be

self-trained directly from raw image pixels, may be another solution for the diagnosis of

pGGNs.27

The Brock model has been externally validated for the cohorts of the Danish Lung

Cancer Screening Trial (DLCST)19

and National Lung Screening Trial (NLST);18

AUCs for

the discrimination of malignant from benign nodules ranged from 0.834 for the former and

0.963 for the latter. AUCs for the validation cohort of the original paper, the British Columbia

Cancer Agency (BCCA) chemoprevention trial cohort, was 0.970.16

In addition, for an

Australian lung cancer screening cohort, Zhao et al.20

tested the utility of the Brock model for

the baseline evaluation of 52 SSNs and demonstrated that the AUC was 0.89. To the contrary,

however, the model performance evaluated in our study was lower than those reported in the

literature. The main reason for such a discrepancy may be that we included patients who

underwent surgical resection of SSNs unlike previous studies. Thus, the proportion of

preinvasive lesions was small (16.5%) and a major portion of our study population consisted

of invasive lesions (83.5%). Such high prevalence of invasive lesions would have affected

our study results. Nevertheless, SSNs of interest in daily clinical practice may be closer to

those in our study. In routine practice, transient SSNs, which are definitely benign, do not

require risk calculation as they are easily confirmed through follow-up CT scans at short-term

intervals.25

In addition, small SSNs <6 mm are usually preinvasive and do not require CT

surveillance. On the other hand, particular concern should be given to persistent SSNs ≥6 mm,

especially to those with solid components. As the role of biopsy or positron emission

Page 18 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



tomography is limited for SSNs,15

we supposed that risk prediction models may provide

value for more appropriate management planning. In this context, we applied prediction

models to surgically resected SSNs for the validation of their clinical utility.

There were several limitations to our study. First, our study was not conducted for a

screening cohort as described earlier in this manuscript. Second, optimal cutoffs for the

models were not obtained from ROC analyses of the original study populations from which

the models were derived. Therefore, performance metrics including sensitivity, specificity,

accuracy, PPV and NPV may have been overestimated in our study. Third, radiologic nodule

information was extracted from our heterogeneous CT dataset, in which CT acquisition

parameters such as radiation dosage, slice thickness or contrast-enhancement were not

uniform across the study population. However, these factors would have had little effect on

the variables we used. In addition, all CT scans had thin-section images (slice thickness ≤1.5

mm). Fourth, nodule size as well as solid portion size were measured as the longest

transverse diameter in accordance with the definition of lesion size and solid proportion in the

original papers. However, recent analyses have revealed that the usage of average diameter as

an input variable may enhance the model performance.28 29

In conclusion, the performance of the Brock model and Lee model for the

differentiation of invasive lesions among SSNs was suboptimal. In particular, both models

showed lower performance for pGGNs compared to that for PSNs. Thus, an alternative

approach such as computer-aided classification should be developed for the pre-operative

diagnosis of invasive lesions among SSNs.

Page 19 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Acknowledgements

We would like to thank Chris Woo, B.A., for editorial assistance.

Contributors

HK and CMP contributed to conception and design; HK, SJ, JHL, SYA, REY, HL, JP,

WHL, EJH, SML and JMG contributed to acquisition of data, or analysis and interpretation

of data; HK, CMP and JMG were involved in drafting the manuscript or revising it critically

for important intellectual content; all authors gave approval of the final version of the

manuscript.

Funding

This study was supported by Basic Science Research Program through the National

Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future

Planning (grant number: 2017R1A2B4008517).

Competing interests

None declared.

Patient consent

The requirement for written informed consent was waived by the Institutional

Review Board.

Ethics approval

Institutional Review Board of Seoul National University Hospital.

Page 20 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Data sharing statement

All data are available from the corresponding author.

Page 21 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



REFERENCES

1. Austin JH, Garg K, Aberle D, et al. Radiologic implications of the 2011 classification of

adenocarcinoma of the lung. Radiology 2013;266:62-71.

2. Yankelevitz DF, Yip R, Smith JP, et al. CT screening for lung cancer: nonsolid nodules in

baseline and annual repeat rounds. Radiology 2015;277:555-64.

3. Scholten ET, de Jong PA, de Hoop B, et al. Towards a close computed tomography

monitoring approach for screen detected subsolid pulmonary nodules? Eur Respir J

2015;45:765-73.

4. Chae HD, Park CM, Park SJ, et al. Computerized texture analysis of persistent part-solid

ground-glass nodules: differentiation of preinvasive lesions from invasive pulmonary

adenocarcinomas. Radiology 2014;273:285-93.

5. Ding H, Shi J, Zhou X, et al. Value of CT characteristics in predicting invasiveness of

adenocarcinoma presented as pulmonary ground-glass nodules. Thorac Cardiovasc

Surg 2017;65:136-41.

6. Jin C, Cao J, Cai Y, et al. A nomogram for predicting the risk of invasive pulmonary

adenocarcinoma for patients with solitary peripheral subsolid nodules. J Thorac

Cardiovasc Surg 2017;153:462-69 e1.

7. Lee SM, Park CM, Goo JM, et al. Invasive pulmonary adenocarcinomas versus preinvasive

lesions appearing as ground-glass nodules: differentiation by using CT features.

Radiology 2013;268:265-73.

8. Li Q, Fan L, Cao ET, et al. Quantitative CT analysis of pulmonary pure ground-glass

nodule predicts histological invasiveness. Eur J Radiol 2017;89:67-71.

9. Liang J, Xu XQ, Xu H, et al. Using the CT features to differentiate invasive pulmonary

adenocarcinoma from pre-invasive lesion appearing as pure or mixed ground-glass

Page 22 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



nodules. Br J Radiol 2015;88:20140811.

10. Moon Y, Sung SW, Lee KY, et al. Pure ground-glass opacity on chest computed

tomography: predictive factors for invasive adenocarcinoma. J Thorac Dis

2016;8:1561-70.

11. Son JY, Lee HY, Kim JH, et al. Quantitative CT analysis of pulmonary ground-glass

opacity nodules for distinguishing invasive adenocarcinoma from non-invasive or

minimally invasive adenocarcinoma: the added value of using iodine mapping. Eur

Radiol 2016;26:43-54.

12. Yanagawa M, Johkoh T, Noguchi M, et al. Radiological prediction of tumor invasiveness

of lung adenocarcinoma on thin-section CT. Medicine (Baltimore) 2017;96:e6331.

13. Zhang Y, Shen Y, Qiang JW, et al. HRCT features distinguishing pre-invasive from

invasive pulmonary adenocarcinomas appearing as ground-glass nodules. Eur Radiol

2016;26:2921-8.

14. Kim H, Park CM, Koh JM, et al. Pulmonary subsolid nodules: what radiologists need to

know about the imaging features and management strategy. Diagn Interv Radiol

2014;20:47-57.

15. Naidich DP, Bankier AA, MacMahon H, et al. Recommendations for the management of

subsolid pulmonary nodules detected at CT: a statement from the Fleischner society.

Radiology 2013;266:304-17.

16. McWilliams A, Tammemagi MC, Mayo JR, et al. Probability of cancer in pulmonary

nodules detected on first screening CT. N Engl J Med 2013;369:910-9.

17. Al-Ameri A, Malhotra P, Thygesen H, et al. Risk of malignancy in pulmonary nodules: a

validation study of four prediction models. Lung Cancer 2015;89:27-30.

18. White CS, Dharaiya E, Campbell E, et al. The Vancouver lung cancer risk prediction

model: assessment by using a subset of the national lung screening trial cohort.

Page 23 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Radiology 2017;283:264-72.

19. Winkler Wille MM, van Riel SJ, Saghir Z, et al. Predictive accuracy of the PanCan lung

cancer risk prediction model -external validation based on CT from the Danish lung

cancer screening trial. Eur Radiol 2015;25:3093-9.

20. Zhao H, Marshall HM, Yang IA, et al. Screen-detected subsolid pulmonary nodules: long-

term follow-up and application of the PanCan lung cancer risk prediction model. Br J

Radiol 2016;89:20160016.

21. Callister ME, Baldwin DR, Akram AR, et al. British Thoracic Society guidelines for the

investigation and management of pulmonary nodules. Thorax 2015;70 Suppl 2:ii1-

ii54.

22. Travis WD, Brambilla E, Noguchi M, et al. International association for the study of lung

cancer/American thoracic society/European respiratory society international

multidisciplinary classification of lung adenocarcinoma. J Thorac Oncol 2011;6:244-

85.

23. Kim H, Park CM, Song YS, et al. Measurement variability of persistent pulmonary

subsolid nodules on same-day repeat CT: what is the threshold to determine true

nodule growth during follow-up? PLoS One 2016;11:e0148853.

24. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more

correlated receiver operating characteristic curves: a nonparametric approach.

Biometrics 1988;44:837-45.

25. MacMahon H, Naidich DP, Goo JM, et al. Guidelines for management of incidental

pulmonary nodules detected on CT images: from the Fleischner society 2017.

Radiology 2017;284:228-43.

26. Song YS, Park CM, Park SJ, et al. Volume and mass doubling times of persistent

pulmonary subsolid nodules detected in patients without known malignancy.

Page 24 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Radiology 2014;273:276-84.

27. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer

with deep neural networks. Nature 2017;542:115-18.

28. Kovalchik SA, Tammemagi M, Berg CD, et al. Targeting of low-dose CT screening

according to the risk of lung-cancer death. N Engl J Med 2013;369:245-54.

29. van Riel SJ, Ciompi F, Jacobs C, et al. Malignancy risk estimation of screen-detected

nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN

guidelines. Eur RadiolPublished Online First: 14 March 2017. doi:10.1007/s00330-

017-4767-2

Page 25 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



FIGURE LEGENDS

Figure 1 Flow chart of patient inclusion and exclusion.

pGGN, pure ground-glass nodule; PSN, part-solid nodule; SSN, subsolid nodule

Page 26 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Flow chart of patient inclusion and exclusion.

127x158mm (600 x 600 DPI)

Page 27 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



CT SCANNING PROTOCOLS FOR THE STUDY POPULATION

Patients underwent CT scans using seven different scanners from four vendors. CT scans in our study were performed with the

following scanning protocols.

Vendor Toshibaa Siemens

b Siemens

b GE

c GE

c Philips

d Philips

d

Machine Aquilion One Definition Sensation

16

LightSpeed

Ultra

Discovery

CT750 HD

Brilliance 64 Ingenuity

kVp 120 120 120 120 120 120 120

mAs Target standard

deviation of

noise, 14

Quality

reference

mAs, 150

Quality

reference

mAs, 100

Noise index,

16.36

Noise

index, 22.67

Reference

mAs, 200

Reference

mAs, 170

Automatic

exposure

control

Sure Exposure CARE

Dose 4D

CARE

Dose 4D

Auto mA Auto mA Doseright

ACS, Z

DOM

Doseright

ACS, Z

DOM

Rotation time

(sec)

0.5 0.5 0.5 0.5 0.5 0.5 0.5

Collimation

(mm)

0.5x100 0.6x64 0.75x16 1.25x8 0.625x64 0.625x64 0.625x64

Slice thickness

(mm)

1 1 1 1 1.25 1 1

Pitch HP 103 1 1 0.875 0.984 1.014 1.0015

Page 28 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Kernel FC 15-H B60f B60f Bone Chest YC 0 YC 0

Increment 1 1 1 1 1.25 1 1

Matrix 512x512 512x512 512x512 512x512 512x512 512x512 512x512

* All patients were scanned in supine position from the lung apex to the base during suspended maximum inspiration.

* For contrast enhancement, a total of 70-90 mL of 370 mgI/mL of the contrast material, iopamidol (Pamiray 370; Dongkook Pharmaceutical,

Seoul, Korea) or iopromide (Ultravist 370; Schering, Berlin, Germany), was injected at a rate of 2.0-2.3mL/sec using a power injector. CT

scanning was performed with a 60-second delay.

aToshiba Medical Systems, Otawara, Japan

bSiemens Healthcare, Forchheim, Germany

cGE Healthcare, Waukesha, WI, USA

dPhilips Healthcare, Cleveland, OH, USA

Page 29 of 29


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Validation of prediction models for risk stratification of

incidentally detected pulmonary subsolid nodules: a

retrospective cohort study in a Korean tertiary medical

center

Journal: BMJ Open

Manuscript ID bmjopen-2017-019996.R1


Date Submitted by the Author: 03-Jan-2018

Complete List of Authors: Kim, Hyungjin; Seoul National University Hospital Park, Chang Min; Department of Radiology Jeon, Sun Kyung; Seoul National University Hospital Lee, Jong Hyuk; Seoul National University Hospital Ahn, Su Yeon; Seoul National University Hospital Yoo, Roh-Eul; Seoul National University Hospital Lim, Hyun-ju; National Cancer center Park, Juil; Seoul National University Hospital Lim, Woo Hyeon; Seoul National University Hospital Hwang, Eui Jin; Seoul National University Hospital

Lee, Sang Min; University of Ulsan College of Medicine Goo, Jin Mo; Seoul National University Hospital


Oncology


Keywords: subsolid nodule, risk prediction model, Brock model, validation, lung cancer


BMJ Open on F


http://bmjopen.bm

j.com/

BM


jopen-2017-019996 on 24 May 2018. D

ownloaded from






center



1, Jong Hyuk Lee

1, Su Yeon Ahn

1, Roh-


1, 4, Juil Park

1, Woo Hyeon Lim

1, Eui Jin Hwang

1, Sang Min Lee

5, Jin

Mo Goo1, 2, 3



Korea








Page 1 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Word count: 3281

Page 2 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



ABSTRACT

Objectives: We aimed to validate the performances of two prediction models (Brock and Lee

models) for the differentiation of minimally-invasive and invasive pulmonary

adenocarcinomas (MIA/IPA) from preinvasive lesions among subsolid nodules (SSNs). The

purpose of our study was to evaluate the feasibility of the two models for the risk




Participants: 410 patients with 410 incidentally detected SSNs who underwent surgical

resection for the pulmonary adenocarcinoma spectrum between 2011 and 2015.


predicted probability of MIA/IPA was calculated from pre-existing logistic regression models

(Brock and Lee models). Areas under the receiver operating characteristic curve (AUCs)

were calculated and compared between models. Performance metrics including sensitivity,


were also obtained.

Results: For pure ground-glass nodules (n=101), the AUC of the Brock model in

differentiating MIA/IPA (59/101) from preinvasive lesions (42/101) was 0.671. Sensitivity,

specificity, accuracy, PPV, and NPV based on the optimal cutoff value were 64.4%, 64.3%,

64.4%, 71.7%, and 56.3%, respectively. Sensitivity, specificity, accuracy, PPV, and NPV

according to the Lee criteria were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%, respectively. For

part-solid nodules (n=309; 26 preinvasive lesions and 283 MIA /IPAs), the AUC was 0.746

Page 3 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



for the Brock model and 0.771 for the Lee model (P=0.574). Sensitivity, specificity, accuracy,

PPV, and NPV were 82.3%, 53.8%, 79.9%, 95.1%, and 21.9%, respectively, for the Brock

model and 77.0%, 69.2%, 76.4%, 96.5%, and 21.7%, respectively, for the Lee model.

Conclusions: The performance of prediction models for the incidentally detected SSNs in

differentiating MIA/IPA from preinvasive lesions might be suboptimal. Thus, an alternative

risk calculation model is required for the incidentally detected SSNs.


� This is the first study to externally validate the performance of pre-existing risk prediction


� This study performed head-to-head comparisons between the prediction models for the risk

stratification of subsolid nodules.

� The main limitation of this study is that it only analyzed surgically resected lung nodules,


� Study population was small to conduct separate analyses for the pure ground-glass nodules

and part-solid nodules.

Page 4 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



INTRODUCTION



(AAH) and adenocarcinoma-in-situ (AIS).1 SSNs are common findings at chest computed


Indeed,






as invasive


dissection14





clinicians to date.


McWilliams et al.16




and was





Page 5 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from










(Brock and Lee models) for the differentiation of minimally invasive adenocarcinomas

(MIAs) and invasive pulmonary adenocarcinomas (IPAs) from preinvasive lesions among

SSNs. The purpose of our study was to evaluate the feasibility of the two models in the risk


Page 6 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



METHODS




Study population











three nodules. A single nodule was selected randomly for these 19 patients in order to remove

within-subject correlation. Therefore, a total of 410 nodules from 410 patients were analyzed

in the present study (figure 1). There were 174 men and 236 women (median, 61 years;

interquartile range: 54, 69 years). As for the nodule type, there were 101 pGGNs and 309

PSNs. IPAs were found in 290 nodules followed by MIA in 52 nodules, AIS in 51 nodules,

and AAH in 17 nodules. Median nodule size was 15.8 mm (interquartile range: 11.8, 20.9

mm) (table 1).


Page 7 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Age (year)a 61 (54, 69)

Sex Male 174 (42.4)

Female 236 (57.6)


AIS 51 (12.4)

MIA 52 (12.7)

IPA 290 (70.7)




No 394 (96.1)


No 366 (89.3)

Nodule size (mm)a 15.8 (11.8, 20.9)

Solid portion size (mm)a 5.7 (1.6, 11.2)

Solid proportion (%)a 41.0 (11.4, 62.0)




No 272 (66.3)


No 276 (67.3)

Nodule count per scana 3 (1, 5)


parentheses).

aData are median (with interquartile range in parentheses).



Page 8 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Data collection








and


















Page 9 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from












Regression coefficients and the model constant were

available from the original paper.16

Nodule size was subjected to power transformation prior

to entry as described previously.16







(nodule size, solid proportion, lobulation, and spiculation) were substituted into the following

regression formula.7

��

�� !" � ��#�#��$� % &��!� �

�� % ��"" � ��#�'��

Therefore, predicted probability was obtained only for PSNs in terms of the Lee model. No

preprocessing of variables was performed.


Page 10 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from











ROC analysis was performed for each nodule type separately and then for the entire SSNs in

the case of the Brock model. In terms of the Lee model, ROC analysis was performed only

for PSNs.


As the



the McNemar test.

Lastly, calibration of the models was assessed using the Hosmer-Lemeshow test for

the 10 probability groups (deciles). All statistical analyses were performed using two

commercial software programs (MedCalc version 12.3.0, MedCalc Software, Mariakerke,

Belgium; and SPSS 19.0, IBM SPSS Statistics, Armonk, NY, USA) and R software, version

3.1.0 (http://www.R-project.org; PredictABEL package). A P-value <0.05 was considered to

indicate statistical significance.

Page 11 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



RESULTS















(n=42)

MIA and IPA

(n=59)

P-value

Age (year)a 57±10 57±10 0.839

Sex Male 16 (38.1) 29 (49.2) 0.270

Female 26 (61.9) 30 (50.8)


cancer

Yes 0 (0) 6 (10.2) 0.040

No 42 (100) 53 (89.8)

Emphysema Yes 5 (11.9) 5 (8.5) 0.738

Page 12 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



No 37 (88.1) 54 (91.5)

Nodule size (mm)a 11.1±4.1 14.2±5.4 0.002


Other lobes 17 (40.5) 28 (47.5)

Lobulation Yes 3 (7.1) 12 (20.3) 0.090

No 39 (92.9) 47 (79.7)

Spiculation Yes 1 (2.4) 7 (11.9) 0.135

No 41 (97.6) 52 (88.1)



parentheses).






larger in invasive lesions (invasive vs. preinvasive lesions: median nodule size, 17.6 mm vs.

13.6 mm, P<0.001; median solid portion size, 8.4 mm vs. 4.6 mm, P<0.001; median solid

proportion, 52.8% vs. 36.8%, P=0.032). Lobulation and spiculation were more frequently

observed in invasive lesions (invasive vs. preinvasive lesions: lobulation, 118/283 vs. 5/26,

P=0.025; spiculation, 122/283 vs. 4/26, P=0.006). There were no significant differences in

age, sex, family history of lung cancer, presence of emphysema, nodule location, and nodule

count per scan (table 3).



Page 13 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Note.

—

Unless

other

wise

specifi

ed,

data

are

numbe

rs of

patient

s

(with

percen

tages

in

parent

heses).






(n=26)

MIA and IPA

(n=283)

P-value

Age (year)a 62 (54, 68) 63 (56, 70) 0.257

Sex Male 7 (26.9) 122 (43.1) 0.109

Female 19 (73.1) 161 (56.9)


cancer

Yes 1 (3.8) 9 (3.2) 0.590

No 25 (96.2) 274 (96.8)

Emphysema Yes 2 (7.7) 32 (11.3) 0.752

No 24 (92.3) 251 (88.7)

Nodule size (mm)a 13.6 (9.8,

16.6)

17.6 (13.3,

22.4)

<0.001

Solid portion size

(mm)a

4.6 (3.4, 5.8) 8.4 (5.2, 13.6) <0.001

Solid proportion (%)a 36.8 (25.1,

63.2)

52.8 (33.4,

66.0)

0.032


Other lobes 14 (53.8) 102 (36.0)

Lobulation Yes 5 (19.2) 118 (41.7) 0.025

No 21 (80.8) 165 (58.3)

Spiculation Yes 4 (15.4) 122 (43.1) 0.006

No 22 (84.6) 161 (56.9)

Nodule count per scana 3 (2,8) 3 (1,4) 0.132

Page 14 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from








specificity, accuracy, PPV, and NPV of 64.4%, 64.3%, 64.4%, 71.7%, and 56.3%. Brock

model for pGGNs showed poor calibration (P<0.001). A nodule size cutoff (10 mm)

suggested by Lee et al.7 was also applied to our study population. The resultant sensitivity,

specificity, accuracy, PPV, and NPV were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%,

respectively. There were no significant differences in diagnostic accuracy between the Brock

model and Lee criteria (Brock model cutoff 10% vs. nodule size cutoff 10mm, P=0.461;

Brock model cutoff 4.29% vs. nodule size cutoff 10 mm, P=0.832).



Note.— Data are in percentages except for AUC. There were 42 preinvasive lesions and 59

invasive lesions (minimally invasive adenocarcinomas and invasive adenocarcinomas).



was used.


Brock model1a 32.2 90.5 56.4 82.6 48.7 0.671

(0.571, 0.762) Brock model2b 64.4 64.3 64.4 71.7 56.3

Lee model

(Nodule sizec)

76.3 42.9 62.4 65.2 56.3 -

Page 15 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from














different between the two models (P=0.574 and 0.169, respectively). In addition, both models

exhibited poor calibration (P<0.001).








Brock modela 82.3 53.8 79.9 95.1 21.9 0.746

(0.694, 0.794)

Lee modelb 77.0 69.2 76.4 96.5 21.7 0.771

(0.720, 0.817)

Page 16 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from






With respect to the pooled analysis for the entire SSNs, the AUC of Brock model was

0.810 (95% CI: 0.769, 0.846). Calibration was also poor in this model (P<0.001).

Page 17 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



DISCUSSION






used and 62.4% for the Lee criteria. For the entire SSNs, the Brock model showed AUC of

0.810.

McWilliams et al.16


model) using participants enrolled in a lung cancer screening study. Thus, the Brock model

initially targeted pulmonary nodules detected on first screening CT. Incidentally detected

nodules as well as surgical candidates were not the original target lesions of this model.

However, at present, the BTS recommends using the same diagnostic approach for nodules

detected incidentally as those detected through screening.21

BTS also recommends using the

Brock model for the risk calculation of both solid and subsolid nodules.21

A cutoff of 10%

predicted probability for malignancy is suggested in order to differentiate high risk SSNs for

the performance of biopsy or surgical resection.21

This quantitative diagnostic approach is to

discern malignant SSNs with an appropriate false positive rate. However, it must be noted

that most persistent SSNs belong to one of the four categories of the adenocarcinoma

spectrum: AAH, AIS, MIA and IPA. Therefore, the potential of the risk prediction model in

discriminating lesions with invasive components (MIA and IPA) should also be tested using

pathologic diagnosis as a reference standard. Indeed, as clinical management strategies differ

substantially between preinvasive and invasive lesions, if it can be feasible to predict the

Page 18 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



invasiveness of SSNs, clinical planning whether to perform annual CT surveillance, limited

resection, or conventional lobectomy can be facilitated.



0.746 and 0.771 with diagnostic accuracies close to 80%. The performance of the prediction

model for PSNs in the original paper by Lee et al.7 was 0.905 (AUC). The study population

of the present study was similar to that of the study by Lee et al.7 An important reason for the

performance drop would be the spectrum effect, which is a common cause of model

performance heterogeneity.25

A variation in the assessment of CT morphologic features

(lobulation and spiculation) would be another potential cause. Past research on distinguishing



0.79 to 0.98.4-6 9 11 13


validated externally.






according to their solid portion size, if they are persistent lesions.26

A few studies have shown

that the solid portions in PSNs are well correlated to the pathologic invasive component.12 27

28 Fleischner Society guideline recommends that PSNs with solid components ≥6 mm be

monitored with CT scans at 3-6 months interval.26

PSNs with solid portions larger than 8 mm

should be biopsied or surgically resected in consideration of invasive adenocarcinomas.26

Page 19 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



BTS also recommends that the solid component size be considered to further refine the

estimate of malignancy risk.21

In addition, growing solid component is also a sign of an

invasive adenocarcinoma as described in both guidelines.21 26














pGGNs.30




AUCs for




In addition, for an






Page 20 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from









intervals.26









screening cohort as described earlier in this manuscript. The prevalence of preinvasive lesions

was low compared to that of the screening setting. Thus, the performance measures in this

study should be carefully interpreted with respect to the target population, which were the

incidentally detected surgical candidates. Second, our retrospective study included a small

number of patients and analyses were conducted separately for pGGNs and PSNs. Separate

analysis of pGGNs and PSNs has resulted in a slight underestimation of the performance of

Brock model. Third, optimal cutoffs for the models were not obtained from ROC analyses of

the original study populations from which the models were derived. Fourth, radiologic nodule




Page 21 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




mm). Fifth, nodule size as well as solid portion size were measured as the longest transverse

diameter in accordance with the definition of lesion size and solid proportion in the original

papers. However, recent analyses have revealed that the usage of average diameter as an

input variable may enhance the model performance.31






Page 22 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Acknowledgements


Contributors





manuscript.

Funding



Planning (grant number: 2017R1A2B4008517). Funder had no role in the study design; in the

collection, analysis and interpretation of the data; in the writing of the report; and in the

decision to submit the paper for publication.

Competing interests

None declared.

Patient consent


Review Board.

Page 23 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Ethics approval




Page 24 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



REFERENCES



2. Yankelevitz DF, Yip R, Smith JP, et al. CT Screening for lung cancer: nonsolid nodules in




2015;45:765-73.






Surg 2017;65:136-41.






Radiology 2013;268:265-73.





Page 25 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from






2016;8:1561-70.




Radiol 2016;26:43-54.





2016;26:2921-8.



2014;20:47-57.


subsolid pulmonary nodules detected at CT: a statement from the Fleischner Society.

Radiology 2013;266:304-17.






model: assessment by using a subset of the National Lung Screening Trial cohort.

Page 26 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Radiology 2017;283:264-72.


cancer risk prediction model -external validation based on CT from the Danish Lung

Cancer Screening Trial. Eur Radiol 2015;25:3093-9.



Radiol 2016;89:20160016.



ii54.




85.






Biometrics 1988;44:837-45.

25. Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using

big datasets from e-health records or IPD meta-analysis: opportunities and challenges.

BMJ 2016;353:i3140.


pulmonary nodules detected on CT images: from the Fleischner Society 2017.

Page 27 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Radiology 2017;284:228-43.

27. Ko JP, Suh J, Ibidapo O, et al. Lung adenocarcinoma: correlation of quantitative CT

findings with pathologic findings. Radiology 2016;280:931-9.

28. Lee KH, Goo JM, Park SJ, et al. Correlation between the size of the solid component on

thin-section CT and the invasive component on pathology in small lung

adenocarcinomas manifesting as ground-glass nodules. J Thorac Oncol 2014;9:74-82.



Radiology 2014;273:276-84.





guidelines. Eur Radiol 2017;27:4019-29.

Page 28 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



FIGURE LEGENDS



Page 29 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




127x157mm (600 x 600 DPI)

Page 30 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from







b Siemens

b GE

c GE

c Philips

d Philips

d


16

LightSpeed

Ultra

Discovery

CT750 HD


kVp 120 120 120 120 120 120 120

mAs Target standard

deviation of

noise, 14

Quality

reference

mAs, 150

Quality

reference

mAs, 100

Noise index,

16.36

Noise

index, 22.67

Reference

mAs, 200

Reference

mAs, 170

Automatic

exposure

control

Sure Exposure CARE

Dose 4D

CARE

Dose 4D


ACS, Z

DOM

Doseright

ACS, Z

DOM

Rotation time

(sec)

0.5 0.5 0.5 0.5 0.5 0.5 0.5

Collimation

(mm)

0.5x100 0.6x64 0.75x16 1.25x8 0.625x64 0.625x64 0.625x64

Slice thickness

(mm)

1 1 1 1 1.25 1 1

Pitch HP 103 1 1 0.875 0.984 1.014 1.0015

Page 31 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Increment 1 1 1 1 1.25 1 1

Matrix 512x512 512x512 512x512 512x512 512x512 512x512 512x512









Page 32 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



TRIPOD Checklist: Prediction Model Validation

Section/Topic Item Checklist Item Page

Title and abstract

Title 1 Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted.

1

Abstract 2 Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions.

3

Introduction

Background and objectives

3a Explain the medical context (including whether diagnostic or prognostic) and rationale for developing or validating the multivariable prediction model, including references to existing models.

5 (line 17), 6 (line 3)

3b Specify the objectives, including whether the study describes the development or validation of the model or both.

6 (line 7)

Methods

Source of data

4a Describe the study design or source of data (e.g., randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable.

7 (line 6-7)

4b Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up.

7 (line 7)

Participants

5a Specify key elements of the study setting (e.g., primary care, secondary care, general population) including number and location of centres.

2 (line 8), 9 (line 3)

5b Describe eligibility criteria for participants. 7

5c Give details of treatments received, if relevant. N/A

Outcome 6a

Clearly define the outcome that is predicted by the prediction model, including how and when assessed.

10 (line 2-3)

6b Report any actions to blind assessment of the outcome to be predicted. -

Predictors

7a Clearly define all predictors used in developing or validating the multivariable prediction model, including how and when they were measured.

9

7b Report any actions to blind assessment of predictors for the outcome and other predictors.

-

Sample size 8 Explain how the study size was arrived at. -

Missing data 9 Describe how missing data were handled (e.g., complete-case analysis, single imputation, multiple imputation) with details of any imputation method.

-

Statistical analysis methods

10c For validation, describe how the predictions were calculated. 10

10d Specify all measures used to assess model performance and, if relevant, to compare multiple models.

10-11

10e Describe any model updating (e.g., recalibration) arising from the validation, if done. N/A

Risk groups 11 Provide details on how risk groups were created, if done. N/A

Development vs. validation

12 For validation, identify any differences from the development data in setting, eligibility criteria, outcome, and predictors.

18, 21

Results

Participants

13a Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful.

12

13b Describe the characteristics of the participants (basic demographics, clinical features, available predictors), including the number of participants with missing data for predictors and outcome.

8

13c For validation, show a comparison with the development data of the distribution of important variables (demographics, predictors and outcome).

-

Model performance

16 Report performance measures (with CIs) for the prediction model. 15-17

Model-updating 17 If done, report the results from any model updating (i.e., model specification, model performance).

N/A

Discussion

Limitations 18 Discuss any limitations of the study (such as nonrepresentative sample, few events per predictor, missing data).

21

Interpretation

19a For validation, discuss the results with reference to performance in the development data, and any other validation data.

19-20

19b Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence.

22

Implications 20 Discuss the potential clinical use of the model and implications for future research. 19

Other information

Page 33 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Supplementary information 21

Provide information about the availability of supplementary resources, such as study protocol, Web calculator, and data sets.

24 (line 2)

Funding 22 Give the source of funding and the role of the funders for the present study.

23 (line 9-11)

We recommend using the TRIPOD Checklist in conjunction with the TRIPOD Explanation and Elaboration document.

Page 34 of 34


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from






center

Journal: BMJ Open

Manuscript ID bmjopen-2017-019996.R2


Date Submitted by the Author: 28-Mar-2018

Complete List of Authors: Kim, Hyungjin; Seoul National University Hospital Park, Chang Min; Department of Radiology Jeon, Sun Kyung; Seoul National University Hospital Lee, Jong Hyuk; Seoul National University Hospital Ahn, Su Yeon; Seoul National University Hospital Yoo, Roh-Eul; Seoul National University Hospital Lim, Hyun-ju; National Cancer center Park, Juil; Seoul National University Hospital Lim, Woo Hyeon; Seoul National University Hospital Hwang, Eui Jin; Seoul National University Hospital

Lee, Sang Min; University of Ulsan College of Medicine Goo, Jin Mo; Seoul National University Hospital


Oncology


Keywords: subsolid nodule, prediction model, logistic model, external validation, adenocarcinoma, Brock model


BMJ Open on F


http://bmjopen.bm

j.com/

BM


jopen-2017-019996 on 24 May 2018. D

ownloaded from






center



1, Jong Hyuk Lee

1, Su Yeon Ahn

1, Roh-


1, 4, Juil Park

1, Woo Hyeon Lim

1, Eui Jin Hwang

1, Sang Min Lee

5, Jin

Mo Goo1, 2, 3



Korea








Page 1 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Word count: 3339

Page 2 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



ABSTRACT

Objectives: To validate the performances of two prediction models (Brock and Lee models)

for the differentiation of minimally-invasive and invasive pulmonary adenocarcinomas

(MIA/IPA) from preinvasive lesions among subsolid nodules (SSNs).



Participants: 410 patients with 410 incidentally detected SSNs who underwent surgical

resection for the pulmonary adenocarcinoma spectrum between 2011 and 2015.


predicted probability of MIA/IPA was calculated from pre-existing logistic models (Brock

and Lee models). Areas under the receiver operating characteristic curve (AUCs) were

calculated and compared between models. Performance metrics including sensitivity,


were also obtained.

Results: For pure ground-glass nodules (n=101), the AUC of the Brock model in

differentiating MIA/IPA (59/101) from preinvasive lesions (42/101) was 0.671. Sensitivity,

specificity, accuracy, PPV, and NPV based on the optimal cutoff value were 64.4%, 64.3%,

64.4%, 71.7%, and 56.3%, respectively. Sensitivity, specificity, accuracy, PPV, and NPV

according to the Lee criteria were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%, respectively.

AUC was not obtained for the Lee model as a single cutoff of nodule size (≥10 mm) was

suggested by this model for the assessment of pure ground-glass nodules. For part-solid

nodules (n=309; 26 preinvasive lesions and 283 MIA /IPAs), the AUC was 0.746 for the

Page 3 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Brock model and 0.771 for the Lee model (P=0.574). Sensitivity, specificity, accuracy, PPV,

and NPV were 82.3%, 53.8%, 79.9%, 95.1%, and 21.9%, respectively, for the Brock model

and 77.0%, 69.2%, 76.4%, 96.5%, and 21.7%, respectively, for the Lee model.

Conclusions: The performance of prediction models for the incidentally detected SSNs in

differentiating MIA/IPA from preinvasive lesions might be suboptimal. Thus, an alternative

risk calculation model is required for the incidentally detected SSNs.


• This is the first study to externally validate the performance of pre-existing risk prediction


• This study performed head-to-head comparisons between the prediction models for the risk

stratification of subsolid nodules.

• The main limitation of this study is that it only analyzed surgically resected lung nodules,


• Study population was small to conduct separate analyses for the pure ground-glass nodules

and part-solid nodules.

Page 4 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



INTRODUCTION



(AAH) and adenocarcinoma-in-situ (AIS).1 SSNs are common findings at chest computed


Indeed,






as invasive


dissection14





clinicians to date.


McWilliams et al.16




and was





Page 5 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from










(Brock and Lee models) for the differentiation of minimally invasive adenocarcinomas

(MIAs) and invasive pulmonary adenocarcinomas (IPAs) from preinvasive lesions among

SSNs. The purpose of our study was to evaluate the feasibility of the two models in the risk


Page 6 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



METHODS




Study population











three nodules. A single nodule was selected randomly for these 19 patients in order to remove

within-subject correlation. Therefore, a total of 410 nodules from 410 patients were analyzed

in the present study (figure 1). There were 174 men and 236 women (median, 61 years;

interquartile range: 54, 69 years). As for the nodule type, there were 101 pGGNs and 309

PSNs. IPAs were found in 290 nodules followed by MIA in 52 nodules, AIS in 51 nodules,

and AAH in 17 nodules. Median nodule size was 15.8 mm (interquartile range: 11.8, 20.9

mm) (table 1).


Page 7 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Age (year)a 61 (54, 69)

Sex Male 174 (42.4)

Female 236 (57.6)


AIS 51 (12.4)

MIA 52 (12.7)

IPA 290 (70.7)




No 394 (96.1)


No 366 (89.3)

Nodule size (mm)a 15.8 (11.8, 20.9)

Solid portion size (mm)a 5.7 (1.6, 11.2)

Solid proportion (%)a 41.0 (11.4, 62.0)




No 272 (66.3)


No 276 (67.3)

Nodule count per scana 3 (1, 5)


parentheses).




Page 8 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Data collection








and


















Page 9 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from












Regression coefficients and the model constant were

available from the original paper.16

Nodule size was subjected to power transformation prior

to entry as described previously.16







(nodule size, solid proportion, lobulation, and spiculation) were substituted into the following

regression formula.7

Logit�Probability� =

0.396 − 0.200 × �nodulesize�mm� − 0.048 × �solidproportion�%� + 1.049 ×

�nonlobulation� + 3.288 × �nonspiculation�

Therefore, predicted probability was obtained only for PSNs in terms of the Lee model. No

preprocessing of variables was performed.


Page 10 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from











ROC analysis was performed for each nodule type separately and then for the entire SSNs in

the case of the Brock model. In terms of the Lee model, ROC analysis was performed only

for PSNs.


As the



the McNemar test.

Lastly, calibration of the models was assessed using the Hosmer-Lemeshow test for

the 10 probability groups (deciles). All statistical analyses were performed using two

commercial software programs (MedCalc version 12.3.0, MedCalc Software, Mariakerke,

Belgium; and SPSS 19.0, IBM SPSS Statistics, Armonk, NY, USA) and R software, version

3.1.0 (http://www.R-project.org; PredictABEL package). A P-value <0.05 was considered to

indicate statistical significance.

Patient and public involvement

Patients or public were not involved in the development of the research question and

outcome measures. No patients were involved in the study design or conduct of the study.

Page 11 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Dissemination of the study results to the study participants is not practical given the

retrospective nature of our study. Lastly, there were no patient advisers.

Page 12 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



RESULTS















(n=42)

MIA and IPA

(n=59)

P-value

Age (year)a 57±10 57±10 0.839

Sex Male 16 (38.1) 29 (49.2) 0.270

Female 26 (61.9) 30 (50.8)


cancer

Yes 0 (0) 6 (10.2) 0.040

No 42 (100) 53 (89.8)

Emphysema Yes 5 (11.9) 5 (8.5) 0.738

Page 13 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



No 37 (88.1) 54 (91.5)

Nodule size (mm)a 11.1±4.1 14.2±5.4 0.002


Other lobes 17 (40.5) 28 (47.5)

Lobulation Yes 3 (7.1) 12 (20.3) 0.090

No 39 (92.9) 47 (79.7)

Spiculation Yes 1 (2.4) 7 (11.9) 0.135

No 41 (97.6) 52 (88.1)



parentheses).






larger in invasive lesions (invasive vs. preinvasive lesions: median nodule size, 17.6 mm vs.

13.6 mm, P<0.001; median solid portion size, 8.4 mm vs. 4.6 mm, P<0.001; median solid

proportion, 52.8% vs. 36.8%, P=0.032). Lobulation and spiculation were more frequently

observed in invasive lesions (invasive vs. preinvasive lesions: lobulation, 118/283 vs. 5/26,

P=0.025; spiculation, 122/283 vs. 4/26, P=0.006). There were no significant differences in

age, sex, family history of lung cancer, presence of emphysema, nodule location, and nodule

count per scan (table 3).



Page 14 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Note.

—

Unless

other

wise

specifi

ed,

data

are

numbe

rs of

patient

s

(with

percen

tages

in

parent

heses).






(n=26)

MIA and IPA

(n=283)

P-value

Age (year)a 62 (54, 68) 63 (56, 70) 0.257

Sex Male 7 (26.9) 122 (43.1) 0.109

Female 19 (73.1) 161 (56.9)


cancer

Yes 1 (3.8) 9 (3.2) 0.590

No 25 (96.2) 274 (96.8)

Emphysema Yes 2 (7.7) 32 (11.3) 0.752

No 24 (92.3) 251 (88.7)

Nodule size (mm)a 13.6 (9.8,

16.6)

17.6 (13.3,

22.4)

<0.001

Solid portion size

(mm)a

4.6 (3.4, 5.8) 8.4 (5.2, 13.6) <0.001

Solid proportion (%)a 36.8 (25.1,

63.2)

52.8 (33.4,

66.0)

0.032


Other lobes 14 (53.8) 102 (36.0)

Lobulation Yes 5 (19.2) 118 (41.7) 0.025

No 21 (80.8) 165 (58.3)

Spiculation Yes 4 (15.4) 122 (43.1) 0.006

No 22 (84.6) 161 (56.9)

Nodule count per scana 3 (2,8) 3 (1,4) 0.132

Page 15 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from








specificity, accuracy, PPV, and NPV of 64.4%, 64.3%, 64.4%, 71.7%, and 56.3%. Brock

model for pGGNs showed poor calibration (P<0.001). A nodule size cutoff (10 mm)

suggested by Lee et al.7 was also applied to our study population. The resultant sensitivity,

specificity, accuracy, PPV, and NPV were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%,

respectively. There were no significant differences in diagnostic accuracy between the Brock

model and Lee criteria (Brock model cutoff 10% vs. nodule size cutoff 10mm, P=0.461;

Brock model cutoff 4.29% vs. nodule size cutoff 10 mm, P=0.832).







was used.


Brock model1a 32.2 90.5 56.4 82.6 48.7 0.671

(0.571, 0.762) Brock model2b 64.4 64.3 64.4 71.7 56.3

Lee model

(Nodule sizec)

76.3 42.9 62.4 65.2 56.3 -

Page 16 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from














different between the two models (P=0.574 and 0.169, respectively). In addition, both models

exhibited poor calibration (P<0.001).








Brock modela 82.3 53.8 79.9 95.1 21.9 0.746

(0.694, 0.794)

Lee modelb 77.0 69.2 76.4 96.5 21.7 0.771

(0.720, 0.817)

Page 17 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from






With respect to the pooled analysis for the entire SSNs, the AUC of Brock model was

0.810 (95% CI: 0.769, 0.846). Calibration was also poor in this model (P<0.001).

Page 18 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



DISCUSSION






used and 62.4% for the Lee criteria. For the entire SSNs, the Brock model showed AUC of

0.810.

McWilliams et al.16


model) using participants enrolled in a lung cancer screening study. Thus, the Brock model

initially targeted pulmonary nodules detected on first screening CT. Incidentally detected

nodules as well as surgical candidates were not the original target lesions of this model.

However, at present, the BTS recommends using the same diagnostic approach for nodules

detected incidentally as those detected through screening.21

BTS also recommends using the

Brock model for the risk calculation of both solid and subsolid nodules.21

A cutoff of 10%

predicted probability for malignancy is suggested in order to differentiate high risk SSNs for

the performance of biopsy or surgical resection.21

This quantitative diagnostic approach is to

discern malignant SSNs with an appropriate false positive rate. However, it must be noted

that most persistent SSNs belong to one of the four categories of the adenocarcinoma

spectrum: AAH, AIS, MIA and IPA. Therefore, the potential of the risk prediction model in

discriminating lesions with invasive components (MIA and IPA) should also be tested using

pathologic diagnosis as a reference standard. Indeed, as clinical management strategies differ

substantially between preinvasive and invasive lesions, if it can be feasible to predict the

Page 19 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



invasiveness of SSNs, clinical planning whether to perform annual CT surveillance, limited

resection, or conventional lobectomy can be facilitated.



0.746 and 0.771 with diagnostic accuracies close to 80%. The performance of the prediction

model for PSNs in the original paper by Lee et al.7 was 0.905 (AUC). The study population

of the present study was similar to that of the study by Lee et al.7 An important reason for the

performance drop would be the spectrum effect, which is a common cause of model

performance heterogeneity.25

A variation in the assessment of CT morphologic features

(lobulation and spiculation) would be another potential cause. Past research on distinguishing



0.79 to 0.98.4-6 9 11 13


validated externally.






according to their solid portion size, if they are persistent lesions.26

A few studies have shown

that the solid portions in PSNs are well correlated to the pathologic invasive component.12 27

28 Fleischner Society guideline recommends that PSNs with solid components ≥6 mm be

monitored with CT scans at 3-6 months interval.26

PSNs with solid portions larger than 8 mm

should be biopsied or surgically resected in consideration of invasive adenocarcinomas.26

Page 20 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



BTS also recommends that the solid component size be considered to further refine the

estimate of malignancy risk.21

In addition, growing solid component is also a sign of an

invasive adenocarcinoma as described in both guidelines.21 26














pGGNs.30




AUCs for




In addition, for an






Page 21 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from









intervals.26









screening cohort as described earlier in this manuscript. The prevalence of preinvasive lesions

was low compared to that of the screening setting. Thus, the performance measures in this

study should be carefully interpreted with respect to the target population, which were the

incidentally detected surgical candidates. Second, our retrospective study included a small

number of patients and analyses were conducted separately for pGGNs and PSNs. Separate

analysis of pGGNs and PSNs has resulted in a slight underestimation of the performance of

Brock model. Third, optimal cutoffs for the models were not obtained from ROC analyses of

the original study populations from which the models were derived. Fourth, radiologic nodule




Page 22 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




mm). Fifth, nodule size as well as solid portion size were measured as the longest transverse

diameter in accordance with the definition of lesion size and solid proportion in the original

papers. However, recent analyses have revealed that the usage of average diameter as an

input variable may enhance the model performance.31






Page 23 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Acknowledgements


Contributors





manuscript.

Funding



Planning (grant number: 2017R1A2B4008517). Funder had no role in the study design; in the

collection, analysis and interpretation of the data; in the writing of the report; and in the

decision to submit the paper for publication.

Competing interests

None declared.

Patient consent


Review Board.

Page 24 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Ethics approval




Page 25 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



REFERENCES



2. Yankelevitz DF, Yip R, Smith JP, et al. CT Screening for lung cancer: nonsolid nodules in




2015;45:765-73.






Surg 2017;65:136-41.






Radiology 2013;268:265-73.





Page 26 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from






2016;8:1561-70.




Radiol 2016;26:43-54.





2016;26:2921-8.



2014;20:47-57.


subsolid pulmonary nodules detected at CT: a statement from the Fleischner Society.

Radiology 2013;266:304-17.






model: assessment by using a subset of the National Lung Screening Trial cohort.

Page 27 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Radiology 2017;283:264-72.


cancer risk prediction model -external validation based on CT from the Danish Lung

Cancer Screening Trial. Eur Radiol 2015;25:3093-9.



Radiol 2016;89:20160016.



ii54.




85.






Biometrics 1988;44:837-45.

25. Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using

big datasets from e-health records or IPD meta-analysis: opportunities and challenges.

BMJ 2016;353:i3140.


pulmonary nodules detected on CT images: from the Fleischner Society 2017.

Page 28 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



Radiology 2017;284:228-43.

27. Ko JP, Suh J, Ibidapo O, et al. Lung adenocarcinoma: correlation of quantitative CT

findings with pathologic findings. Radiology 2016;280:931-9.

28. Lee KH, Goo JM, Park SJ, et al. Correlation between the size of the solid component on

thin-section CT and the invasive component on pathology in small lung

adenocarcinomas manifesting as ground-glass nodules. J Thorac Oncol 2014;9:74-82.



Radiology 2014;273:276-84.





guidelines. Eur Radiol 2017;27:4019-29.

Page 29 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from



FIGURE LEGENDS



Page 30 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




127x157mm (600 x 600 DPI)

Page 31 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from







b Siemens

b GE

c GE

c Philips

d Philips

d


16

LightSpeed

Ultra

Discovery

CT750 HD


kVp 120 120 120 120 120 120 120

mAs Target standard

deviation of

noise, 14

Quality

reference

mAs, 150

Quality

reference

mAs, 100

Noise index,

16.36

Noise

index, 22.67

Reference

mAs, 200

Reference

mAs, 170

Automatic

exposure

control

Sure Exposure CARE

Dose 4D

CARE

Dose 4D


ACS, Z

DOM

Doseright

ACS, Z

DOM

Rotation time

(sec)

0.5 0.5 0.5 0.5 0.5 0.5 0.5

Collimation

(mm)

0.5x100 0.6x64 0.75x16 1.25x8 0.625x64 0.625x64 0.625x64

Slice thickness

(mm)

1 1 1 1 1.25 1 1

Pitch HP 103 1 1 0.875 0.984 1.014 1.0015

Page 32 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Increment 1 1 1 1 1.25 1 1

Matrix 512x512 512x512 512x512 512x512 512x512 512x512 512x512









Page 33 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Section/Topic Item Checklist Item Page

Title and abstract

Title 1 Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted.

1

Abstract 2 Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions.

3

Introduction

Background and objectives

3a Explain the medical context (including whether diagnostic or prognostic) and rationale for developing or validating the multivariable prediction model, including references to existing models.

5 (line 17), 6 (line 3)

3b Specify the objectives, including whether the study describes the development or validation of the model or both.

6 (line 7)

Methods

Source of data

4a Describe the study design or source of data (e.g., randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable.

7 (line 6-7)

4b Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up.

7 (line 7)

Participants

5a Specify key elements of the study setting (e.g., primary care, secondary care, general population) including number and location of centres.

2 (line 8), 9 (line 3)

5b Describe eligibility criteria for participants. 7

5c Give details of treatments received, if relevant. N/A

Outcome 6a

Clearly define the outcome that is predicted by the prediction model, including how and when assessed.

10 (line 2-3)

6b Report any actions to blind assessment of the outcome to be predicted. -

Predictors

7a Clearly define all predictors used in developing or validating the multivariable prediction model, including how and when they were measured.

9

7b Report any actions to blind assessment of predictors for the outcome and other predictors.

-

Sample size 8 Explain how the study size was arrived at. -

Missing data 9 Describe how missing data were handled (e.g., complete-case analysis, single imputation, multiple imputation) with details of any imputation method.

-

Statistical analysis methods

10c For validation, describe how the predictions were calculated. 10

10d Specify all measures used to assess model performance and, if relevant, to compare multiple models.

10-11

10e Describe any model updating (e.g., recalibration) arising from the validation, if done. N/A

Risk groups 11 Provide details on how risk groups were created, if done. N/A

Development vs. validation

12 For validation, identify any differences from the development data in setting, eligibility criteria, outcome, and predictors.

18, 21

Results

Participants

13a Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful.

12

13b Describe the characteristics of the participants (basic demographics, clinical features, available predictors), including the number of participants with missing data for predictors and outcome.

8

13c For validation, show a comparison with the development data of the distribution of important variables (demographics, predictors and outcome).

-

Model performance

16 Report performance measures (with CIs) for the prediction model. 15-17

Model-updating 17 If done, report the results from any model updating (i.e., model specification, model performance).

N/A

Discussion

Limitations 18 Discuss any limitations of the study (such as nonrepresentative sample, few events per predictor, missing data).

21

Interpretation

19a For validation, discuss the results with reference to performance in the development data, and any other validation data.

19-20

19b Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence.

22

Implications 20 Discuss the potential clinical use of the model and implications for future research. 19

Other information

Page 34 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from




Supplementary information 21

Provide information about the availability of supplementary resources, such as study protocol, Web calculator, and data sets.

24 (line 2)

Funding 22 Give the source of funding and the role of the funders for the present study.

23 (line 9-11)

We recommend using the TRIPOD Checklist in conjunction with the TRIPOD Explanation and Elaboration document.

Page 35 of 35


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960



jopen.bmj.com

/B

MJ O


ay 2018. Dow

nloaded from


Documents

bmjopen.bmj.com · For peer review only TRIPOD Checklist: Prediction Model Validation Section/Topic Item Checklist Item Page Title and abstract Title 1 Identify the study as developing