Editorial Team Comments changes to the text of the ... · From a clinical perspective, understand the effects of more intensive management of patients who return after discharge in

Editorial Team Comments**Report from The BMJ’s manuscript committee meeting** Present: Elizabeth Loder (chair); Rafael Perera (statistician); Wim Weber; Jose Merino; Helen MacDonald; Daoxin Yin; Tim Feeney; John Fletcher

1. We noted that this paper covers a particularly important topic. It’s clear that the aim is “improve care” so that patients don’t get readmitted, yet it's not obvious this has been the outcome. It is understandable that you will start to see a shift away from readmissions and towards observation or ED to home visits and an attempt to decrease readmits, and this paper quantifies those changes nicely. However, as one of the reviewers points out, it's not entirely clear if this is a bad thing. Are people suffering worse outcomes from these? If not, then they were possibly being readmitted in the past and not needing to be, in which case this is fine. However, if they are worse off, then this policy is having a detrimental effect on patient care. We thank the editorial team for these thoughtful comments. As per the editorial team’s suggestions, we have made several major changes to the manuscript. We have now performed an analysis of 30-day post-discharge mortality across all conditions (and by individual conditions) to examine whether shifts in ED and observation stay use impacted outcomes from 2012 to 2015. In addition, we have added several subgroup analyses (per the editor and reviewers’ suggestions) to understand if these shifts are concentrated among specific patients – interestingly, we find that increases in observation stay use are more pronounced among non-white compared to white patients. We have detailed these additional analyses below, and in response to question #4, and have made substantial changes to the text of the manuscript to reflect these changes. From a policy perspective, our finding that 30-day total hospital revisits actually increased (while 30-day readmissions declined over the same time period) is important for a few reasons. First, policymakers in the US have argued that reductions in 30-day readmissions reflect improved discharge planning and care transitions during index hospitalization, as well as better care coordination and quality in the post-discharge period. Our finding, however, of a concomitant increase in 30-day total hospital revisits suggests that reductions in readmissions are unlikely to have been driven by widespread improvements in quality of care (one would expect total hospital revisits to also decline if discharge planning, care transitions, and post-discharge care coordination were actually improving), but instead, simply reflect hospitals more aggressively managing and treating patients who do return to the hospital in EDs and observation stays. Second, 30-day readmission measures are increasingly being incorporated into US value-based programs (e.g. HRRP and now outpatient P4P) to evaluate provider/hospital care quality, compare performance, and reward or financially penalize providers/hospitals. Our findings suggest that focusing solely on 30-day readmissions may provide an “illusory” sense of improvement in quality of care, and importantly, may also impede and distort fair comparisons of performance, since hospitals’ ability to manage patients in EDs and observation units vary widely. Third, our findings indicate that a 30-day “return to hospital” metric, which captures all post-discharge encounters (inpatient, ED, and observation stays), may provide a more comprehensive, accurate, and fair assessment of hospital quality and performance.

From a clinical perspective, understand the effects of more intensive management of patients who return after discharge in EDs and observation units will require further study. We agree that it will be important to evaluate how patient experience and outcomes have changed. Unfortunately, measures of patient experience are not captured in claims data among patients who return to an ED or observation unit. We do, however, now show trends in 30-day post-discharge mortality for target conditions in our study (Figure 2 and eTable 2). Importantly, we observed no change in mortality for all conditions from 2012 to 2015.

We have made the following changes/additions:Page 11-12: “Overall, we observed no significant changes in mortality within 30 days of discharge across three targeted conditions (-0.0034 [95% CI -0.012 to 0.0054]) from 2012 to 2015. Thirty-day post-discharge mortality did not change among patients hospitalized for HF (0.00 [95% CI -0.011 to 0.010), AMI (-0.006 [95% CI -0.015 to 0.002), or pneumonia (-0.004 [95% CI -0.013 to 0.005]) (Figure 2).”

Page 12: “In the United States, nationwide declines in readmission rates for target conditions have been viewed as markers of improvements in quality of care. Our findings suggest that this success may be illusory because total hospital revisits after discharge are, in fact, rising. If reductions in readmissions were being driven by widespread improvements in discharge planning, care transitions and post-discharge care following a hospitalization – as intended by health policymakers – total hospital revisits within 30 days of discharge would also be expected to decline. Instead, much of the reduction in readmissions appears to reflect intensified efforts to manage patients who return to a hospital after discharge in observation units and EDs, potentially because the 30-day readmission measure used to evaluate hospital performance does not include these types of post-discharge encounters.”

Page 11-12: “The increase in use of observation stays and ED visits (compared with inpatient hospitalizations) among patients who return after discharge may be a good thing if it reflects that patients are, on average, returning with lower severity illness that can be safely managed in a non-admission setting. They may also be beneficial to patient care. For instance, observation stays have been associated with higher patient satisfaction compared with inpatient hospitalizations,12 although they can also result in higher out-of-pocket expenditures and more financial hardship for patients compared with inpatient hospitalizations.13 14”

“However, there is reason to believe that the increasing use of ED visits for post-discharge care may be problematic. Prior data suggest that hospitals that tend to manage patients in EDs rather than admitting them for an inpatient stay have higher rates of early death after discharge.15 Although we observed no change in 30-day post-discharge mortality for target conditions during the HRRP (2012 to 2015), several recent analyses have found that the implementation of the HRRP was associated with an increase in 30-day post-discharge mortality among patients hospitalized for HF and pneumonia compared with pre-HRRP (pre-2010) trends, and that this increase was concentrated entirely among patients not readmitted after discharge.6 16-18 Whether intensified efforts to manage returning patients in EDs and observation units can explain increases in mortality observed in the years that preceded our study period is an important area for further research.”

Page 14: “Our findings have important policy implications for value-based programs that use the 30-day readmission measure to evaluate hospital and provider care quality. First, exclusively focusing on 30-day readmissions while ignoring other types of hospital revisits overestimates the clinical and financial benefits of incentives to reduce readmissions. Second, using 30-day readmissions as the sole quality metric may impede fair comparisons of hospital performance, particularly given wide variation in ED triage patterns and the availability and use of observation units.19 Finally, given these limitations, the 30-day readmission rate appears to be an inappropriate target for financial incentives for hospitals (as used in the HRRP) or outpatient practices (as being increasingly used in pay-for-performance programs). Measuring all revisits within 30 days of discharge (a “30-day return to hospital” metric) may instead provide a more comprehensive, accurate, and fair assessment of provider and hospital care quality.20”

Figure 2. Risk-Standardized Mortality within 30-Days of Discharge Among Medicare Patients Hospitalized for Heart Failure, Acute Myocardial Infarction, or Pneumonia

This figure demonstrates spline-fitted trends of 30-day post discharge mortality by target condition. Heart failure is shown in blue, acute myocardial infarction in red, and pneumonia in green.

eTable 2. Risk-Standardized Mortality within 30-Days of Discharge Among Medicare Patients Hospitalized for Heart Failure, Acute Myocardial Infarction, or Pneumonia

Monthly Change 30-Day Post-Discharge Mortality

(95% confidence interval)All conditions -0.003 (-0.012 to 0.005) Heart Failure 0.00 (-0.011 to 0.010) Acute Myocardial Infarction -0.006 (-0.015 to 0.002)

Pneumonia -0.004 (-0.013 to 0.005)

2. We felt that the "intervention" (readmission penalties) is not described well enough for an audience that is unfamiliar with the US system and is a system wide policy intervention rather than one that clinicians could implement. Could you provide more context for international readers? Thank you for this suggestion. We now provide a more detailed description of readmission penalties to provide better context for readers. If the editorial team does not feel that this level of description is enough for an audience unfamiliar with the HRRP, we would be happy to provide more details. The primary focus of this study was to highlight the limitations of US policymakers focusing solely on national declines in 30-day readmissions as a metric for quality of care/performance/utilization, particularly given that this metric is increasingly being used in other value-based payment programs (e.g. outpatient P4P).

Page 6: “In the US, the Centers for Medicare and Medicaid Services (CMS) has implemented national initiatives that aim to push clinicians and hospitals to reduce readmissions for Medicare fee for service patients over the age of 65. In 2009, for example, CMS began publicly reporting 30-day readmission rates as a measure of hospital performance. One year later, the Hospital Readmissions Reduction Program (HRRP) was established, mandating that CMS impose financial penalties on acute-care hospitals in the US with higher than expected 30-day readmission rates after a hospitalization for common conditions. Penalties under the HRRP began in 2012 and are capped at 3% of Medicare payments to hospitals; 82% of hospitals were penalized in fiscal year 2019.2”

3. Implicit in the discussion of the results is the assumption that the changes noted are a consequence of the policy change. But if the policy was introduced in 2012 we have only "after" data. These trends may be part of a wider picture that has less to do with the change in policy. Are you able to provide any "before" information? The primary focus of this study was to demonstrate the potential pitfalls of solely using 30-day readmission rates as a quality or performance measure in US value-based programs, rather than to focus on the HRRP. Unfortunately, we do not have access to Medicare claims outpatient data (which contain information about ER visits and observation stays) prior to 2012.

As now described in our detailed response to Question #1, our finding that 30-day total hospital revisits have increased suggest that reductions in readmissions are unlikely to have been driven by widespread improvements in quality of care (one would expect total hospital revisits to also decline if discharge planning, care transitions, and post-discharge care coordination were actually improving), but instead, may simply reflect more aggressive management of patients who do return to the hospital in EDs and observation stays. This finding is important because 30-day readmission measures are increasingly being incorporated into US value-based programs (HRRP and P4P) to evaluate

provider/hospital care quality, compare performance, and determine financial penalties or rewards.

Page 14: “Our findings have important policy implications for value-based programs that use the 30-day readmission measure to evaluate hospital and provider care quality. First, exclusively focusing on 30-day readmissions while ignoring other types of hospital revisits overestimates the clinical and financial benefits of incentives to reduce readmissions. Second, using 30-day readmissions as the sole quality metric may impede fair comparisons of hospital performance, particularly given wide variation in ED triage patterns and the availability and use of observation units.19 Finally, given these limitations, the 30-day readmission rate appears to be an inappropriate target for financial incentives for hospitals (as used in the HRRP) or outpatient practices (as being increasingly used in pay-for-performance programs). Measuring all revisits within 30 days of discharge (a “30-day return to hospital” metric) may instead provide a more comprehensive, accurate, and fair assessment of provider and hospital care quality.20”

4. And as reviewers suggest, might you provide information about mortality, patient subgroups, etc.? We thank the editorial board and reviewers for this suggestion. We have performed several subgroup analyses, and have also evaluated trends in mortality, all of which have been added to the text and figures/tables of the manuscript.

Page 9: “Hospital revisits within 30-days of discharge by patient subgroups (age, sex, and race) were also evaluated, as shown in Figure 3A-C. The monthly rate of change in total hospital revisits per 100 patient discharges did not significantly differ among patients <80 years of age compared with patients ≥80 years of age, nor did trends in ED treat-and-discharge visits, observation stays, and readmissions (Figure 3A). In addition, there was no significant difference in revisit trends among men compared to women (Figure 3B). The monthly rate of change in total hospital revisits were also similar among white patients compared to non-white patients, as were monthly rates of change in ED treat-and-discharge visits (Figure 3C). However, use of observation stays within 30 days of discharge increased significantly faster among non-white patients (+0.029 [95% CI 0.024 to 0.034]) compared with white patients (+0.021 [95% CI 0.018 to 0.024]; p-value for difference, 0.006).”

Page 9-10: “Overall, we observed no significant changes in mortality within 30-days of discharge for aggregate conditions (-0.0034 [95% CI -0.012 to 0.0054]) across the study period. Thirty-day post-discharge mortality did not change among patients hospitalized for HF (0.00 [95% CI -0.011 to 0.010), AMI (-0.006 [95% CI -0.015 to 0.002), or pneumonia (-0.004 [95% CI -0.013 to 0.005]) (Figure 2).”

Table 3. Risk-Standardized Monthly Rate of Change in Hospital Revisits, Emergency Department Treat-and-Discharge Visits, Observation Stays, and Readmissions within 30-days of Discharge for Target Conditions by Patient Subgroups.

A. AgeMonthly Change in # Revisits per 100 Patient Discharges (95% CI)

<80 y.o. ≥80 y.o. P value

Single Revisit a Any Hospital Revisit +0.016 (0.007 to 0.027) +0.015 (0.004 to 0.026) 0.89 ED Treat-and-Discharge Visit +0.024 (0.016 to 0.032) +0.019 (0.012 to 0.027) 0.36 Observation Stay +0.023 (0.020 to 0.026) +0.021 (0.018 to 0.023) 0.30 Readmission -0.014 (-0.025 to -0.003) -0.012 (-0.023 to 0.00) 0.80

All Revisits b Total Hospital Revisits +0.030 (0.015 to 0.045) +0.015 (0.012 to 0.028) 0.08 ED Treat-and-Discharge Visit +0.027 (0.017 to 0.037) +0.020 (0.011 to 0.029) 0.29 Observation Stay +0.024 (0.021 to 0.027) +0.020 (0.017 to 0.024) 0.08 Readmission -0.021 (-0.034 to -0.009) -0.026 (-0.038 to -0.014) 0.56

B. SexMonthly Change in # Revisits per 100 Patient Discharges

(95% CI)

Men Women P value

Single Revisit a Any Hospital Revisit +0.016 (0.006 to 0.026) +0.016 (0.005 to 0.027) 1.00 ED Treat-and-Discharge Visit +0.022 (0.015 to 0.029) +0.021 (0.014 to 0.029) 0.84 Observation Stay +0.023 (0.021 to 0.025) +0.020 (0.017 to 0.024) 0.13 Readmission -0.013 (-0.025 to -0.001) -0.012 (-0.023 to -0.002) 0.90

All Revisits b Total Hospital Revisits +0.024 (0.012 to 0.037) +0.021 (0.006 to 0.035) 0.75 ED Treat-and-Discharge Visit +0.024 (0.015 to 0.034) +0.023 (0.013 to 0.032) 0.88 Observation Stay _+0.024 (0.022 to 0.027) +0.021 (0.017 to 0.025) 0.20 Readmission -0.024 (-0.037 to -0.012) -0.023 (-0.034 to -0.011) 0.91

C. Race

Monthly Change in # Revisits per 100 Patient Discharges (95% CI)

White Non-White P value

Single Revisit a Any Hospital Revisit +0.017 (0.008 to 0.027) +0.009 (-0.005 to 0.023) 0.36 ED Treat-and-Discharge Visit +0.022 (0.015 to 0.029) +0.020 (0.011 to 0.028) 0.71 Observation Stay +0.021 (0.018 to 0.023) +0.027 (0.023 to 0.031) 0.01 Readmission -0.011 (-0.022 to -0.001) -0.021 (-0.035 to -0.007) 0.26


P value for difference in monthly rate of change among subgroups.a Among patients with multiple hospital revisits within 30-days of discharge, only a single revisit for each type of encounter was counted. For “any hospital revisit,” only a single encounter of any type after discharge was counted.b Among patients with multiple hospital revisits within 30-days of discharge, all visits for each type of encounter were counted.

Figure 2. Risk-Standardized Mortality within 30-Days of Discharge Among Medicare Patients Hospitalized for Heart Failure, Acute Myocardial Infarction, or PneumoniaThis figure demonstrates spline-fitted trends of 30-day post discharge mortality by target condition. Heart failure is shown in blue, acute myocardial infarction in red, and pneumonia in green.



(95% confidence interval)All conditions -0.003 (-0.012 to 0.005) Heart Failure 0.00 (-0.011 to 0.010) Acute Myocardial Infarction -0.006 (-0.015 to 0.002) Pneumonia -0.004 (-0.013 to 0.005)

5. We wonder why you needed to model potential outcome in month k instead of just model (using splines) observed outcome. We were not entirely clear on what the advantage of this is and would like you to comment on it (as an appendix if not part of the flow of the paper).

Thank you for raising this point. We did not model the potential outcome as we did not observe the potential outcome. We modeled the observed outcome within each month as a way to estimate the mean of the potential outcome. A conventional spline model including calendar month and other confounders is, in our opinion, has limitations. This model assumes the effect of the other confounders (e.g. demographics and comorbidities) is the same at any given calendar month, which is unrealistic. In addition, the large volume of data allows us to use a more sophisticated strategy. By modeling the observed outcome within each month, we allow the effects of demographics and comorbidities to be potentially different at different calendar months, a more robust strategy that minimizes potential bias due to model misspecification. We provide more details regarding our statistical approach under “Detailed Statistical Methods” and have added a section entitled “Inferential Strategy” to the appendix.

Inferential StrategyLet be the month indicator taking values 1, 2, …, 45, indicating the 45 months from January 𝑨2012 to September 2015. Let be the potential outcome (e.g. 30-day post-discharge 𝒀𝒌

readmission) under the care in month . Let be the observed outcome, 𝒌 𝒀 = ∑𝟒𝟓𝒊 = 𝟏𝑰(𝑨 = 𝒌)𝒀𝒌

where I is the indicator function. Let R indicate the set of index hospitalizations in 2014. Let 𝝁𝒌 be the average of the potential outcome over the population R. Let be a vector = 𝑬(𝒀𝒌|𝑹) 𝒀𝒌 𝑿

that includes demographics and comorbidities variables. Under the assumption that and 𝒀𝒌 𝑨are independent conditional on . We have 𝑿 𝝁𝒌 = 𝑬(𝒀𝒌│𝑹) = 𝑬[𝑬(𝒀𝒌│𝑿)│𝑹] = 𝑬[𝑬(𝒀𝒌│𝑿,𝑨 = 𝒌)│𝑹] = 𝑬[𝑬(𝒀│𝑿,𝑨 = 𝒌)│𝑹](1)Equation (1) suggests that we can estimate by𝝁𝒌

,𝝁𝒌 = 𝑬[𝑬(𝒀│𝑿,𝑨 = 𝒌)│𝑹]where is the predicted observed outcome based on a regression model fitted to 𝑬(𝒀│𝑿,𝑨 = 𝒌)data from month , and is the empirical average over the hospitalizations of (𝑿, 𝒀) 𝒌 𝑬[ ∙ │𝑹]2014 in our analysis data set. Therefore, the strategy to estimate is the following:𝝁𝒌1. Fit a logistic regression model using data in month (𝑴𝒌 𝒌 𝒌 = 𝟏,𝟐,…,𝟒𝟓)2. Compute the average of the predicted outcome under using the data from 2014, which 𝑴𝒌 𝑿is 𝝁𝒌

Once we obtain ( , we fit a smoothing spline model to to estimate the 𝝁𝒌 𝒌 = 𝟏,𝟐,…,𝟒𝟓) 𝝁𝒌longitudinal trend. Because we already standardized the case-mix to that of 2014, the trend reflects effect of some other factors beyond case-mix.

6. On a related matter, we wondered why you didn't use a seasonal component for this estimate as there appears to be some seasonal issue going on in the potential outcome data. Thank you. The smoothing spline we fitted to the data has sufficient degree of freedom to accommodate seasonal effect.

7. With your estimation of linear change (monthly), would you check if the data (predicted outcome) meets linear regression assumptions? Thank you. The linear model is not intended to fully capture the longitudinal trend of the hospital revisits. The curves in the graphs clearly demonstrate that for some outcomes (e.g. readmission), a linear model is not the best model. The purpose of the linear model is to offer a summary measure of slope. A parallel scenario, as an example, is that a correlation coefficient is still meaningful even the relationship between two continuous variables is not strictly linear.

8. Please include a statement in the methods section about whether you have involved patients or the public in planning or interpreting the study. If not, it is fine to simply say that. Please also include a statement about how you plan to disseminate the results of the study to relevant groups. Thank you. We have now added the following statement to the manuscript:

Page 8: “Patients and the public were not involved in planning, design, or interpretation of the study. The study involved examination of existing claims data and no participants were recruited for this analysis. We intend to engage patients and health policymakers by disseminating this research through press releases, blog posts, and at research meetings.”

Response to Reviewer #1: 1. Wadhera et al. describe a retrospective analysis of all revisits after discharge in the US HRRP. They show an increase in total revisits, with a decrease in readmissions. They reflect upon the 'actual' effect of the program, and wonder if it has really improved the quality of care - however they stay quite superficial when discussing. Readmission literature is extensive and therefore often not 'sexy', the good thing is the authors have a large bulk of data and the message is simple, clear and important. However, I do miss a more in-depth discussion about the use of readmissions/revisits as a quality indicator (this paper yet again shows the complexity around this - is getting readmitted always really a bad idea?) and another caveat is the lack of comparing to 'before' the program. They did do this in their JAMA paper. These facets might put the paper into a better, more balanced perspective. Especially for the reader who is not very into the 'readmission literature'. We thank the reviewer for these thoughtful comments. We have made several changes to strengthen the manuscript as per the reviewer’s suggestions. First, to provide more balance and depth, we evaluated changes in post-discharge mortality across our study period (Figure 2). Second, we have performed additional analyses to evaluate whether trends in 30-day total hospital revisits (and each type of visit) have changed differentially among patient subgroups. Third, we have made several changes to the discussion (shown below) and now place our findings in the context of prior literature that has shown that quality of care does not appear to differ at hospitals with high vs. low readmission rates. Unfortunately, we lack outpatient claims data prior to 2012 (which are required to capture ED and observation stays) and cannot provide more data prior to this year.

The primary aim of this study was to highlight that although 30-day readmissions declined from 2012 through 2015, 30-day total hospital revisits actually increased. Policymakers in the United States have argued that reductions in 30-day readmissions reflect improved discharge planning and care transitions during index hospitalization, as well as better care coordination and quality in the post-discharge period. Our finding, however, of a concomitant increase in 30-day total hospital revisits suggests that reductions in readmissions are unlikely to have been driven by widespread improvements in quality of care (one would expect total hospital revisits to also decline if discharge planning, care transitions, and post-discharge care coordination were actually improving), but instead, simply reflect hospitals more aggressively managing and treating patients who do return to the hospital in EDs and observation stays, potentially because these post-discharge events are not “counted” by the readmission measure.

This finding is important because 30-day readmission measures are increasingly being incorporated into US value-based programs (HRRP and P4P) to evaluate provider/hospital care quality, compare performance, and reward or financially penalize hospitals. Our findings suggest that focusing solely on 30-day readmissions may provide an “illusory” sense of quality improvement, and importantly, may also impede and distort fair comparisons of performance, since hospitals’ ability to manage patients in EDs and observation units vary widely. Given our findings, we suggest a 30-day “return to hospital” metric, which captures all post-discharge encounters (inpatient, ED, and observation

stays), to provide a more comprehensive, accurate, and fair assessment of hospital quality and performance.

Page 12: “In the United States, nationwide declines in readmission rates for target conditions have been viewed as markers of improvements in quality of care. Our findings suggest that this success may be illusory because total hospital revisits after discharge are, in fact, rising. If reductions in readmissions were being driven by widespread improvements in discharge planning, care transitions and post-discharge care following a hospitalization – as intended by health policymakers – total hospital revisits within 30 days of discharge would also be expected to decline. Instead, much of the reduction in readmissions appears to reflect intensified efforts to manage patients who return to a hospital after discharge in observation units and EDs, potentially because the 30-day readmission measure used to evaluate hospital performance does not include these types of post-discharge encounters.”



Page 14: “Our findings have important policy implications for value-based programs that use the 30-day readmission measure to evaluate hospital and provider care quality. First, exclusively focusing on 30-day readmissions while ignoring other types of hospital revisits overestimates the clinical and financial benefits of incentives to reduce readmissions. Second, using 30-day readmissions as the sole quality metric may impede fair comparisons of hospital performance, particularly given wide variation in ED triage patterns and the availability and use of observation units.19 Finally, given these limitations, the 30-day readmission rate appears to be an inappropriate target for financial incentives for hospitals (as used in the HRRP) or outpatient practices (as being increasingly used in pay-for-performance programs). Measuring all revisits within 30 days of discharge (a “30-day return to hospital” metric) may

instead provide a more comprehensive, accurate, and fair assessment of provider and hospital care quality.20”






2. There is also a report from 2018 http://medpac.gov/docs/default-source/reports/jun18_medpacreporttocongress_sec.pdf?sfvrsn=0 showing results of the HRRP, but giving more thorough explanations. It would be interesting to see the authors reflections upon these. Thank you for highlighting the Medicare Payment and Advisory Commission report, which made two important conclusions: 1. Readmissions have declined substantially for

target conditions since 2010, 2. Mortality has declined substantially for target conditions since 2010.

However, several evaluations by independent groups contradict MedPAC’s main findings. Two peer-reviewed studies have shown that much of the previously observed decline in risk-adjusted readmissions was explained by an artificial increase in coded severity of illness that occurred in 2011.

1. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/26632522. https://www.healthaffairs.org/doi/abs/10.1377/hlthaff.2018.05178

In addition, several studies have shown that the HRRP was associated with an increase in mortality among Medicare patients hospitalized with heart failure, compared to trends prior to the program.

1. https://jamanetwork.com/journals/jamacardiology/fullarticle/26632132. https://jamanetwork.com/journals/jama/article-abstract/27193073. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2728002

3. Is there other evidence whether readmission programs (in and outside of the US) influence overall revisits? I cannot imagine there is no data on this (considering the bulk of readmission literature). I would like to know 'what is new'. In the authors recent JAMA paper discussing effect of program on mortality they do reflect upon this (33). Also the accompanying editorial by Fonarow has cited relevant papers. Thank you for raising this point. To our knowledge, there are no published studies that evaluate 30-day overall hospital revisits (inpatient + emergency department + observation) after discharge and compare them to 30-day readmission rates. Consequently, we believe this study makes a novel and unique contribution.

4. It might be useful to explain a bit earlier why the authors chose to take two comparisons (all revisits and singles, and within 30 days). We thank the reviewer for this suggestion. We now, in the introduction, highlight the rational for evaluating single and all revisits within 30 days of discharge.

Page 7: “Therefore, in this study, we aimed to answer three policy-relevant questions. First, have total hospital revisits within 30 days of discharge following a hospitalization for conditions targeted by CMS readmission measures changed over time? Second, how have rates of ED treat-and-discharge visits, observation stays, and readmissions each contributed to changes in total hospital revisits? Third, do these patterns differ if all 30-day post-discharge revisits per patient are counted, to provide a more comprehensive assessment of hospital utilization after discharge, rather than just the first revisit as done by CMS for the readmission measure?”

5. Why only patients above 65? I think in younger patients it is also relevant to see if they come back or not. We focused on Medicare patients 65 years or older because this is the population targeted by nationally mandated policies (e.g. HRRP, P4P) – implemented by the Centers for

https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2663252

https://www.healthaffairs.org/doi/abs/10.1377/hlthaff.2018.05178

https://jamanetwork.com/journals/jamacardiology/fullarticle/2663213

https://jamanetwork.com/journals/jama/article-abstract/2719307

Medicare and Medicaid Services - that aim to reduce readmissions in the United States. Currently, these programs do not target patients less than 65 years of age.

Page 6: “In the US, the Centers for Medicare and Medicaid Services (CMS) has implemented national initiatives that aim to push clinicians and hospitals to reduce readmissions for Medicare fee for service patients over the age of 65. In 2009, for example, CMS began publicly reporting 30-day readmission rates as a measure of hospital performance. One year later, the Hospital Readmissions Reduction Program (HRRP) was established, mandating that CMS impose financial penalties on acute-care hospitals in the US with higher than expected 30-day readmission rates after a hospitalization for common conditions.”

6. Please see above earlier general comments. Discussion could be more thorough and more reflective. As a reader: Thank you. We have made several changes to the discussion (as detailed above) to provide more thorough and balanced insights regarding what our findings mean in the context of the broader literature on readmissions.

Is it true? Yes, methods seem thorough. Is it new? Yes, I think these results are but I am not 100 % sure about the other literature. Does it matter? Yes, but it could be more clear from the paper why it matters.

Please enter your name: Louise van Galen

Job Title: Research Fellow, Internal Medicine Resident

Institution: Amsterdam University Medical Centre, Location VUmc

Response to Reviewer #2: The authors present an important, clearly described description of the changes in readmissions and "revisits" among Medicare beneficiaries in the US between 2012 and 2015. Given the international emphasis on such outcomes, it is timely and fills a much need knowledge gap. The use of national data - along with the specific sample is appropriate, and the analysis seems appropriate overall as well. Several issues, if dealt with, could improve the impact of the piece considerably.

1. At present, this is really just a descriptive piece with no real comparison group. A comparison group is not obvious here, but some additional analysis (or even additional description) would flesh out these findings. Compare race/ethnic groups? age groups? Geographic regions? these would enhance the rigor, and better help explain the findings. We know Medicare expenditures vary widely in US regions/states, so its a little too broad strokes just to do national figures and interpret these. Without significant additions, this should probably be a letter with ~ 500-750 words, not a 2000 word piece. We thank the reviewer for these thoughtful suggestions. As the reviewer notes, there is no robust comparison group. We have, however, made several additions to the manuscript as per the reviewer’s suggestion, to strengthen the study. First, as shown in Table 3A-C, we now perform several additional analyses to evaluate patterns of 30-day total hospital revisits (overall and by type of revisit) by patient subgroups (<80 years of age vs 80 years or older; men vs. women; white vs. nonwhite). Interestingly, we find that the rate of increase in observation use after discharge is more pronounced among non-white patients compared to white patients. Second, we now also evaluate trends in 30-day post-discharge mortality by condition and across all conditions (Figure 2 and eTable 2).

The following additions have been made to the manuscript and figures/tables:Page 11: “Patient subgroups (age, sex, and race) were also evaluated, as shown in Figure 3A-C. Counting all revisits per patient, the monthly change in total hospital revisits per 100 patient discharges did not significantly differ among patients <80 years of age compared with patients ≥80 years of age, nor did trends in ED treat-and-discharge visits, observation stays, and readmissions (Figure 3A). In addition, there was no significant difference in revisit trends among men compared to women (Figure 3B). The monthly change in total hospital revisits were also similar among white patients compared to non-white patients, as were ED treat-and-discharge visits (Figure 3C). However, increases in observation stays within 30 days of discharge were more pronounced among non-white patients (+0.029 [95% CI 0.024 to 0.034]) compared with white patients (+0.021 [95% CI 0.018 to 0.024]; p-value for difference, 0.006).”

Page 11-12: “Overall, we observed no significant changes in mortality within 30 days of discharge across three targeted conditions (-0.0034 [95% CI -0.012 to 0.0054]) from 2012 to 2015. Thirty-day post-discharge mortality did not change among patients hospitalized for HF (0.00 [95% CI -0.011 to 0.010), AMI (-0.006 [95% CI -0.015 to 0.002), or pneumonia (-0.004 [95% CI -0.013 to 0.005]) (Figure 2).”







(95% CI)

Men Women P value



C. Race





P value for difference in monthly rate of change among subgroups.

a Among patients with multiple hospital revisits within 30-days of discharge, only a single revisit for each type of encounter was counted. For “any hospital revisit,” only a single encounter of any type after discharge was counted.b Among patients with multiple hospital revisits within 30-days of discharge, all visits for each type of encounter were counted.






2. These changes are relatively small. Given a small time frame for this level of policy evaluation (3-4 years), I think this needs to mentioned in the discussion that the clinical significance of this is uncertain Thank you. We agree that the clinical significance of these findings is unclear and warrants further study – this is now stated explicitly in the discussion (shown below). In addition, as described in response to comment #1, we now have performed an additional analysis that evaluates trends in post-discharge mortality over this period (Figure 2 and eTable 2 shown above), which did not significant change by condition or in aggregate.

Though the clinical significance of these changes need to be explored further, we do believe our findings have policy significance. Policymakers in the US have argued that reductions in 30-day readmissions reflect improved discharge planning and care transitions during index hospitalization, as well as better care coordination and quality in the post-discharge period. Our finding, however, of a concomitant increase in 30-day total hospital revisits

suggests that reductions in readmissions are unlikely to have been driven by widespread improvements in quality of care (one would expect total hospital revisits to also decline if discharge planning, care transitions, and post-discharge care coordination were actually improving), but instead, may reflect hospitals more aggressively managing and treating patients who do return to the hospital in EDs and observation stays, potentially because these post-discharge events are not “counted” by the readmission measure.

This finding is important because 30-day readmission measures are increasingly being incorporated into US value-based programs (e.g. HRRP and P4P) to evaluate provider/hospital care quality, compare performance, and reward or financially penalize hospitals. Our findings suggest that focusing solely on 30-day readmissions may not provide a comprehensive sense of care quality, and importantly, may also impede and distort fair comparisons of performance, since hospitals’ ability to manage patients in EDs and observation units vary widely. Given our findings, we suggest a 30-day “return to hospital” metric, which captures all post-discharge encounters (inpatient, ED, and observation stays), to provide a more comprehensive, accurate, and fair assessment of hospital quality and performance.

We have made the following changes to the manuscript as per the reviewer’s suggestion:Page 11-12: “The increase in use of observation stays and ED visits (compared with inpatient hospitalizations) among patients who return after discharge may be a good thing if it reflects that patients are, on average, returning with lower severity illness that can be safely managed in a non-admission setting. They may also be beneficial to patient care. For instance, observation stays have been associated with higher patient satisfaction compared with inpatient hospitalizations,12 although they can also result in higher out-of-pocket expenditures and more financial hardship for patients compared with inpatient hospitalizations.13 14”


3. in the discussion, the authors portray an increase in ED visit and DC and OBS stays as a bad thing, citing 1 paper. I'm not sure this is enough to pronounce this trend as negative, especially when the associated cost difference of inpatient stay- obs/ED might be significant, and reduced cost is also a significant goal of initiatives to reduce readmission. Examining mortality in this dataset, estimating cost differences, or looking at diagnoses associated with "revisits" would

strengthen the argument here. The authors need to at least have more equipoise in their discussion about the implications of the trend Thank you – we completely agree with the reviewer’s constructive comment. As described in the responses above, we now evaluate trends in 30-day post-discharge mortality (shown above), and discuss the fact that we observed no change in this outcome. We also now offer more equipoise regarding the clinical significance of these findings in the discussion (as shown above).

Response to Reviewer #3:Thank you for the opportunity to review this manuscript. The use of a large administrative dataset to highlight the challenges of 30 day readmissions as a measure of quality are well described. This paper highlights how an unintended consequence of the use of 30 day readmissions as a quality measure may be longer short stays or ED visits which are not formally considered readmission but still include a patient returning to hospital for care. Overall, this paper makes this case forcefully but some additional discussion regarding unintended consequences of health policy or administrative decisions would be warranted, perhaps drawing on the literature from elsewhere... there has been much written on this topic regarding the star rating system in the UK and ED wait times.

1. Page 7, line 45 - there is an error in text reading "Poison regression", should be Poisson We thank the reviewer for noticing this error, which has now been corrected.

2. Page 10, lines 3 to 10 - The authors note that early death is associated with ED visits or short stay 30 days after hospital discharge , however the outcome of death is not explored in any depth in the paper nor it is noted as a data limitation. Offering some explanation as to why this was decided would be useful and would in particular strengthen this important argument in the paper. We thank the reviewer for raising this important point. We have now performed an additional analysis that evaluates trends in 30-day post-discharge mortality (Figure 2 and eTable 2). In addition, we have made several changes to the text of the manuscript.

Page 10-11: “Overall, we observed no significant changes in mortality within 30 days of discharge across three targeted conditions (-0.0034 [95% CI -0.012 to 0.0054]) from 2012 to 2015. Thirty-day post-discharge mortality did not change among patients hospitalized for HF (0.00 [95% CI -0.011 to 0.010), AMI (-0.006 [95% CI -0.015 to 0.002), or pneumonia (-0.004 [95% CI -0.013 to 0.005]) (Figure 2).”


“However, there is reason to believe that the increasing use of ED visits for post-discharge care may be problematic. Prior data suggest that hospitals that tend to manage patients in EDs rather than admitting them for an inpatient stay have higher rates of early death after discharge.15 Although we observed no change in 30-day post-discharge mortality for target conditions during the HRRP (2012 to 2015), several recent analyses have found that the implementation of the HRRP was associated with an increase in 30-day post-discharge mortality among patients hospitalized for HF and pneumonia compared with pre-HRRP (pre-2010) trends, and that this increase was concentrated entirely among patients not readmitted after discharge.6 16-18 Whether intensified efforts to manage returning patients in EDs and

observation units can explain increases in mortality observed in the years that preceded our study period is an important area for further research.”






Response to Reviewer #4

Comments: Thank you for giving me the opportunity to review "Thirty-Day Hospital Revisits After Discharge for Conditions Targeted by Readmission Measures in the United States: National Retrospective Analysis" (MS BMJ-2019-049877).

The authors present findings from a descriptive study examining trends over three and a half years in hospital revisits, encompassing ED treat-and-discharge visits, observation stays, and hospital readmissions. They find that, although readmissions per 100 patient discharges have declined, ED visits and observations per 100 patient discharges have increased, suggesting that hospitals have responded to Hospital Readmissions Reduction Program (HRRP) incentives by shifting their re-visiting patients away from readmissions and towards ED and observation stays.

The particular strengths of the paper are the comprehensive data used by the authors and the significance of the research question they are asking. A weakness of the paper is that, given the data, the authors could push their analyses a bit further to elucidate mechanisms and heterogeneity in responses.

Major comments: 1. The authors present an interesting finding (shifts from readmissions to ED and observational stays) that seems appropriate for a brief report but perhaps a little on the skinny side for a full-fledged article. Here are some ways they could unpack their findings a bit further: 1.1. Can they cluster or stratify by hospital-types or hospitals? Basic characteristics like bed size categories, urban-rural, private vs. non-profit, % Medicare patients, and % managed care patients might be helpful for identifying patterns related to conjectured HRRP responses. 1.2. Related to 1.1, can they analyze this at the hospital level (each hospital contributes according to the # of admissions occurring at that hospital)? 1.3. Related to 1.1 and 1.2, which hospitals would have been expected to have been penalized by HRRP (i.e. prior to HRRP, being closer to the penalty threshold) and do we see greater shifts for these hospitals? This would support their claim that hospitals are responding to HRRP, not by improving in the care provided during the index admission but by gaming the classification of revisits. We thank the reviewer for these constructive suggestions. As per the reviewer’s suggestion, we have pushed our analyses further to strengthen the manuscript. We have performed several additional analyses that were requested by the editorial team that continue to focus on national, rather than hospital-level trends. First, to provide more balance and depth, we now evaluate changes in post-discharge mortality across our study period (Figure 2). Second, we have performed additional analyses to evaluate whether trends in 30-day total hospital revisits (and each type of visit) have changed differentially among patient subgroups. These findings are shown below. The primary aim of our study was to characterize the discordant trends in 30-day readmissions and 30-day hospital revisits on a national scale, as policymakers have primarily only focused on the former. We do agree with the reviewer that evaluating within hospital associations between changes in readmissions and ED/observation stays (stratified by penalty and non-penalty hospitals),

though beyond the scope of the current analysis, is an important and interesting area for future study.







(95% CI)

Men Women P value



C. Race





P value for difference in monthly rate of change among subgroups.

a Among patients with multiple hospital revisits within 30-days of discharge, only a single revisit for each type of encounter was counted. For “any hospital revisit,” only a single encounter of any type after discharge was counted.b Among patients with multiple hospital revisits within 30-days of discharge, all visits for each type of encounter were counted.






2. I am not clear on the purpose and utility behind the methods that use index hospitalizations for the year 2014 to estimate the "potential outcomes" for the outcome measures between January 2012 and September 2015 for what looks like a straightforward descriptive paper. What exactly is the treatment requiring the use of the potential outcomes approach and why fix to a specific year? Perhaps they are constrained by the granularity of the data in ways that haven't been made clear in the paper? Thank you for raising this point. We chose the cohort of 2014 as the reference population because it was the most recent year that included a full 12 calendar months of index hospitalizations. The reason to use the concept of potential outcomes is that we need to

adjust for case-mix differences over time. Therefore, we are technically dealing with bias, at least to the same extent as a typical causal inference problem using observational data. We also need to account for the sampling variation in this analysis. Therefore, this paper deals with two major statistical inferential issues: bias and variation. In this sense, it is not descriptive.

2.1. Further elaboration of the rationale and citations in the biostatistics literature would be helpful. This as well as 2.2 could be done in the Appendix. 2.2. Specific modeling equations (with time indices, especially with respect to the calculation involving data from 2014) at each step in the protocol would clarify things. Thank you for these suggestions. In addition to Detailed Statistical Methods described in our appendix, we have now added a section labeled Inferential Strategy to address the reviewer’s comments.

Appendix. Inferential StrategyInferential Strategy

Let be the month indicator taking values 1, 2, …, 45, indicating the 45 months from January 𝑨2012 to September, 2015. Let be the potential outcome (e.g. 30-day post-discharge 𝒀𝒌






Relevant citation: Andrea Rotnitzky, Lingling Li, Xiaochun Li, A note on overadjustment in inverse probability weighted estimation, Biometrika, Volume 97, Issue 4, December 2010, Pages 997–1001, https://doi.org/10.1093/biomet/asq049

https://doi.org/10.1093/biomet/asq049

Minor comments: 1. The authors might want to state more explicitly in the abstract the motivation for the research question they are trying to answer. It wasn't until the introduction that the authors made explicit the reason this question was an important question to answer (namely, that hospitals are gaming the HRRP narrowly-defined incentives). Thank you for this suggestion. We have now changed the Objective section of the Abstract to more explicitly highlight why this question is important to answer:

Page 2 – “Objective: In the United States, thirty-day readmissions are increasingly used to measure quality of care and evaluate hospital performance under value-based payment programs. As a result, readmission rates for targeted conditions have declined modestly. Policymakers have attributed this to improved care transitions and post-discharge care during hospitalization, but it is possible that these declines are occurring because patients who return to the hospital are more likely to be managed in Emergency Departments (EDs) and as observation stays. This study evaluated whether has there has been a change in total hospital revisits within 30 days of discharge after a hospitalization for target conditions.”

2. Reporting the number of patients in addition to the number of hospitalizations in Table 1 would be helpful. Are the percentages reporting % of patients with these characteristics or % of hospitalizations? Thank you. Table 1 reports % of hospitalizations with these characteristics. We have changed the title to ensure this is explicitly clear.

3. Reporting the total number (over all years) of the various hospital revisit measures in the last column of eTable 1 would be useful. Thank you. We have now added total numbers of each type of revisit across all years in eTable 1 of the supplement.

eTable 1. Number of Hospital Revisits Within 30-Days of Discharge from 2012 to 20152012 2013 2014 2015a Total

Total Hospitalizations (n) 832,669 816,014 780,217 609,840 3,038,740

First Revisit b Any revisit 230,043 222,048 216,629 171,394 840,114 ED treat-and-discharge 70,826 69,879 68,897 55,453 265,055 Observation stay 18,989 21,052 21,794 18,248 80,083 Readmission 167,524 158,245 153,157 120,738 599,664

All Revisits c Total revisits 290,209 280,005 274,547 219,649 1,064,410 ED treat-and-discharge 80,827 79,663 78,886 63,818 303,194 Observation stay 19,909 22,117 22,894 19,249 84,169 Readmission 189,473 178,225 172,767 136,582 677,047

a From January 1, 2015 to September 30, 2015. b Among patients with multiple hospital revisits within 30 days of discharge, only the first revisit for each type of encounter was counted.

c Among patients with multiple hospital revisits within 30 days of discharge, all visits for each type of encounter were counted.

4. p. 7, l. 45: "Poison" should be "Poisson" Thank you for catching this error! Now corrected.

5. Great figures! Thank you!!

Reviewer: 5

Comments: The article was well-written and highlighted an important aspect related to 30-day readmissions i.e., illusory aspect in their reduction while organizations have come up with smart strategies to mask the growth in another direction i.e., ED and OBS. I do have two key questions to the authors: 1) What was the rationale behind the selection of the demographic and clinical comorbidity profiles of patients hospitalized in 2014 as a reference for a study cohort that spanned from January 2012 to September 2015? Thank you for raising this important point. We chose 2014 as a reference because it was the most recent year that included a full 12 calendar months of data (in contrast to 2015, for which only 9 months of data were available). We have now made this explicitly clear in the methods sections with the following change:

Page 9: “After constructing separate models for each month (45 models for 45 months), we estimated the mean of the potential outcomes for each respective month using the demographics and clinical comorbidity profiles of patients hospitalized in 2014 as a reference, as this was the most recent year that contained 12 calendar months of data.”

We have now provided more details regarding our analytic approach in the Appendix (entitled “Inferential Strategy”)Appendix. Detailed Statistical Methods

We selected index hospitalizations in the year 2014 and used demographics/clinical comorbidity profiles of this population as a reference to estimate mean potential outcomes (e.g. readmission, ED treat-and-discharge and observation unit stay) in each of the 45 months from January 2012 to September 2015. Let be the potential outcome in 𝒀𝒌 (𝒌 = 𝟏,𝟐,…, 𝟒𝟓)month , be a vector of patient characteristics including demographics, type of index 𝒌 𝑿hospitalization (HF, AMI, and pneumonia) and comorbidities, and be the month indicator. 𝑴Let be the observed outcome. Under the no uncontrolled confounding 𝒀 = ∑𝟒𝟓

𝒌 = 𝟏𝑰(𝑴 = 𝒌)𝒀𝒌

assumption, we have:

. (1)𝑬(𝒀𝒌) = 𝑬[𝑬(𝒀𝒌│𝑿)] = 𝑬[𝑬(𝒀𝒌│𝑿,𝑴 = 𝒌)] = 𝑬[𝑬(𝒀│𝑿,𝑴 = 𝒌)]

In equation (1), the outside equation is with respect to the distribution of for index 𝑿hospitalizations in 2014. Equation (1) suggests that we can fit a regression model using data within month to obtain predicted value , which suggests the following 𝒌 𝑬(𝒀│𝑿,𝑴 = 𝒌)estimator:

. (2)𝑬(𝒀𝒌) = 𝑬[𝑬(𝒀│𝑿,𝑴 = 𝒌)]

Therefore, we fit logistic regression model (for single count of event) and Poisson regression model (for multiple counts of event) within each month , where the dependent variable is 𝒌

the outcomes of our interest and the independent variables are elements of . For the 𝑿Poisson models, we also account for the length of follow-up with an offset (e.g. a patient that died at day 20 after discharge was followed for only 20 days instead of 30). The scale parameter of the Poisson model was estimated using the deviance to be more robust to deviation from Poisson distribution. The fitted models were then used to predict the outcomes of the hospitalizations in 2014, which were then averaged to obtain .𝑬(𝒀𝒌)

Once we obtained , we fitted smooth spline of against , where 𝑬(𝒀𝒌) (𝒌 = 𝟏,𝟐,…,𝟒𝟓) 𝑬(𝒀𝒌) 𝒌the maximum degree of freedom is set at 5. We also fit a separate simple linear regression to summarize the rate of change in event rates.

Inferential StrategyLet be the month indicator taking values 1, 2, …, 45, indicating the 45 months from January 𝑨2012 to September 2015. Let be the potential outcome (e.g. 30-day post-discharge 𝒀𝒌






2) In addition to the #visits, have the authors explored how has the strategy of utilizing ED and OBS impacted the length of stay on the readmit encounters or the days between all encounters? Thank you for this suggestion. We appreciate the importance of this question, particularly given recent interest in potentially using “Days at home” as a quality metric in the United States. Though beyond the scope of our current analysis, we do plan to pursue this question in future studies.

Documents

Editorial Team Comments changes to the text of the ... · From a clinical perspective, understand the effects of more intensive management of patients who return after discharge in