CMS’ Star Ratings Miscalculated

Modern Healthcare is reporting that the Centers for Medicare and Medicaid Services (CMS) has miscalculated hospitals star ratings since they were first released in 2016 (1). Officials at Rush University Medical Center in Chicago exclusively disclosed their analysis and correspondence to Modern Healthcare. The investigators found that instead of evenly weighting the eight measures in the safety of care group, the CMS' star ratings formula relied heavily on one measure— The Patient Safety and Adverse Events Composite, known as PSI 90 —for the first four releases of the ratings and then complication rates from hip and knee replacements for the latest release. The single measure accounted for about 98% of a hospital's performance in the safety group, according to Rush's analysis. The safety group can also greatly influence a hospital's overall star rating, the analysis concluded. Rush's findings likely prompted the CMS to announce this week that it would postpone the July release of its star ratings (1).

The statistical model the CMS uses likely caused the miscalculation. The model, called latent variable modeling, uses scores for seven groups of measures to calculate the star ratings:

1. Mortality
2. Safety of Care
3. Readmission
4. Patient Experience
5. Effectiveness of Care
6. Timeliness of Care
7. Efficient Use of Medical Imaging

The three outcome groups—mortality, safety and readmissions—are each weighted the most at 22% each. Measures within each group are supposed to be evenly weighted to calculate the hospital's performance in that area. Rush's analysis found that the weight given to the PSI-90 measure was much greater than the seven other measures in the safety group. Specifically, PSI-90 was weighted 1,010 times stronger than the catheter-associated urinary tract infections measure, 81 times stronger than the C. difficile infection rates measure, 51 times stronger than the central line-associated bloodstream infection rates measure and 20 times stronger than either the surgical site infection rate measure.

Latent variable modeling changes the weighting and is inappropriate for measuring clinical outcomes, said David Levine, senior vice president of advanced analytics and informatics at Vizient (1). "Given the disproportionate weighting of the safety scores over time, they did not represent a composite measure," said Dr. Omar Lateef, an author of the analysis and Rush's senior vice president and chief medical officer (1). Lateef said he and his colleagues at Rush were alarmed by a rating drop from 5 to 3 stars because they have improved performance on five of the eight safety measures since the December release. " Lateef added that although CMS was initially dismissive of Rush’s concerns that CMS has come around since presented with Rush’s analysis.

CMS announced earlier this week that it was delaying release of the star ratings "to address stakeholders concerns." No date has been set for when the new ratings will be released.

Richard A. Robbins, MD

Editor, SWJPCC

Reference

1. Maria Castellucci M. CMS star rating system has been wrong for two years, health system finds. Modern Healthcare. June 15, 2018. Available at: http://www.modernhealthcare.com/article/20180615/TRANSFORMATION01/180619933?utm_source=modernhealthcare&utm_medium=email&utm_content=20180615-TRANSFORMATION01-180619933&utm_campaign=am (accessed 6/15/18).

Cite as: Robbins RA. CMS' star ratings miscalculated. Southwest J Pulm Crit Care. 2018;16(6):338-9. doi: https://doi.org/10.13175/swjpcc078-18 PDF 

ACGME Proposes Dropping the 16 Hour Resident Shift Limit

The Accreditation Council for Graduate Medical Education (ACGME) is proposing that first-year residents would no longer be limited to 16-hour shifts during the 2017-2018 academic year under a controversial proposal released today (1). Instead, individual residency programs could assign first-year trainees to shifts as long as 28 hours, the current limit for all other residents. The 28-hour maximum includes 4 transitional hours that's designed in part to help residents improve continuity of care. The plan to revise training requirements does not change other rules designed to protect all residents from overwork. including the maximum80 hours per week.

The ACGME capped the shifts of first-year residents at 16 hours in 2011 as a part of an ongoing effort to make trainee schedules more humane and avoid clinical errors caused by sleep deprivation. ACGME CEO Thomas Nasca, MD, told Medscape Medical News that the problem arises largely from first-year residents not being on the same schedule as supervising residents and others on their "home" educational team (2). On a 16-hour clock, first-year residents can end up working under relative strangers, said Dr Nasca. "The lack of synchronization is very disruptive." The solution, he said, is putting everyone on the same clock.

The ACGME touts a study published in the New England Journal of Medicine in February showing that longer shifts and less rest in between for surgical residents did not affect the rate of serious complications or surgical fatalities (3). A review in 2014 suggested that patient outcomes might be worse with the restrictions (4).

Both the American Medical Student Association, the Committee of Interns and Residents, and Public Citizen oppose the move. The ACGME proposal will go to the group's board of directors for a final decision after a 45-day comment period. More information on the proposal is available for download at https://www.acgme.org/. Comments can be submitted to cprrevision@acgme.org.

References

1. ACGME. ACGME task force presents new residency training requirements for public comment. November 4, 2016. Available at: https://www.acgme.org/Portals/0/PDFs/CPRNewsRelease_Fall2016_FINAL.pdf (accessed 11/4/16).
2. Lowes R. Let first-year residents work longer shifts, ACGME proposes. Medscape. November 4, 2016. Available at: http://www.medscape.com/viewarticle/871432?nlid=110468_3901&src=wnl_newsalrt_161104_MSCPEDIT&uac=9273DT&impID=1228495&faf=1 (accessed 11/4/16).
3. Bilimoria KY, Chung JW, Hedges LV, et al. National cluster-randomized trial of duty-hour flexibility in surgical training. N Engl J Med. 2016 Feb 25;374(8):713-27. TU[CrossRef]UTH HTU[PubMed]UT
4. Ahmed N, Devitt KS, Keshet I, et al. A systematic review of the effects of resident duty hour restrictions in surgery: impact on resident wellness, training, and patient outcomes. Ann Surg. 2014 Jun;259(6):1041-53. HTU[CrossRef]UTH HTU[PubMed]UT 

Cite as: Robbins RA. ACGME proposes dropping the 16 hour resident shift limit. Southwest J Pulm Crit Care. 2016;13(5):216-7. doi: http://dx.doi.org/10.13175/swjpcc113-16 PDF