CMS’ Star Ratings Miscalculated

Modern Healthcare is reporting that the Centers for Medicare and Medicaid Services (CMS) has miscalculated hospitals star ratings since they were first released in 2016 (1). Officials at Rush University Medical Center in Chicago exclusively disclosed their analysis and correspondence to Modern Healthcare. The investigators found that instead of evenly weighting the eight measures in the safety of care group, the CMS' star ratings formula relied heavily on one measure— The Patient Safety and Adverse Events Composite, known as PSI 90 —for the first four releases of the ratings and then complication rates from hip and knee replacements for the latest release. The single measure accounted for about 98% of a hospital's performance in the safety group, according to Rush's analysis. The safety group can also greatly influence a hospital's overall star rating, the analysis concluded. Rush's findings likely prompted the CMS to announce this week that it would postpone the July release of its star ratings (1).

The statistical model the CMS uses likely caused the miscalculation. The model, called latent variable modeling, uses scores for seven groups of measures to calculate the star ratings:

1. Mortality
2. Safety of Care
3. Readmission
4. Patient Experience
5. Effectiveness of Care
6. Timeliness of Care
7. Efficient Use of Medical Imaging

The three outcome groups—mortality, safety and readmissions—are each weighted the most at 22% each. Measures within each group are supposed to be evenly weighted to calculate the hospital's performance in that area. Rush's analysis found that the weight given to the PSI-90 measure was much greater than the seven other measures in the safety group. Specifically, PSI-90 was weighted 1,010 times stronger than the catheter-associated urinary tract infections measure, 81 times stronger than the C. difficile infection rates measure, 51 times stronger than the central line-associated bloodstream infection rates measure and 20 times stronger than either the surgical site infection rate measure.

Latent variable modeling changes the weighting and is inappropriate for measuring clinical outcomes, said David Levine, senior vice president of advanced analytics and informatics at Vizient (1). "Given the disproportionate weighting of the safety scores over time, they did not represent a composite measure," said Dr. Omar Lateef, an author of the analysis and Rush's senior vice president and chief medical officer (1). Lateef said he and his colleagues at Rush were alarmed by a rating drop from 5 to 3 stars because they have improved performance on five of the eight safety measures since the December release. " Lateef added that although CMS was initially dismissive of Rush’s concerns that CMS has come around since presented with Rush’s analysis.

CMS announced earlier this week that it was delaying release of the star ratings "to address stakeholders concerns." No date has been set for when the new ratings will be released.

Richard A. Robbins, MD

Editor, SWJPCC

Reference

1. Maria Castellucci M. CMS star rating system has been wrong for two years, health system finds. Modern Healthcare. June 15, 2018. Available at: http://www.modernhealthcare.com/article/20180615/TRANSFORMATION01/180619933?utm_source=modernhealthcare&utm_medium=email&utm_content=20180615-TRANSFORMATION01-180619933&utm_campaign=am (accessed 6/15/18).

Cite as: Robbins RA. CMS' star ratings miscalculated. Southwest J Pulm Crit Care. 2018;16(6):338-9. doi: https://doi.org/10.13175/swjpcc078-18 PDF 

VA Announces Aggressive New Approach to Produce Rapid Improvements in VA Medical Centers

The U.S. Department of Veterans Affairs (VA) announced steps that it is taking as part of an aggressive new approach to produce rapid improvements at VA’s low-performing medical facilities nationwide (1). VA defines its low-performing facilities as those medical centers that receive the lowest score in its Strategic Analytics for Improvement and Learning (SAIL) star rating system, or a one-star rating out of five. The SAIL star rating was initiated in 2016 and uses a variety of measures including mortality, length of hospital stay, readmission rates, hospital complications, physician productivity and efficiency. A complete listing of the VA facilities, their star ratings and the metrics used to determine the ratings is available through the end of fiscal year 2017 (2). Based on the latest ratings, the VA currently has 15 one-star facilities including Denver, Loma Linda, and Phoenix in the Southwest (Table 1).  

Table 1. VA facilities with one-star ratings (1).

1. Big Spring (Texas)
2. Denver (Colorado)
3. Dublin (Georgia)
4. El Paso (Texas)
5. Jackson, (Mississippi)
6. Hampton (Virginia)
7. Harlingen (Texas)
8. Loma Linda (California)
9. Memphis (Tennessee)
10. Murfreesboro (Tennessee)
11. Nashville (Tennessee)
12. Phoenix (Arizona)
13. Roseburg (Oregon)
14. Walla Walla (Washington).
15. Washington (DC)

The steps VA is taking to produce rapid improvements at its low-performing facilities include (Table 2):

Table 2. VA steps to produce rapid improvements at low-performing facilities (1).

1. Central, national accountable leadership – VA has designated Dr. Peter Almenoff, Director of VA’s Office of Reporting, Analytics, Performance, Improvement and Deployment (RAPID) Healthcare Improvement Center, to oversee improvement at each of the centers.
2. Comprehensive analysis and identification of improvement targets – VA is employing a new initiative, known as Strategic Action Transformation (STAT), that uses a rigorous and formal approach based on clinical performance indicators to identify vulnerabilities in each low-performing facility and set specific targets for improvement.
3. Provision of national resources for improvement – VA’s RAPID team of experts will use sophisticated statistical tools to track the progress of improvement against these targets, and, where warranted, will dispatch a team of expert improvement coaches quickly to the medical centers to assist them in meeting the goals.
4. Accountability for results –VA’s Central Office will review each of the facilities quarterly, and if the facilities fail to make rapid substantial progress in their improvement plan, VA leadership will take prompt action, including changing the leadership of the medical center.

VA secretary David Shulkin stated that “President Trump has made it clear that our Veterans deserve only the best when it comes to their healthcare, and that’s why we are focusing on improving our lowest performing facilities nationwide” (1). The VA recently removed the Roseburg Oregon VA Medical Center director who was accused of manipulating hospital admissions to improve the hospital’s rating (3). Almenoff, the overseer of improvement, was transferred from his position as the VA Integrated Network 15 director in 2008 when the Marion VA came under fire for substandard care raising concerns from several Illinois legislators, including the then junior senator from Illinois, Barack Obama (4).

A major hurdle will be for the VA to hire sufficient staff to improve care. As of the end of June, the VA reported 35,554 job vacancies system-wide, and VA Secretary David Shulkin has cited challenges with hiring doctors and nurses, particularly mental health care professionals (5). The agency set a goal to hire 1,000 mental health care workers in 2017. The VA hired 900 last year, but lost 945. The Veterans Access, Choice and Accountability Act of 2014 appropriated several billion dollars to the VA but this apparently did not lead to hiring of sufficient healthcare providers.

Richard A. Robbins, MD

Editor, SWJPCC

References

1. VA Office of Public and Intergovernmental Affairs. VA announces aggressive new approach for low-performing medical centers. February 1, 2018. Available at: https://www.va.gov/opa/pressrel/pressrelease.cfm?id=4004 (accessed 2/14/18).
2. US Department of Veterans Affairs. Quality care. Available at: https://www.va.gov/QUALITYOFCARE/measure-up/Strategic_Analytics_for_Improvement_and_Learning_SAIL.asp (accessed 2/14/18).
3. Phillips D. Director of veterans hospital accused of manipulating ratings is replaced. New York Times. February 1, 2018. Available at: https://www.nytimes.com/2018/02/01/us/veterans-roseburg-director.html (accessed 2/14/18).
4. Durbin D, Obama B, Costello J, Shimkus J. Letter to The Honorable James B. Peake, M.D., Secretary of Veterans' Affairs. July 23, 2008. Available at: https://votesmart.org/public-statement/363179/letter-to-the-honorable-james-b-peake-md-secretary-of-veterans-affairs?flavour=mobile&utm_source=votesmart&utm_medium=mobile-link&utm_campaign=flavourswitch#.WoR3Dq6nGUk (accessed 2/14/18).
5. Wentling N. Federal unions march on VA headquarters to protest staffing shortages. Stars and Stripes. February 13, 2018. Available at: https://www.stripes.com/federal-unions-march-on-va-headquarters-to-protest-staffing-shortages-1.511543 (accessed 2/14/18).

Cite as: Robbins RA. VA announces aggressive new approach to produce rapid improvements in VA medical centers. Southwest J Pulm Crit Care. 2018;16(2):91-3. doi: https://doi.org/10.13175/swjpcc034-18 PDF