Telemedicine Using Stationary Hard-Wire Audiovisual Equipment or Robotic Systems in Critical Care: A Brief Review

Nidhi S. Nikhanj, MD^1,2

Robert A. Raschke, MD^1,2

Robert Groves, MD^1,2

Rodrigo Cavallazzi, MD³

Ken S. Ramos, MD¹

¹Arizona College of Medicine-Phoenix

Phoenix, AZ USA

²Banner University Medical Center-Phoenix

Phoenix, AZ USA

³University of Louisville School of Medicine

Louisville, KY USA

A shortage of critical care physicians in the United States has been widely recognized and reported (1). Most intensive care units (ICUs) do no not have a formally-trained intensivist in their staff despite compelling evidence that high-intensity intensivist staffing leads to better patient outcomes (1,2). Critical care telemedicine is one potential solution that has expanded rapidly since its inception in 2000 (3). In its simplest form, telemedicine leverages audiovisual technology and the electronic medical record to provide remote two-way communication between a physician and a patient. Current telemedicine models differ by the type of hardware facilitating remote audiovisual interaction, the location of the provider, and the type of patient-care service provided. We collectively have experience with several of these models and feel that future telemedicine programs will likely integrate the most advantageous aspects of each with an increasing role for telemedicine robotics.

The dominant current model for providing critical care telemedicine in large healthcare systems utilizes stationary hard-wired audiovisual equipment linking each ICU room to a centralized control location (4). Typically, this control center provides surveillance of a large number of patients using computerized decision support software linked to the EMR – a single physician can cover approximately 100 patients with the appropriate support infrastructure. This model also provides the ability to remotely “round” on ICU patients and to quickly respond to questions posed by nursing or medical emergencies across a broad geographic range. This approach requires a high up-front capital cost approximated at 50-100K per hospital bed covered (5).

Data supporting the benefit of this model of ICU telemedicine has been mixed, but several considerations are important in appraising the literature. A double-blinded RCT for ICU telemedicine intervention is not feasible. Heterogeneity in clinical workflows and staffing models across the country should be considered when assessing the internal validity and generalizability of published studies. For instance, Thomas and colleagues concluded that a telemedicine ICU service resulted in no overall improvement in mortality or length of stay (LOS) (6), but the tele-intensivists in the study were limited by only being allowed to intervene in the care of less than a third of the study patients. Nassar and colleagues published a negative study in a healthcare system in which resident and attending physicians were already available in-house for overnight patient care (7). Likely, the potential benefit of a telemedicine program can be optimized in a clinical setting in which other physicians are not physically available at the locality 24/7 and telemedicine intensivists are allowed to appropriately intervene when indicated.

Despite these difficulties, there is a growing body of evidence that suggests a centralized telemedicine ICU model is effective in a number of areas including: improvements in compliance with evidence based practices (8, 9), increased job satisfaction of ICU nurses (10) and reduction in the cost of care of the sickest patients in the institutional setting (11). Other studies suggest that a telemedicine platform can reduce mortality and LOS by allowing for earlier intensivist involvement, promoting adherence to best practices, shortening alarm response times and improving access to ICU performance data that can be used to drive continuous quality improvement (12,13).

Commercially available telemedicine robots are mobile units equipped with a digital camera, microphone and monitor screen that provides two-way audiovisual communications with the control center via a wireless internet connection (14). Telemedicine robots can be operated with much lower initial capital costs - for instance, an ICU group at a large acute care hospital might provide coverage at a rural healthcare setting using a single robot (15). Such a system can be used for daily rounding or for reactive consultation. Like hard-wired systems, telemedicine robots have been shown to be well accepted by providers (16) and patients (17), and their use has been associated with reduced ICU length-of-stay and decreased delay in response to clinical events by the physician (18).

Telemedicine robotic systems have several disadvantages – they do not provide large-scale EMR surveillance leveraging computerized decision support logic and they are significantly less efficient than hard-wired systems for high-volume patient care since they have to physically relocate from patient room to patient room.  However, unique capabilities of telemedicine robots are being developed that cannot be duplicated by hard-wired systems. Telemedicine robots can be equipped with a digital stethoscope (19). They can perform physical examination elements that require tactile communication – such as the determination of the Glasgow coma scale (20). A robotic arm can be used to remotely perform point-of-care ultrasonography. This has been successfully operationalized for cardiac, abdomino-pelvic, and vascular indications (21,22). Telemedicine robots have been developed that can place peripheral or central venous catheters (23). The development of surgical robots that incorporate tomographic capability and that can perform battlefield stabilization procedures in either autonomous or teleoperative modes (24) provide a glimpse of the potential for telemedicine robots in the ICU.

Although healthcare systems currently implementing telemedicine services will likely choose either a hard-wired or a robotic model – largely based on cost and the volume of required services - we believe the optimal telemedicine system of the future will and should incorporate both technologies. Real-time data acquisition coupled with ready access to timely interventions constitute the basis for faster deployment of precision health care strategies in the ICU setting.

References

1. Kelley MA, Angus D, Chalfin DB, Crandall ED, et al. The critical care crisis in the United States: A report from the profession. Chest. 2004;125:1514-7. [CrossRef] [PubMed]
2. Pronovost PJ, Angus DC, Dorman T, Robinson KA, et al. Physician staffing patterns and clinical outcomes in critically ill patients. JAMA. 2002;288:2151-62. [CrossRef] [PubMed]
3. Rosenfeld BA, Dorman T, Breslow MJ, et al. Intensive care unit telemedicine: alternate paradigm for providing continuous intensivist care. Crit Care Med. 2000;28:3925-31. [CrossRef] [PubMed]
4. Kahn JM, Cicero BD, Wallace DJ, Iwashyna TJ. Adoption of intensive care unit telemedicine in the United States. Crit Care Med. 2014;42:362-8. [CrossRef] [PubMed]
5. Kumar G, Falk DM, Bonello RS, et al. The costs of critical care telemedicine programs: A systematic review and analysis. Chest. 2013;143:19-29. [CrossRef] [PubMed]
6. Thomas EJ, Lucke JF, Wueste L. Association of telemedicine for remote monitoring of intensive care patients weith mortality, complications and length of stay. JAMA. 2009;302:2671-78. [CrossRef] [PubMed]
7. Nassar BS, Vaughan MS, Jiang L, Reisinger HS, et al. Impact of an intensive care unit telemedicine program on patient outcomes in an integrated health care system. JAMA Intern Med. 2014;174:1160-7. [CrossRef] [PubMed]
8. Ventataraman R, Ramakrishnan N. Outcomes related to telemedicine in the intensive care Unit. Crit Care Clinics 2015;31:225-37. [CrossRef] [PubMed]
9. Youn BA. ICU process improvement using telemedicine to enhance compliance and documentation for the ventilator bundle. Chest. 2006;130:(meeting abstracts) 226S-c.
10. Hoonakker PL, Carayon P, McGuire K, et al. Motivation and job satisfaction of tele-ICU nurses. J Crit Care. 2013;28:890-901. [CrossRef] [PubMed]
11. Franzini L, Sail KR, Thomas EJ, et al. Costs and cost-effectiveness of a telemedicine intensive care unit program in six intensive care units in a large health care system. J Crit Care. 2011;26:329e1-6. [CrossRef] [PubMed]
12. Lilly CM, Cody S, Zhao H. Hospital mortality, length of stay and preventable complications among critically ill patients before and after tele-ICU reengineering of critical care processes. JAMA. 2011;305:2175-83. [CrossRef] [PubMed]
13. Lilly CM, Zubrow MT, Kempner KM, Reynolds H, et al. Critical Care telemedicine: Evolution and state of the art. Crit Care Med. 2014;42:2429-36. [CrossRef] [PubMed]
14. Chung KK, Grathwohl KW, Poropatich RK, Wolf SE, et al. Robotic telepresence: Past present and future. Journal of Cardiothoracic and Vascular Anesthesia. 2007;21:593-6. [CrossRef] [PubMed]
15. Murray C, Ortiz E, Kubin C. Application of a robot for critical care rounding in small rural hospitals. Crit Care Nurs Clin North Am. 2014;26:477-85. [CrossRef] [PubMed]
16. Reynolds EM, Grujovski A, Wright T, Foster M, Reynolds HN. Utilization of robotic remote presence technology within North American intensive care units. Telemedicine and e-health. 2012;18:507-15. [CrossRef] [PubMed]
17. Sucher JF, Todd SR, Jones SL, Throckmorton T, et al. Robotic telepresence: A helpful adjunct that is viewed favorably by critically ill surgical patients. Am J Surg. 2011;202:843-7. [CrossRef] [PubMed]
18. Vespa PM, Miller C, Hu X, Nenov V, et al. Intensive care unit robotic telepresence facilitates rapid physician response to unstable patients and decreased cost in neurointensive care. Surgical Neurology. 2007;67:331-7. [CrossRef] [PubMed]
19. Lakhe A, Sodhi I, Warrier J, Sinha V. Development of digital stethoscope for telemedicine. J Med Eng Technol. 2016;40:20-4. [CrossRef] [PubMed]
20. Adcock AK, Kosiorek H, Parich P, Chauncey A, Wu Q, Demaerschalk BM. Reliability of robotic telemedicine for assessing critically ill patients with the full outline of unresponsiveness score and Glasgow coma scale. Telemed J E Health. 2017 Jan 13. [CrossRef] [PubMed]
21. Avgousti S, Panayides AS, Jossif AP, Christoforou EG, et al. Cardiac ultrasonography over 4G wireless networks using a tele-operated robot. Healthc Technol Lett. 2016;3:212-7. [CrossRef] [PubMed]
22. Georgescu M, Sacccomandi A, Baudron B, Arbeille PL. Remote sonography in routine clinical practice between two isolated medical centers and the university hospital using a robotic arm: A 1-year study. Telemed J E Health. 2016;22:276-81. [CrossRef] [PubMed]
23. Kobayashi Y, Hong J, Hamano R, Okada K, Fujie MG, Hashizume M. Development of a needle insertion manipulator for central venous catheterization. Int J Med Robot. 2012;8(1):34–44. [CrossRef] [PubMed]
24. Garcia P, Rosen J, Kapoor C, Noakes M, et al. Trauma Pod: a semi-automated telerobotic surgical system. Int J Med Robot. 2009;5:136-46. [CrossRef] [PubMed]

Cite as: Nikhanj NS, Raschke RA, Groves R, Cavallazzi R, Ramos KS. Telemedicine using stationary hard-wire audiovisual equipment or robotic systems in critical care: a brief review. Southwest J Pulm Crit Care. 2017;15(1):50-3. doi: https://doi.org/10.13175/swjpcc087-17 PDF

Clinical Performance of an Interactive Clinical Decision Support System for Assessment of Plasma Lactate in Hospitalized Patients with Organ Dysfunction

Robert A. Raschke, MD MS

Hargobind Khurana, MD 

Huw Owen-Reece, MBBS 

Robert H. Groves Jr, MD

Steven C. Curry, MD

Mary Martin, PharmD

Brenda Stoffer, RN BSN

Banner University Medical Center Phoenix

Phoenix, AZ USA

Abstract

Purpose: Elevated plasma lactate concentration can be a useful measure of tissue hypo-perfusion in acutely deteriorating patients, focusing attention on the need for urgent resuscitation. But lactate is not always assessed in a timely fashion in patients who have deteriorating vital signs. We hypothesized that an electronic medical record (EMR)-based decision support system could interact with clinicians to prompt assessment of plasma lactate in appropriate clinical situations in order to risk stratify a population of inpatients and identify those who are acutely deteriorating in real-time.

Methods: All adult patients admitted to our hospital over a three month period were monitored by an EMR-based lactate decision support system (lactate DSS) programmed to detect patients exhibiting acute organ dysfunction and engage the clinician in the decision to order a plasma lactate concentration. Inpatient mortality was determined for the five risk categories that this system generated, and chart review was performed on a high-risk subgroup to describe the spectrum of bedside events that triggered the system logic.

Results: The lactate DSS segregated inpatients into five strata with mortality rates of 0.8% (95%CI:0.6-1.0%); 2.7% (95%CI:1.0-4.4%); 7.9% (95%CI: 6.0-10.1%), 13.0% (95%CI: 9.0-17.8%) and 42.1% (95%CI: 32.0-52.4%), achieving a discriminant accuracy of 80% (95%CI:76-84%) by AUROC for predicting inpatient mortality. Classification into the two highest risk strata had a positive predictive value for detecting acute life-threatening clinical events of 54% (95%CI: 41.5-66.5%).

Conclusions: Our lactate decision support system is different than previously-described computerized “early warning systems”, because it engages the clinician in decision-making and incorporates clinical judgment in risk stratification. Our system has favorable operating characteristics for the prediction of inpatient mortality and real-time detection of acute life-threatening deterioration.

Introduction

Over 700,000 deaths occur annually in U.S. hospitals (1). Sepsis accounts directly for 37% and indirectly for 56% of these deaths (2). Other common causes of inpatient mortality such as acute hemorrhage and venous thromboembolism (3) share certain early clinical findings with sepsis, in that they may present with deterioration of vital signs and biochemical variables before life-threatening manifestations become obvious (4). Recognition of these findings provides an opportunity for early intervention, which has been shown to improve mortality (5,6). Studies have shown that failure to rapidly recognize acute clinical deterioration is one of the most common root causes of preventable inpatient mortality (4,8).

Early warning systems (EWSs) are a type of clinical decision support system (CDSS) utilized to provide surveillance of hospitalized patients in order to alert clinicians when a patient has findings associated with acute deterioration (19). These typically monitor for abnormal vital signs or laboratory evidence of organ dysfunction, but have included many other types of clinical and laboratory variables (20-23). Modern EWSs utilize logistic regression to weight up to 36 different independent variables and yield highly stratified risk scores (24-26).

We had previous experience developing a simple EWS that triggered when at least two systemic inflammatory response syndrome (SIRS) criteria plus at least one of 14 acute organ dysfunction (OD) parameters was detected. Although this system references SIRS it was found to be nonspecific for sepsis (27), and was subsequently employed in our healthcare system to identify patients deteriorating in real-time regardless of the cause. Subsequent research showed that our SIRS/OD alert system was triggered during the course of 19% of admissions, and that patients who triggered the alert had an odds ratio of 30.1 (95% CI: 26.1-34.5) for inpatient mortality (28). We hypothesized that this SIRS/OD alert system could be used to identify high risk patients who might be further risk-stratified by obtaining a plasma lactate concentration.

Elevated plasma lactate concentration is a particularly useful biochemical marker of acute decompensation. Hyperlactemia is pathophysiologically associated with acute tissue hypoperfusion, and clinically associated with organ dysfunction and mortality (7-11). Hyperlactemia is also associated with the need for urgent clinical interventions such as transfusion and urgent surgery in trauma patients (13,14), and resuscitation of medical patients with sepsis or other life-threatening illnesses (5,15). Lactate assessment is integral to the definition of sepsis (7,16), and an essential component of the Surviving Sepsis Campaign sepsis resuscitation bundle (6). Lactate assessment is integral to achieving sepsis bundle compliance as defined by the Centers for Medicare and Medicaid Services (CMS), which has mandated participating hospitals to report as a measure of quality of care. However, lactate is only ordered about half the time that it ought to be in patients with severe sepsis and septic shock (17,18). To our knowledge, only one previously reported EWS incorporates lactate assessment (29), but this system passively utilized lactate concentration results obtained on admission from the emergency room and was not used for surveillance during hospitalization.

We sought to use our SIRS/OD alert system to actively trigger lactate assessment to identify patients suffering from sepsis or any other life-threatening disease process requiring immediate intervention during hospitalization. We hypothesized that the resulting “lactate decision support system” (lactate DSS) would provide inpatient mortality risk stratification with high discriminant accuracy, and detect acute life-threatening events with high positive predictive value compared to contemporary EWSs.

A lactate DSS with these favorable characteristics could theoretically be used to guide emergent interventions in an effort to save lives, although it was not our aim at this time to perform an interventional trial. The specific aims of this study were to pilot an interactive lactate DSS in our healthcare system, and to calculate its discriminant accuracy for mortality risk stratification, and its positive predictive value as a real-time early warning system.

Methods

We prospectively studied a cohort of all adult inpatients admitted to Banner-University Medical Center - Phoenix, a 650-bed academic hospital in Phoenix Arizona, during the first quarter of 2014. Our research was part of an ongoing system-level patient safety project and was approved by our Institutional Review Board.

The decision support logic was developed at Banner Health using Discern Expert^® (Cerner Corporation, North Kansas City MO, USA). The lactate decision support system (lactate DSS) monitored each patient in our EMR for vital signs and laboratory results consistent with SIRS and organ dysfunction, using criteria derived from the standard definition of sepsis (5-7) (Table 1).

Table 1. Lactate DSS trigger logic

If criteria for SIRS and organ dysfunction overlapped in any eight-hour window, the lactate DSS was triggered to respond. An electronic notification was generated to the patient’s nurse and physician alerting them to the possibility of acute clinical deterioration suggested by SIRS and organ dysfunction, and recommending evaluation and resuscitation if appropriate. Decision support included automatic generation of an order for a STAT plasma lactate if one was not previously ordered by the clinician, interactively prompting the clinician to cancel it if they felt it was unnecessary.

Adult admissions during the three-month study period and subsequent inpatient mortality were enumerated using our hospital’s general financial database: MedSeries4^® (Siemens Corporation, Washington DC). Although some patients triggered the lactate DSS multiple times over the course of their hospital stay, only the first trigger event was included in our analysis.

Inpatient mortality rates with ninety-five percent confidence intervals were calculated for each of five subgroups: 1) patients who did not exhibit SIRS and organ dysfunction during their hospitalization and therefore did not trigger a lactate DSS response; 2) patients who triggered a lactate DSS response, for whom a DSS-generated lactate order was cancelled by their clinician; 3) patients who triggered the lactate DSS and had a lactate concentration <2.2 mmol/L (normal for our laboratory); 4) patients who triggered the lactate DSS and had an elevated lactate of 2.2-3.9 mmol/L; and 5) patients who triggered the lactate DSS and had a highly elevated lactate >4.0 mmol/L.

It was our hypothesis that mortality in patients who triggered the lactate DSS logic would be equivalent whether the clinician chose not to cancel a DSS-generated lactate order, or the clinician had already entered a lactate order themselves. Therefore, we classified patients into the subgroups above regardless of whether their lactate order was DSS-generated or entered independently by the clinician. In order to confirm the validity of this hypothesis, the mortality rate of all patients with any lactate concentration result (the sum of groups 3, 4 and 5 above), and mortality rates within each lactate concentration strata, were separately analyzed to determine if mortality depended on the method of lactate order entry.

Stratified likelihood ratios and the area under the receiver operating curve (AUROC) generated using the five subgroups described above were calculated to determine the discriminant accuracy of the lactate DSS for the outcome of inpatient mortality.

A subgroup analysis was performed of all study patients with an elevated lactate >2.2 mmol/L (above the upper limit of normal range at our laboratory) detected by a DSS-generated lactate order during the first six weeks of the study. These patients’ charts were reviewed in order to characterize the acute clinical events that triggered a lactate DSS response in this subgroup of patients. A physician researcher reviewed progress notes, laboratory and microbiology results at the time of system activation, and for 72 hours afterwards to make this determination. Patient were determined to be suffering an acute life-threatening clinical event if a new-onset or rapidly-progressive disease process was present at the time the lactate DSS was triggered that required emergent treatment with any one of the following: >1 L intravenous fluid resuscitation, vasopressor infusion, >2 units of packed red blood cell transfusion, endotracheal intubation, advanced cardiac life support, or emergent surgical intervention. Minor clinical events included any diagnosis that required initiation of treatment not included in the definition of acute life threatening clinical events above. False alerts were said to have occurred when no evidence was found that the patient was clinically deteriorating in temporal relationship to lactate DSS activation, or within 72 hours. The positive predictive value of the system was calculated for the real-time detection of acute life-threatening clinical events. Microsoft Research and VassarStats^® on-line statistical software were used for statistical calculations.

Results

8,867 adult patients were admitted during our three-month study period. One hundred and ninety-six of 8867 patients (2.2% 95%CI: 1.9-2.5%) died while in the hospital. Seventy percent (138/196) of these inpatient deaths occurred in the 16% (1400/8867) of patients who triggered a lactate DSS response.

Four hundred seventy-nine of 1400 patients who triggered the lactate DSS already had a clinician-ordered lactate. A DSS-generated order for plasma lactate was entered for the remaining 921 patients, but clinicians cancelled 337 of these. DSS-generated lactate orders were resulted for the remaining 584 patients. These patients were merged with 479 patients who had clinician –ordered lactates for the purposes of further analysis after confirmation that mortality did not depend on how the lactate was ordered (Figure 1).

Figure 1. Stratification of inpatients into five subgroups by the lactate DSS.

Patients who did not trigger the lactate DSS logic (n=7467) had a mortality rate of 0.78% (95%CI: 0.58-0.98). Patients who triggered the lactate DSS and for whom a DSS-generated lactate order was cancelled by the clinician (n=337) had mortality of 2.7% (95%CI: 1.0-4.4%). Patients who triggered the lactate DSS and had a lactate concentration in the normal range (< 2.2 mmol/L; n=721) had mortality of 7.9% (95%CI: 6.0-10.1%), and those with elevated lactates of 2.2-3.9 and >4.0 mmol/L (n=247 and n=95) had mortality rates of 13.0% (95%CI: 9.0-17.8%) and 42.1% (95%CI: 32.0-52.4%) respectively (Figure 2).

Figure 2. Inpatient mortality rates (Y-axis: Percent mortality) with 95% confidence intervals for five subgroups of patients stratified by lactate DSS.

The mortality of patients who triggered a lactate DSS response and for whom a lactate concentration was resulted did not depend on whether the order was DSS-generated or entered by the clinician (13.0% versus 12.1% (P=0.71)). Clinician-entered lactate orders were closely temporally related to the onset of organ dysfunction, preceding lactate DSS triggering by < six hours in 52%, <12 hours in 64%, and <24 hours in 75% of cases. Likelihood ratios for mortality in subgroups of patients with lactates <2.2, 2.2-3.9, and >4.0 mmol/L were 6.1 (95%CI: 5.4-6.9), 11.8 (95%CI: 9.5-14.7), and 32.4 (95%CI: 22.0-47.1) respectively.

Five-strata of mortality risk generated by the lactate DSS yielded an AUROC of 0.80 (95% CI: 0.76-0.84) (Figure 3).

Figure 3. Receiver-operating characteristic curve for mortality risk stratification by the lactate DSS.  

Focused chart review was performed on 61 patients who had elevated lactate (>2.2 mmol/L) detected by a DSS-generated lactate order. Thirty-three (54%) were experiencing acute life-threatening clinical events at the time the lactate DSS was triggered. These included 18 episodes of sepsis. Sepsis was due to pneumonia in nine patients, catheter-associated blood stream infection, bowel perforation, cellulitis, ascending cholangitis, endocarditis, liver abscess, cholecystitis, perianal abscess, or an unidentified source. Other acute life threatening clinical events included five cases of acute gastrointestinal hemorrhage, three of acute respiratory failure, and one each of post-operative bleeding, cardiogenic shock, acute liver failure, retroperitoneal bleeding, acute myocardial infarction, subdural hematoma, and cerebral dural sinus thrombosis. Twenty-one (64%) of these events occurred outside the intensive care unit. The positive predictive value of the detection of SIRS, organ dysfunction and elevated lactate by the lactate DSS for acute life-threatening clinical events was 54% (95%CI: 41.5-66.5%).

Ten minor clinical events included anemia, atrial fibrillation, post-op third spacing, transient mild hypotension associated with end stage liver disease, sedation related to narcotics, and dialysis disequilibrium. There were 18 false alerts among patients with SIRS, organ dysfunction and elevated lactate detected by the system. (18/61=29%).

Discussion

Our lactate DSS effectively segregated a population of adult inpatients into five subgroups with increasing inpatient mortality. Clinician engagement was critically important in achieving this result. About a quarter (337/1400) of patients who triggered the lactate DSS (simultaneously exhibited SIRS and organ dysfunction) were doing well enough in their clinician’s opinion that the DSS-generated lactate order was cancelled. Clinicians exercised good judgment in this regard, identifying a subgroup of patients with inpatient mortality rate not significantly higher than the overall mortality of all patients admitted during the study. This supports our decision to incorporate clinician judgment in our risk stratification method.

Approximately half of patients (721/1400) who triggered the lactate DSS turned out to have a normal lactate concentration, yet suffered inpatient mortality ten-times higher than patients who did not trigger the system. This likely represents the independent association between SIRS and organ dysfunction with the risk for mortality (27, 31,32).

One hundred twenty-nine patients over 3 months (14.5 per 1000 patient admissions) triggered the lactate DSS and were found to have an elevated lactate concentration because of a DSS-generated lactate order. These patients had >50% probability of experiencing an acute life-threatening clinical event at the time the lactate DSS was triggered, and subsequently suffered 50% inpatient mortality.

Our lactate DSS is consistent with the new definition of sepsis because it uses organ dysfunction in addition to SIRS criteria (7). As stated in the new definition of sepsis, “Nonspecific SIRS criteria such as pyrexia or neutrophilia will continue to aid in the general diagnosis of infection” (7). Although these criteria are nonspecific, they appear to be relatively sensitive for sepsis (7,27). Our lactate DSS has excellent discriminant accuracy for predicting inpatient mortality (AUROC=0.80). It is comparable to other criteria such SOFA (AUROC = 0.74) and the Logistic Organ Dysfunction System (AUROC=0.75).The five strata into which it segregates patients could further translate into a decision support-guided treatment protocol, directing appropriate real-time interventions such as those proposed in Table 2.

Table 2. Proposed stratified clinical response to lactate DSS.

* Our data indicate that RRT activation would occur about twice a week at our hospital.

Our lactate DSS is different than EWSs because it specifically prompts assessment of plasma lactate in patients exhibiting SIRS and organ dysfunction, rather than simply generating a warning. But a discussion of the operating characteristics of previously reported EWSs is useful for purposes of comparison. A review of 33 EWSs has reported AUROCs ranging from 0.66-0.78 (19). Several more recent EWSs reported AUROCs of 0.81-0.88 (23,24,26,33), but AUROC comparisons are confounded by lack of consensus regarding which clinical outcome to analyze. Authors have variously chosen 24-hour mortality, ICU transfer, and cardiac arrest, among other outcomes (20,23,24). Many EWSs yield highly stratified results, which may increase the AUROC by adding detail to the shape of the ROC curve, but this will not improve clinical discrimination unless each resulting strata has a distinct clinical response. If a EWS is simply used to activate a rapid response team (RRT), the clinically-achievable discriminant accuracy is best described by a polygonal AUROC derived from a single cutoff with two resulting strata (activate the RRT, or do not activate the RRT). This two-strata AUROC will invariably be lower than the highly stratified AUROC that many authors report (23,24,26,33). Our AUROC analysis is based on 5 strata, each of which could reasonably trigger a distinct clinical response (Table 2).

Our lactate decision support system has a positive predictive value (PPV) for acute life-threatening clinical events that is superior to that of our previous “sepsis alert” (27) and to those reported in several reviews of EWSs. One review of 39 EWSs reported PPVs ranging from 13.5-26.1% (34), and another review of 25 systems reported a median PPV of 36.7% with interquartile range 29.3-43.8% (34). PPV was not reported for several of the most elegant and well-studied EWSs (22,23,25,32). From the perspective of bedside clinicians and rapid response team members, the efficiency of an alert system is strongly influenced by the PPV, because a poor PPV translates to frequent false alerts. The PPV is of particularly concern when the pretest probability of the outcome of interest is low, as in the case of inpatient mortality (2% at our hospital). Bayes theory indicates that a test with relatively good AUROC will have a poor PPV if the pretest probability is low enough.

Our study has several limitations. Our sample size is small compared to many contemporary EWS studies. We did not have the resources to perform focused chart reviews on all study patients and therefore had to limit individual case analysis to a subgroup of study patients. Our simple treatment of vital sign abnormalities as markers of SIRS is not as elaborate as in many EWSs. Our study is only hypothesis-generating, whereas several EWSs are well validated (25,32). We cannot provide data on how our alert might change bedside interventions by clinicians. To our knowledge, no study to date has proven that using a computerized decision support system or EWS to trigger rapid clinical intervention actually improves patient outcomes.

Conclusions

We developed an automated decision-support system that prompts assessment of plasma lactate concentration in patients exhibiting SIRS and organ dysfunction. Our lactate decision support system is different than previously-described EWSs because it engages the clinician in decision-making and incorporates clinical judgment into risk stratification. This system has favorable operating characteristics for the prediction of inpatient mortality and for detecting acute life-threatening events in real time. We have proposed a stratified clinical response based on classification of patients into five subgroups by this system that requires further testing, but our current study was not designed to demonstrate a benefit on clinical outcomes. Our lactate DSS has the potential to improve sepsis bundle compliance by helping clinicians appropriately order lactate concentrations in patients deteriorating due to the onset of sepsis – a hypothesis we are currently investigating. It also has potential for easy generalizability, particularly to other healthcare systems that share the same EMR as ours, but requires further refinement and validation.

Author Contributions

All authors were involved in conceptualization, design and implementation of the decision support system described in this manuscript, and in preparation of the manuscript, and all approve of the content of the manuscript and vouch for the validity of the data. We list below additional contributions from several of the authors:

RAR: data analysis and interpretation, main author of initial draft of the manuscript.

HOW: data analysis and interpretation, contribution to discussion/conclusions

HK: directly in charge of design and pilot implementation team for the decision support system, data interpretation, contribution to discussion, conclusions

RHG: data interpretation, contribution to discussion, conclusions

SCC: data analysis and interpretation, contribution to discussion, conclusions. Manuscript editing.

MM: data collection and analysis

BS: data collection and analysis

References

1. Center for Disease Control. Trends in inpatient hospital deaths: National Hospital Discharge Survey: 2000:2010. NCHS Data Brief 118; 2013. [PubMed]
2. Liu V, Escobar G, Greene JD, Soule J, et al. Hospital deaths in patients with sepsis from two independent cohorts. JAMA. 2014;312:90-92. [CrossRef] [PubMed]
3. Nichols L, Chew B. Causes of sudden unexpected death of adult hospital patients. J Hosp Med. 2012;7:706-8. [CrossRef] [PubMed]
4. McGloin H, Adam SK, Singer M. Unexpected deaths and referrals to intensive care of patients on general wards. Are some cases potentially avoidable? J R Coll Physicians Lond. 1999;33:255-9. [PubMed]
5. Rivers E, Nguyen B, Havstad S, Ressler J, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. NEJM. 2001;345:1368-77. [CrossRef] [PubMed]
6. Dellinger RP, Levy MM, Rhodes A, Annane D, et al. Surviving sepsis campaign: International guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41:580. [CrossRef] [PubMed]
7. Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016 Feb 23;315(8):801-10. [CrossRef] [PubMed]
8. National Patient Safety Agency. Safer care for the acutely ill patient: learning from serious incidents. 2007;Report # PSO/5. Available online at: http://www.nrls.npsa.nhs.uk/resources/?EntryId45=59828 (accessed 5/9/17).
9. Gultepe E, Green JP, Nguyen H, Adams J, et al. From vital signs to clinical outcomes for patients with sepsis: a machine learning basis for a clinical decision support system. J Am Med Inform Assoc. 2014;21:315-325. [CrossRef] [PubMed]
10. Jansen TC, van Bommel J, Woodward R, Mulder PG, Bakker J. Association between blood lactate levels, sequential organ failure assessment sub-scores, and 28-day mortality during early and late intensive care unit stay: a retrospective observational study. Crit Care Med. 2009;37:2369-74. [CrossRef] [PubMed]
11. Bakker J, Gris P, Coffernils M, Kahn RJ, Vincent JL. Serial blood lactate levels can predict the development of multiple organ failure following septic shock. Am J Surg. 1996;171:221-6. [CrossRef] [PubMed]
12. Jansen TC, van Bommel J, Bakker J. Blood lactate monitoring in critically ill patients: a systematic health technology assessment. Crit Care Med. 2009;37:2827-39. [CrossRef] [PubMed]
13. Guyette F, Suffoletto B, Castillo JL, Quintero J, Callaway C, Puyana JC. Prehospital serum lactate as a predictor of outcomes in trauma patients: A retrospective observational study. J Trauma. 2011;70:782–6. [CrossRef] [PubMed]
14. Vandromme MJ, Griffin RL, Weinberg JA, Rue LW 3rd, Kerby JD. Lactate is a better predictor than systolic blood pressure for determining blood requirement and mortality: Could prehospital measures improve trauma triage? J Am Coll Surg. 2010;210:861–9. [CrossRef] [PubMed]
15. Jansen TC, van Bommel J, Schoonderbeek FJ, Sleeswijk Visser SL, et al. Early lactate-guided therapy in intensive care unit patients: a multicenter, open-label, randomized controlled trial. Am J Respir Crit Care Med. 2010;182:752-61. [CrossRef] [PubMed]
16. Levy MM, Fink MP, Marshall JC, Abraham E, et al. International Sepsis Definitions Conference. Crit Care Med. 2003;31(4):1250. [CrossRef] [PubMed]
17. Gao R, Melody T, Daniels DF, Giles S and Fox S. The impact of compliance with 6-hour and 24-hour sepsis bundles on hospital mortality in patients with severe sepsis: a prospective observational study. Crit Care. 2005;9:R764–R770. [CrossRef] [PubMed]
18. Levy MM, Dellinger RP, Townsend SR, et al. The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med. 2010; 36: 222–31. [CrossRef] [PubMed]
19. Smith GB, Prytherch DR, Schmidt PE, Featherstone PI. Review and performance evaluation of aggregate weighted "track and trigger" systems. Resuscitation. 2008;77:170-9. [CrossRef] [PubMed]
20. Hodgetts TJ, Kenward G, Vlachonikolis IG, Payne S, Castle N. The identification of risk factors for cardiac arrest and formulation of activation criteria to alert a medical emergency team. Resuscitation. 2002;54:125-31. [CrossRef] [PubMed]
21. Kho A, Rotz D, Alrahi K, Cardenas W, et al. Utility of commonly captured data from an HER to identify hospitalized patients at risk for clinical deterioration. AMIA 2007 symposium proceedings. 404-8.[CrossRef]
22. Howell MD, Donnino M, Clardy P, Talmor D, Shapiro NI. Occult hypoperfusion and mortality in patients with suspected infection. Intensive Care Med. 2007;33:1892-9. [CrossRef] [PubMed]
23. Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, et al. Early detection of impending physiological deterioration among patients who are not in intensive care: Development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7:388-95. [CrossRef] [PubMed]
24. Prytherch DR, Smith GB, Schmidt P, Featherstone PI. ViEWS – towards a national early warning score for detecting adult inpatient deterioration. Resuscitation. 2010;81:932-7. [CrossRef] [PubMed]
25. Bailey TC, Yixin C, Mao Y, Lu C, et al. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med. 2013;8:236-42. [CrossRef] [PubMed]
26. Churpek MM, Yuen TC, Winslow C, Robicsek AA, et al. Multicenter development and validation of a risk stratification tool for ward patients. Am J Resp Crit Care Med. 2014;190:649-55. [CrossRef] [PubMed]
27. Raschke RA, Owen-Reece H, Khurana H, Groves RH Jr, et al. Clinical performance of an automated systemic inflammatory response syndrome (SIRS)/organ dysfunction alert: a system-based patient safety project. Southwest J Pulm Crit Care. 2014;9:223-9. [CrossRef]
28. Khurana, H, Groves RH, Simons MP, Martin M, Stoffer B, et al. Real-time automated continuous sampling of electronic medical records predicts hospital mortality. Am J Med. 2016 Jul;129(7):688-698.e2. [CrossRef] [PubMed]
29. Jo S, Lee JB, Jin YH, Jeong TO, et al. Modified early warning score with rapid lactate level in critically ill medical patients: the ViEWS-L score. Emerg Med J. 2013;30:123-9. [CrossRef] [PubMed]
30. Rangel-Frausto MS, Pittet, D, Costigan M, et al. The natural history of the Systemic Inflammatory Response Syndrome (SIRS): A prospective study. JAMA. 1995;273:117-123. [CrossRef] [PubMed]
31. Matthew M, Churpek F, Zadravecz J, et al. Incidence and prognostic value of the Systemic Inflammatory Response Syndrome and organ dysfunctions in ward patients. Am J Resp Crit Care Med. 2015;192:958-64. [CrossRef] [PubMed]
32. Pittet D, Range-Frausto S, Tarara LN, Lin N, et al. Systemic inflammatory response syndrome, sepsis, severe sepsis and septic shock: incidence, morbidities and outcomes in surgical ICU patients. Intensive Care Med. 1995;21:302-9. [CrossRef] [PubMed]
33. Kellett J, Kim A. Validation of an abbreviated Vitalpac early warning score (ViEWS) in 75,419 consecutive admission to a Canadian regional hospital. Resuscitation. 2012;83:297-302. [CrossRef] [PubMed]
34. Smith GB, Prytherch DR, Schmidt PE, Featherstone PI, et al. A review and performance evaluation of single-parameter "track and trigger" systems. Resuscitation. 2008;79:11-21. [CrossRef] [PubMed]
35. Gao H, McDonnell A, Harrison DA, Moore T, et al. Systemic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med. 2007;33:667-79. [CrossRef] [PubMed]

Cite as: Raschke RA, Khurana H, Owen-Reece H, Groves RH Jr, Curry SC, Martin M, Stoffer B. Clinical performance of an interactive clinical decision support system for assessment of plasma lactate in hospitalized patients with organ dysfunction. Southwest J Pulm Crit Care. 2017;14:241-52. doi: https://doi.org/10.13175/swjpcc058-17 PDF 

Design of an Electronic Medical Record (EMR)-Based Clinical Decision Support System to Alert Clinicians to the Onset of Severe Sepsis

Stephanie Fountain, MD

James Perry III, MD

Brenda Stoffer

Robert Raschke, MD

Banner University Medical Center Phoenix

Phoenix, AZ, USA

Abstract

Background: The aim of our study was to design an electronic medical record ­based alert system to detect the onset of severe sepsis with sensitivity and positive predictive value (PPV) above 50%.

Methods: The PPV for each of seven potential criteria for suspected infection (white blood cell count (WBCC) >12 or <4 x 10⁹ /L, immature granulocyte count >0.1 K/uL or immature granulocyte % >1%, temperature >38 C. or <36 C. or the initiation of broad­spectrum antibiotics) was determined by chart review of 160 consecutive patients who had evidence of organ system failure (as defined by standard criteria)plus at least one of the proposed criteria. Then, using only criteria with calculated PPV >50%, the charts of sixty consecutive patients who met CMS criteria for severe sepsis were reviewed to calculate the sensitivity of organ dysfunction plus any one of the suspected infection criteria.

Results: Four proposed criteria for suspected infection had PPV >50%: WBCC >12 x 10 9 /L (69%; 95%CI:53­84%), Temperature >38C. (84%; 95%CI:68­100%), Temperature <36C. (57% 95%CI:36­78%), and initiation of antibiotics (70% 95%CI:56­84%). These four criteria were present in 53/60 of the patients with severe sepsis by CMS criteria, yielding a sensitivity of 88.3% (95%CI: 80.2­96.4%). Alert criteria were satisfied before the onset of severe sepsis in 25/53 cases, and within 90 minutes afterwards in 28/53 cases.

Conclusions: Our criteria for suspected infection plus organ dysfunction yields reasonable sensitivity and PPV for the detection of severe sepsis in real­time.

Editor's Note: For accompanying editorial click here

Introduction

The American College of Chest Physicians and the Society for Critical Care Medicine define sepsis as a systemic inflammatory syndrome in response to infection and defined sepsis as “severe” when associated with acute organ dysfunction (1,2). The incidence of severe sepsis varies depending on the method of data abstraction from 300 to >1,000 per 100,000 person-years with an in-hospital mortality of 14.7% to 29.9% (3). Severe sepsis was estimated to cost U.S. healthcare system more than $24 billion in 2007 (4). The incidence and mortality of severe sepsis is expected to continue to rise (3-6).

Early recognition of severe sepsis and rapid implementation of standardized treatment bundles is associated with improved patient outcomes (7-12), but compliance rates with standardized time-sensitive treatment bundles for severe sepsis are generally in the 30% range (13). One reason may be that clinicians do not always recognize the onset of severe sepsis and therefore don’t have the opportunity to initiate all the elements required for bundle compliance in time. Therefore, a system that could alert providers to the onset of severe sepsis could help them achieve bundle compliance.

Clinical Decision Support Systems (CDSSs) use innovative software incorporated into electronic medical records (EMRs) to augment the awareness and expert knowledge of clinicians by providing pertinent and timely information at the point of care. CDSSs are adept at performing surveillance of electronic data to identify patients with vital signs and laboratory findings consistent with clinical deterioration. Several researchers have previously attempted to identify patients with severe sepsis in real-time with EMR-based CDSSs, but these systems suffered poor positive predictive value (PPV) and uncertain sensitivity (14,15). The PPV of a CDSS surveillance alert is important because it is inversely related to the proportion of false alerts. False alerts lead to clinician alert fatigue and subsequent disregard of alert recommendations (16,17).  High sensitivity is another important operating characteristic, but sensitivity is typically only achievable at the cost of reducing PPV. 

The goal of this pilot study was to develop criteria that could be used in a CDSS to identify patients at the onset of severe sepsis in real-time in order to alert clinicians. We chose to operationalize severe sepsis as organ system dysfunction due to infection, without requiring systemic inflammatory response syndrome, since a recent study that showed that the requirement of SIRS in the definition of severe sepsis excludes 1-in-8 patients suffering organ system dysfunction due to infection (18).  Organ dysfunction already has a standard definition based on laboratory results and vital signs (2) that are  discrete and easily extracted from the EMR by CDSS logic, but suspected infection does not.   Thus, a specific aim of this study is to determine optimal EMR-based criteria to define suspected infection in relation to the diagnosis of severe sepsis.  Our hypothesis was that we could identify a set of criteria for suspected infection which would have acceptable sensitivity and PPV for severe sepsis when combined with standard organ system dysfunction criteria.      

Methods

We chose seven potential criteria to identify suspected infection: the presence of a white blood cell count (WBCC) >12 x 10⁹/L or <4 x 10⁹/L, immature granulocyte count >0.1 K/uL or immature granulocyte % >1%, temperature >38 C. or <36 C. or the initiation of broad-spectrum antibiotics (piperacillin/tazobactam, third or fourth-generation cephalosporin, aminoglycoside, carbapenem, or vancomycin).  Organ system dysfunction was identified in the EMR as previously described and delineated in table 1.  

Table 1. Suspected infection and organ dysfunction criteria.

Our study occurred in two phases.  In the first, we tested individual criteria related to suspected infection in order to determine which had PPV >50% and were therefore incorporated into the second phase of the study.  In the second phase, we combined those accepted criteria for suspected infection with organ system dysfunction criteria and calculated the sensitivity for the diagnosis of severe sepsis as defined by Centers for Medicare and Medicaid (CMS).

Phase 1.  We used Cerner Discern^® to access clinical data in our Cerner Millennium^® EMR (Cerner Corporation, North Kansas City MO, USA) in order to identify a retrospective cohort of 160  Banner Health inpatients who satisfied any one of the seven potential suspected infection criteria plus one organ system dysfunction criteria (Table 1) within an eight-hour window.

The cohort consisted of four groups of forty patients each based on the type of suspected infection criteria present: abnormal WBCC, abnormal temperature, elevated immature granulocytes and initiation of antibiotics.  Patients were also selected so that half met criteria in the emergency department and half on the hospital wards.  Patient selection was otherwise consecutive. Chart reviews were performed by physician research staff to determine whether each patient was suffering the onset of severe sepsis at the time suspected infection and organ dysfunction criteria were satisfied. Such patients were considered to be true positive for the purposes of calculating PPVs. We decided a-priori that individual criteria that did not achieve at least 50% PPV would not be used in our final list of accepted criteria for suspected infection to be used in phase 2 of our study. We also compared PPV for each criteria between emergency department patients and inpatients.

Phase 2. The charts of sixty consecutive patients who met CMS criteria for severe sepsis in Banner Health were reviewed to calculate sensitivity of the combination of any one of the suspected infection criteria accepted in phase 1, plus one organ system dysfunction criteria occurring together within a six-hour window. The gold standard for the diagnosis of severe sepsis, and the time of onset of severe sepsis, were determined using CMS criteria by trained Banner Health data extraction staff for the primary purpose of regulatory reporting to CMS. The chart of each patient identified with severe sepsis by CMS methodology was reviewed to determine how many exhibited criteria for suspected infection and organ system dysfunction within 8 hours before, or 90 minutes after the onset of severe sepsis determined by CMS methodology.  [The rationale for this time window was that a hypothetical alert triggered by these criteria would only be valuable if it identified patients before, or shortly after the onset of severe sepsis].  We considered these to be true positive for the purposes of calculating sensitivity.

Results

Phase 1: PPVs with 95% confidence intervals for each of the potential criteria for suspected infection are listed in Table 2 below.

Table 2. PPV and 95% CI for individual suspected infection criteria (when found in temporal association with organ system dysfunction) for the clinical diagnosis of severe sepsis.

Only WBCC had a significantly different PPV when used in the emergency department vs the inpatient wards:  84% vs 50% (p=0.03).   

Immature granulocytes and WBCC <4 x 10⁹/L had PPV <50% and could be excluded from the set of accepted criteria with no loss of sensitivity. The set of accepted criteria include: WBCC >12 x 10⁹/L. temperature >38 or <36 and initiation of antibiotics. Finding any one of these accepted criteria in association with organ system dysfunction yielded a PPV of 70% (95%CI: 61-78%) for the diagnosis of severe sepsis.  

In 35/115 cases in which patients with one of these accepted criteria for suspected infection were not suffering an infection (false positive) the actual diagnoses included: cardiovascular diseases (s/p coronary artery bypass, myocardial infarction, cardiogenic shock), post-operative state, endocrinological disorders (hypothyroidism, diabetic ketoacidosis, adrenal failure), central nervous system pathology (intracranial hemorrhage, subarachnoid hemorrhage, seizure), obstetrical complications (placenta previa, spontaneous hemorrhage), and gastrointestinal hemorrhage.

Conclusions

Our data suggests that the best criteria set for suspected infection are likely to be: WBCC >12 x 10⁹/L, temperature >38 or <36 C. or initiation of broad spectrum antibiotics. The PPV of this set of criteria is likely to be >60%.  Leukopenia, and elevated immature granulocyte counts each had poor PPV and their exclusion would not significantly diminish the sensitivity of the set of criteria. 

Compared to other alert systems, this logic is novel for its abandonment of the use of SIRS criteria and the inclusion of antibiotic initiation. It could be argued that initiation of antibiotics should not be used to identify suspected infection because the clinician starting antibiotics is obviously already aware of infection.  However, unpublished analysis of 323 Banner health patients who qualified for severe sepsis by CMS criteria showed that 76% of those who failed bundle compliance received appropriate and timely antibiotics, but failed other important aspects of care, such as getting blood cultures before starting antibiotics and assessing lactate concentration. This suggests that a severe sepsis alert, triggering when a clinician enters an order for antibiotics could potentially assist the clinician in ordering other bundle elements. Exclusion of antibiotic initiation from our accepted criteria would have reduced the sensitivity of our alert logic to 75%.

The operating characteristics of our CDSS compares favorably to four previously published severe sepsis surveillance CDSSs which utilized SIRS criteria (see table 3 below). 

Table 3. Operating characteristics of CDSSs designed to provide surveillance for severe sepsis.

One of the strengths of this alert logic is that is it widely generalizable. It only includes data that is collected on most, if not all, hospitalized patients. It does not require additional tests or measurements that may limit its utility to a smaller patient population. It does not require physicians or ancillary staff to perform additional tasks or deviate from their standard workflow. Another strength of this logic is that it was created within the software program Cerner Discern^® in our Cerner Millennium^® EMR, one of the most widely used EMRs across the country. This would potentially allow seamless integration into any hospital system using this software, improving patient care and fulfilling “meaningful use” mandate of the Affordable Care Act.  However, our study is only a small pilot study. These results will need further validation using a larger data set. Further studies are needed to show whether a CDSS using these criteria can improve clinical outcomes of patients with severe sepsis.

Refenences

1. Geroulanos S, Douka ET. Historical perspective of the word "sepsis". Intensive Care Med. 2006 Dec;32(12):2077. [CrossRef] [PubMed]
2. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, Schein RM, Sibbald WJ. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992 Jun;101(6):1644-55. [CrossRef] [PubMed]
3. Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med. 2013 May;41(5):1167-74. [CrossRef] [PubMed]
4. Lagu T, Rothberg MB, Shieh MS, Pekow PS, Steingrub JS, Lindenauer PK. Hospitalizations, costs, and outcomes of severe sepsis in the United States 2003 to 2007. Crit Care Med. 2012 Mar;40(3):754-61. [CrossRef] [PubMed]
5. Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med. 2007 May;35(5):1244-50. [Pubmed]
6. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001 Jul;29(7):1303-10. [CrossRef] [PubMed]
7. Miller RR 3rd, Dong L, Nelson NC, Brown SM, Kuttler KG, Probst DR, Allen TL, Clemmer TP; Intermountain Healthcare Intensive Medicine Clinical Program. Multicenter implementation of a severe sepsis and septic shock treatment bundle. Am J Respir Crit Care Med. 2013 Jul 1;188(1):77-82. [CrossRef] [PubMed]
8. Levy MM, Rhodes A, Phillips GS, Townsend SR, Schorr CA, Beale R, Osborn T, Lemeshow S, Chiche JD, Artigas A, Dellinger RP. Surviving Sepsis Campaign: association between performance metrics and outcomes in a 7.5-year study. Intensive Care Med. 2014 Nov;40(11):1623-33. [CrossRef] [PubMed]
9. Levy MM, Dellinger RP, Townsend SR, Linde-Zwirble WT, Marshall JC, Bion J, Schorr C, Artigas A, Ramsay G, Beale R, Parker MM, Gerlach H, Reinhart K, Silva E, Harvey M, Regan S, Angus DC. The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med. 2010 Feb;36(2):222-31. [CrossRef] [PubMed]
10. Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 2013 Feb;39(2):165-228. [CrossRef] [PubMed]
11. Levy MM, Pronovost PJ, Dellinger RP, Townsend S, Resar RK, Clemmer TP, Ramsay G. Sepsis change bundles: converting guidelines into meaningful change in behavior and clinical outcome. Crit Care Med. 2004 Nov;32(11 Suppl):S595-7. [CrossRef] [PubMed]
12. ProCESS Investigators, Yealy DM, Kellum JA, Huang DT, Barnato AE, Weissfeld LA, Pike F, Terndrup T, Wang HE, Hou PC, LoVecchio F, Filbin MR, Shapiro NI, Angus DC. A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014 May 1;370(18):1683-93. [CrossRef] [PubMed]
13. Gao F, Melody T, Daniels DF, Giles S, Fox S. The impact of compliance with 6-hour and 24-hour sepsis bundles on hospital mortality in patients with severe sepsis: a prospective observational study. Crit Care. 2005;9(6):R764-70. [CrossRef] [PubMed]
14. Tafelski S, Nachtigall I, Deja M, Tamarkin A, Trefzer T, Halle E, Wernecke KD, Spies C. Computer-assisted decision support for changing practice in severe sepsis and septic shock. J Int Med Res. 2010 Sep-Oct;38(5):1605-16. [CrossRef] [PubMed]
15. Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, French B, Fuchs BD. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med. 2015 Jan;10(1):26-31. [CrossRef] [PubMed]
16. Guidi JL, Clark K, Upton MT, et al. Clinician Perception of the Effectiveness of an Automated Early Warning and Response System for Sepsis in an Academic Medical Center. Ann Am Thorac Soc. 2015 Oct;12(10):1514-9. [CrossRef] [PubMed]
17. Shapiro NI, Howell MD, Talmor D, et al. Implementation and outcomes of the Multiple Urgent Sepsis Therapies (MUST) protocol. Crit Care Med. 2006 Apr;34(4):1025-32. [CrossRef] [PubMed]
18. Kaukonen KM, Bailey M, Pilcher D, Cooper DJ, Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med. 2015 Apr 23;372(17):1629-38. [CrossRef] [PubMed]
19. Alsolamy S, Al Salamah M, Al Thagafi M, et al. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak. 2014 Dec 5;14:105. [CrossRef] [PubMed]
20. Amland RC, Lyons JJ, Greene TL, Haley JM. A two-stage clinical decision support system for early recognition and stratification of patients with sepsis: an observational cohort study. JRSM Open. 2015 Oct 8;6(10):2054270415609004. [CrossRef] [PubMed]
21. Semler MW, Weavind L, Hooper MH, et al. An Electronic Tool for the Evaluation and Treatment of Sepsis in the ICU: A Randomized Controlled Trial. Crit Care Med. 2015 Aug;43(8):1595-602. [CrossRef] [PubMed]
22. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011 May;57(5):500-4. [CrossRef] [PubMed]

Cite as: Fountain S, Perry J III, Stoffer B, Raschke R. Design of an electronic medical record (EMR)-based clinical decision support system to alert clinicians to the onset of severe sepsis. Southwest J Pulm Crit Care. 2016 Apr;12(4):153-60. doi: http://dx.doi.org/10.13175/swjpcc021-16 PDF