Design of an Electronic Medical Record (EMR)-Based Clinical Decision Support System to Alert Clinicians to the Onset of Severe Sepsis

Stephanie Fountain, MD

James Perry III, MD

Brenda Stoffer

Robert Raschke, MD

Banner University Medical Center Phoenix

Phoenix, AZ, USA

Abstract

Background: The aim of our study was to design an electronic medical record ­based alert system to detect the onset of severe sepsis with sensitivity and positive predictive value (PPV) above 50%.

Methods: The PPV for each of seven potential criteria for suspected infection (white blood cell count (WBCC) >12 or <4 x 10⁹ /L, immature granulocyte count >0.1 K/uL or immature granulocyte % >1%, temperature >38 C. or <36 C. or the initiation of broad­spectrum antibiotics) was determined by chart review of 160 consecutive patients who had evidence of organ system failure (as defined by standard criteria)plus at least one of the proposed criteria. Then, using only criteria with calculated PPV >50%, the charts of sixty consecutive patients who met CMS criteria for severe sepsis were reviewed to calculate the sensitivity of organ dysfunction plus any one of the suspected infection criteria.

Results: Four proposed criteria for suspected infection had PPV >50%: WBCC >12 x 10 9 /L (69%; 95%CI:53­84%), Temperature >38C. (84%; 95%CI:68­100%), Temperature <36C. (57% 95%CI:36­78%), and initiation of antibiotics (70% 95%CI:56­84%). These four criteria were present in 53/60 of the patients with severe sepsis by CMS criteria, yielding a sensitivity of 88.3% (95%CI: 80.2­96.4%). Alert criteria were satisfied before the onset of severe sepsis in 25/53 cases, and within 90 minutes afterwards in 28/53 cases.

Conclusions: Our criteria for suspected infection plus organ dysfunction yields reasonable sensitivity and PPV for the detection of severe sepsis in real­time.

Editor's Note: For accompanying editorial click here

Introduction

The American College of Chest Physicians and the Society for Critical Care Medicine define sepsis as a systemic inflammatory syndrome in response to infection and defined sepsis as “severe” when associated with acute organ dysfunction (1,2). The incidence of severe sepsis varies depending on the method of data abstraction from 300 to >1,000 per 100,000 person-years with an in-hospital mortality of 14.7% to 29.9% (3). Severe sepsis was estimated to cost U.S. healthcare system more than $24 billion in 2007 (4). The incidence and mortality of severe sepsis is expected to continue to rise (3-6).

Early recognition of severe sepsis and rapid implementation of standardized treatment bundles is associated with improved patient outcomes (7-12), but compliance rates with standardized time-sensitive treatment bundles for severe sepsis are generally in the 30% range (13). One reason may be that clinicians do not always recognize the onset of severe sepsis and therefore don’t have the opportunity to initiate all the elements required for bundle compliance in time. Therefore, a system that could alert providers to the onset of severe sepsis could help them achieve bundle compliance.

Clinical Decision Support Systems (CDSSs) use innovative software incorporated into electronic medical records (EMRs) to augment the awareness and expert knowledge of clinicians by providing pertinent and timely information at the point of care. CDSSs are adept at performing surveillance of electronic data to identify patients with vital signs and laboratory findings consistent with clinical deterioration. Several researchers have previously attempted to identify patients with severe sepsis in real-time with EMR-based CDSSs, but these systems suffered poor positive predictive value (PPV) and uncertain sensitivity (14,15). The PPV of a CDSS surveillance alert is important because it is inversely related to the proportion of false alerts. False alerts lead to clinician alert fatigue and subsequent disregard of alert recommendations (16,17).  High sensitivity is another important operating characteristic, but sensitivity is typically only achievable at the cost of reducing PPV. 

The goal of this pilot study was to develop criteria that could be used in a CDSS to identify patients at the onset of severe sepsis in real-time in order to alert clinicians. We chose to operationalize severe sepsis as organ system dysfunction due to infection, without requiring systemic inflammatory response syndrome, since a recent study that showed that the requirement of SIRS in the definition of severe sepsis excludes 1-in-8 patients suffering organ system dysfunction due to infection (18).  Organ dysfunction already has a standard definition based on laboratory results and vital signs (2) that are  discrete and easily extracted from the EMR by CDSS logic, but suspected infection does not.   Thus, a specific aim of this study is to determine optimal EMR-based criteria to define suspected infection in relation to the diagnosis of severe sepsis.  Our hypothesis was that we could identify a set of criteria for suspected infection which would have acceptable sensitivity and PPV for severe sepsis when combined with standard organ system dysfunction criteria.      

Methods

We chose seven potential criteria to identify suspected infection: the presence of a white blood cell count (WBCC) >12 x 10⁹/L or <4 x 10⁹/L, immature granulocyte count >0.1 K/uL or immature granulocyte % >1%, temperature >38 C. or <36 C. or the initiation of broad-spectrum antibiotics (piperacillin/tazobactam, third or fourth-generation cephalosporin, aminoglycoside, carbapenem, or vancomycin).  Organ system dysfunction was identified in the EMR as previously described and delineated in table 1.  

Table 1. Suspected infection and organ dysfunction criteria.

Our study occurred in two phases.  In the first, we tested individual criteria related to suspected infection in order to determine which had PPV >50% and were therefore incorporated into the second phase of the study.  In the second phase, we combined those accepted criteria for suspected infection with organ system dysfunction criteria and calculated the sensitivity for the diagnosis of severe sepsis as defined by Centers for Medicare and Medicaid (CMS).

Phase 1.  We used Cerner Discern^® to access clinical data in our Cerner Millennium^® EMR (Cerner Corporation, North Kansas City MO, USA) in order to identify a retrospective cohort of 160  Banner Health inpatients who satisfied any one of the seven potential suspected infection criteria plus one organ system dysfunction criteria (Table 1) within an eight-hour window.

The cohort consisted of four groups of forty patients each based on the type of suspected infection criteria present: abnormal WBCC, abnormal temperature, elevated immature granulocytes and initiation of antibiotics.  Patients were also selected so that half met criteria in the emergency department and half on the hospital wards.  Patient selection was otherwise consecutive. Chart reviews were performed by physician research staff to determine whether each patient was suffering the onset of severe sepsis at the time suspected infection and organ dysfunction criteria were satisfied. Such patients were considered to be true positive for the purposes of calculating PPVs. We decided a-priori that individual criteria that did not achieve at least 50% PPV would not be used in our final list of accepted criteria for suspected infection to be used in phase 2 of our study. We also compared PPV for each criteria between emergency department patients and inpatients.

Phase 2. The charts of sixty consecutive patients who met CMS criteria for severe sepsis in Banner Health were reviewed to calculate sensitivity of the combination of any one of the suspected infection criteria accepted in phase 1, plus one organ system dysfunction criteria occurring together within a six-hour window. The gold standard for the diagnosis of severe sepsis, and the time of onset of severe sepsis, were determined using CMS criteria by trained Banner Health data extraction staff for the primary purpose of regulatory reporting to CMS. The chart of each patient identified with severe sepsis by CMS methodology was reviewed to determine how many exhibited criteria for suspected infection and organ system dysfunction within 8 hours before, or 90 minutes after the onset of severe sepsis determined by CMS methodology.  [The rationale for this time window was that a hypothetical alert triggered by these criteria would only be valuable if it identified patients before, or shortly after the onset of severe sepsis].  We considered these to be true positive for the purposes of calculating sensitivity.

Results

Phase 1: PPVs with 95% confidence intervals for each of the potential criteria for suspected infection are listed in Table 2 below.

Table 2. PPV and 95% CI for individual suspected infection criteria (when found in temporal association with organ system dysfunction) for the clinical diagnosis of severe sepsis.

Only WBCC had a significantly different PPV when used in the emergency department vs the inpatient wards:  84% vs 50% (p=0.03).   

Immature granulocytes and WBCC <4 x 10⁹/L had PPV <50% and could be excluded from the set of accepted criteria with no loss of sensitivity. The set of accepted criteria include: WBCC >12 x 10⁹/L. temperature >38 or <36 and initiation of antibiotics. Finding any one of these accepted criteria in association with organ system dysfunction yielded a PPV of 70% (95%CI: 61-78%) for the diagnosis of severe sepsis.  

In 35/115 cases in which patients with one of these accepted criteria for suspected infection were not suffering an infection (false positive) the actual diagnoses included: cardiovascular diseases (s/p coronary artery bypass, myocardial infarction, cardiogenic shock), post-operative state, endocrinological disorders (hypothyroidism, diabetic ketoacidosis, adrenal failure), central nervous system pathology (intracranial hemorrhage, subarachnoid hemorrhage, seizure), obstetrical complications (placenta previa, spontaneous hemorrhage), and gastrointestinal hemorrhage.

Conclusions

Our data suggests that the best criteria set for suspected infection are likely to be: WBCC >12 x 10⁹/L, temperature >38 or <36 C. or initiation of broad spectrum antibiotics. The PPV of this set of criteria is likely to be >60%.  Leukopenia, and elevated immature granulocyte counts each had poor PPV and their exclusion would not significantly diminish the sensitivity of the set of criteria. 

Compared to other alert systems, this logic is novel for its abandonment of the use of SIRS criteria and the inclusion of antibiotic initiation. It could be argued that initiation of antibiotics should not be used to identify suspected infection because the clinician starting antibiotics is obviously already aware of infection.  However, unpublished analysis of 323 Banner health patients who qualified for severe sepsis by CMS criteria showed that 76% of those who failed bundle compliance received appropriate and timely antibiotics, but failed other important aspects of care, such as getting blood cultures before starting antibiotics and assessing lactate concentration. This suggests that a severe sepsis alert, triggering when a clinician enters an order for antibiotics could potentially assist the clinician in ordering other bundle elements. Exclusion of antibiotic initiation from our accepted criteria would have reduced the sensitivity of our alert logic to 75%.

The operating characteristics of our CDSS compares favorably to four previously published severe sepsis surveillance CDSSs which utilized SIRS criteria (see table 3 below). 

Table 3. Operating characteristics of CDSSs designed to provide surveillance for severe sepsis.

One of the strengths of this alert logic is that is it widely generalizable. It only includes data that is collected on most, if not all, hospitalized patients. It does not require additional tests or measurements that may limit its utility to a smaller patient population. It does not require physicians or ancillary staff to perform additional tasks or deviate from their standard workflow. Another strength of this logic is that it was created within the software program Cerner Discern^® in our Cerner Millennium^® EMR, one of the most widely used EMRs across the country. This would potentially allow seamless integration into any hospital system using this software, improving patient care and fulfilling “meaningful use” mandate of the Affordable Care Act.  However, our study is only a small pilot study. These results will need further validation using a larger data set. Further studies are needed to show whether a CDSS using these criteria can improve clinical outcomes of patients with severe sepsis.

Refenences

1. Geroulanos S, Douka ET. Historical perspective of the word "sepsis". Intensive Care Med. 2006 Dec;32(12):2077. [CrossRef] [PubMed]
2. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, Schein RM, Sibbald WJ. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992 Jun;101(6):1644-55. [CrossRef] [PubMed]
3. Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med. 2013 May;41(5):1167-74. [CrossRef] [PubMed]
4. Lagu T, Rothberg MB, Shieh MS, Pekow PS, Steingrub JS, Lindenauer PK. Hospitalizations, costs, and outcomes of severe sepsis in the United States 2003 to 2007. Crit Care Med. 2012 Mar;40(3):754-61. [CrossRef] [PubMed]
5. Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med. 2007 May;35(5):1244-50. [Pubmed]
6. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001 Jul;29(7):1303-10. [CrossRef] [PubMed]
7. Miller RR 3rd, Dong L, Nelson NC, Brown SM, Kuttler KG, Probst DR, Allen TL, Clemmer TP; Intermountain Healthcare Intensive Medicine Clinical Program. Multicenter implementation of a severe sepsis and septic shock treatment bundle. Am J Respir Crit Care Med. 2013 Jul 1;188(1):77-82. [CrossRef] [PubMed]
8. Levy MM, Rhodes A, Phillips GS, Townsend SR, Schorr CA, Beale R, Osborn T, Lemeshow S, Chiche JD, Artigas A, Dellinger RP. Surviving Sepsis Campaign: association between performance metrics and outcomes in a 7.5-year study. Intensive Care Med. 2014 Nov;40(11):1623-33. [CrossRef] [PubMed]
9. Levy MM, Dellinger RP, Townsend SR, Linde-Zwirble WT, Marshall JC, Bion J, Schorr C, Artigas A, Ramsay G, Beale R, Parker MM, Gerlach H, Reinhart K, Silva E, Harvey M, Regan S, Angus DC. The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med. 2010 Feb;36(2):222-31. [CrossRef] [PubMed]
10. Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 2013 Feb;39(2):165-228. [CrossRef] [PubMed]
11. Levy MM, Pronovost PJ, Dellinger RP, Townsend S, Resar RK, Clemmer TP, Ramsay G. Sepsis change bundles: converting guidelines into meaningful change in behavior and clinical outcome. Crit Care Med. 2004 Nov;32(11 Suppl):S595-7. [CrossRef] [PubMed]
12. ProCESS Investigators, Yealy DM, Kellum JA, Huang DT, Barnato AE, Weissfeld LA, Pike F, Terndrup T, Wang HE, Hou PC, LoVecchio F, Filbin MR, Shapiro NI, Angus DC. A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014 May 1;370(18):1683-93. [CrossRef] [PubMed]
13. Gao F, Melody T, Daniels DF, Giles S, Fox S. The impact of compliance with 6-hour and 24-hour sepsis bundles on hospital mortality in patients with severe sepsis: a prospective observational study. Crit Care. 2005;9(6):R764-70. [CrossRef] [PubMed]
14. Tafelski S, Nachtigall I, Deja M, Tamarkin A, Trefzer T, Halle E, Wernecke KD, Spies C. Computer-assisted decision support for changing practice in severe sepsis and septic shock. J Int Med Res. 2010 Sep-Oct;38(5):1605-16. [CrossRef] [PubMed]
15. Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, French B, Fuchs BD. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med. 2015 Jan;10(1):26-31. [CrossRef] [PubMed]
16. Guidi JL, Clark K, Upton MT, et al. Clinician Perception of the Effectiveness of an Automated Early Warning and Response System for Sepsis in an Academic Medical Center. Ann Am Thorac Soc. 2015 Oct;12(10):1514-9. [CrossRef] [PubMed]
17. Shapiro NI, Howell MD, Talmor D, et al. Implementation and outcomes of the Multiple Urgent Sepsis Therapies (MUST) protocol. Crit Care Med. 2006 Apr;34(4):1025-32. [CrossRef] [PubMed]
18. Kaukonen KM, Bailey M, Pilcher D, Cooper DJ, Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med. 2015 Apr 23;372(17):1629-38. [CrossRef] [PubMed]
19. Alsolamy S, Al Salamah M, Al Thagafi M, et al. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak. 2014 Dec 5;14:105. [CrossRef] [PubMed]
20. Amland RC, Lyons JJ, Greene TL, Haley JM. A two-stage clinical decision support system for early recognition and stratification of patients with sepsis: an observational cohort study. JRSM Open. 2015 Oct 8;6(10):2054270415609004. [CrossRef] [PubMed]
21. Semler MW, Weavind L, Hooper MH, et al. An Electronic Tool for the Evaluation and Treatment of Sepsis in the ICU: A Randomized Controlled Trial. Crit Care Med. 2015 Aug;43(8):1595-602. [CrossRef] [PubMed]
22. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011 May;57(5):500-4. [CrossRef] [PubMed]

Cite as: Fountain S, Perry J III, Stoffer B, Raschke R. Design of an electronic medical record (EMR)-based clinical decision support system to alert clinicians to the onset of severe sepsis. Southwest J Pulm Crit Care. 2016 Apr;12(4):153-60. doi: http://dx.doi.org/10.13175/swjpcc021-16 PDF

Clinical Performance Of An Automated Systemic Inflammatory Response Syndrome (SIRS) / Organ Dysfunction Alert: A System-Based Patient Safety Project

Robert A Raschke, MD, MS

Huw Owen-Reece, MBBS

Hargobind Khurana, MD 

Robert H Groves Jr, MD

Steven C Curry, MD

Mary Martin, PharmD

Brenda Stoffer

Suresh Uppalapu, MD 

Heemesh Seth, DO

Nithya Menon, MD 

Banner Good Samaritan Medical Center, Phoenix Arizona

Abstract

Objective: We have employed our electronic medical record (EMR) in an effort to identify patients at the onset of severe sepsis through an automated analysis that identifies simultaneous occurrence of systemic inflammatory response syndrome (SIRS) and organ dysfunction. The purpose of this study was to determine the positive predictive value of this alert for severe sepsis and other important outcomes in hospitalized adults.

Design: Prospective cohort.

Setting: Banner Good Samaritan Medical Center, Phoenix AZ

Patients: Forty adult inpatients who triggered alert logic within our EMR indicating simultaneous occurrence of SIRS and organ dysfunction.

Interventions: Interview of bedside nurse and chart review within six hours of alert firing to determine the clinical event that triggered each alert.

Results: Eleven of 40 patients (28%) had a major clinical event (immediately life-threatening illness) associated with the alert firing. Severe sepsis or septic shock accounted for four of these – yielding a positive predictive value of 0.10 (95%CI: 0.04-0.23) of the alert for detection of severe sepsis. The positive predictive value of the alert for detection of major clinical events was 0.28 (95%CI: 0.16-0.43), and for detecting either a major or minor clinical event was 0.45 (95%CI: 0.31-0.60). Twenty-two of 40 patients (55%) experienced a false alert.

Conclusions: Our first-generation SIRS/organ dysfunction alert has a low positive predictive value for severe sepsis, and generates many false alerts, but shows promise for the detection of acute clinical events that require immediate attention. We are currently investigating refinements of our automated alert system which we believe have potential to enhance patient safety.

Introduction

Severe sepsis is defined as systemic inflammatory response syndrome (SIRS) of infectious etiology with secondary organ dysfunction. It is estimated that 750,000 patients suffer severe sepsis annually in the United States - 3 cases per 1000 population (1). Mortality has fallen over the past several decades, but ranges from 20-30% in recent studies (1,3). Results of recent treatment trials for severe sepsis are consistent with the hypothesis that early diagnosis and treatment are important (2,3), but reliable systems for early recognition of severe sepsis in hospitalized patients are not widely available.  

We have sought to improve patient safety at our institution by using our integrated electronic medical record (EMR) to identify patients at the onset of severe sepsis through a logic algorithm that analyzes vital signs and laboratory data. This logic function identifies patients with simultaneous systemic inflammatory response syndrome (SIRS) and organ dysfunction, but cannot distinguish whether an acute infection is the cause of these findings. The purpose of this study was to determine what clinical events – infectious or non-infectious - actually cause the vital sign and laboratory changes that trigger this alert, and what the positive predictive valve of the alert is for detecting the onset of severe sepsis in hospitalized adult patients.

Methods

This was a prospective cohort study carried out at Banner Good Samaritan Medical Center – a 700-bed University-affiliated teaching hospital in Phoenix AZ. It was part of an ongoing quality improvement project and was thereby exempted from IRB approval. The SIRS / organ dysfunction alert logic was developed at Banner Health using Cerner Discern Expert^®, Cerner Corporation, North Kansas City MO, USA. The logic function monitored the EMR for standard SIRS criteria and laboratory evidence of organ dysfunction with thresholds consistent with standard definition of severe sepsis (Table 1) (4,5).

Table 1. Specific criteria for the logic function of our SIRS/organ failure alert. 

When any single criterion for SIRS was met, the program searched the prior 6 hours for the most recent vital signs, and the prior 30 hours for the most recent white blood cell count. If a second SIRS criterion was met, the program identified the patient as exhibiting SIRS, but did not trigger an alert. When any single laboratory criterion for organ dysfunction was met (table 1), the program identified the patient as suffering organ dysfunction. If criteria for SIRS and organ dysfunction overlap in any 8 hour window, the alert fired, triggering a real-time notification in the Cerner Millenium^® EMR alerting clinicians to the possibility of severe sepsis or septic shock. The alert has been in clinical application since 2010. 

We sampled 40 non-consecutive inpatients in the first three months of 2014 by a nonrandom method blinded to the patient’s clinical condition. On days of data collection, all alerts that had fired within the prior 6 hours were reviewed, regardless of patient location or diagnosis. The patient bedside was visited by a physician researcher during the six-hour window after alert firing and the nurse interviewed in order to determine the circumstances that caused the alert to fire. The patient might be briefly examined if necessary to confirm the nursing impression. Chart review was also performed to assist in this determination. Demographics, admission diagnosis, vital signs and laboratory data that triggered the alert logic, and any treatment the associated clinical event required were also recorded. Chart review was repeated 48 hours later to review microbiological test results and physician progress notes that might shed further light on the clinical event that triggered the alert.

The “clinical event” associated with each alert was defined as the most likely acute explanation for the vital sign and laboratory abnormalities that triggered the alert. A clinical event might be an acute illness, such as pneumonia with septic shock, or a non-illness event, such as initiation of dialysis. Clinical events could include the illness that necessitated admission if the alert fired within 24 hours of admission, or secondary illnesses - for instance, a catheter-associated blood stream infection.

The severity of clinical events related to alert firings were classified into three tiers. 

1. Major clinical events were acute life-threatening illnesses that required emergent resuscitation with any one or more of the following: >1 L intravenous fluid resuscitation, vasopressor infusion, >2 units of packed red blood cell transfusion, endotracheal intubation, advanced cardiac life support, or emergent surgical intervention.
2. Minor clinical events were acute non-life-threatening illnesses that required urgent treatments not included in the definition of major clinical events above.
3. False alerts were said to have occurred when no acute illness was recognized in temporal relationship to the alert firing. 

The positive predictive value of the alert for detecting severe sepsis, major clinical events, and major or minor clinical events were calculated, with 95% confidence intervals.

Results

Nineteen women and 21 men, with ages ranging from 22 to 103 years were included. Twenty-two of forty (55%) were in the ICU at the time the alert fired, and 18 on the floors. Vital signs and laboratory values that triggered the alert logic are listed in Table 2. 

Table 2. SIRS / organ dysfunction alert trigger criteria in forty patients. 

Eleven of 40 patients (28%) had a major clinical event associated with the alert firing – two of these occurred outside the ICU. Severe sepsis or septic shock accounted for four of these major clinical events – yielding a positive predictive value of 0.10 (95%CI: 0.04-0.23) of the alert for detection of severe sepsis or septic shock. The seven remaining patients with major events suffered acute pulmonary edema, pulmonary embolism, ischemic bowel, pancreatitis, acute cardiogenic shock, acute right heart failure secondary to pulmonary hypertension, and an incarcerated enteric hernia. The positive predictive value of the alert for detection of major clinical events was 0.28 (95%CI: 0.16-0.43).

Major clinical events were clearly recognized before the alert fired in nine of 11 cases, as evidenced by the patient having been admitted or transferred to the intensive care unit specifically for the event of interest, and/or having received treatment such as intubation or initiation of intravenous vasopressors before the alert fired. In two cases, the alert fired at about the same time that treatment of the acute clinical event commenced, and it was unclear what role it played in clinical recognition of the event.

Seven of 40 patients (17%) had a minor clinical event associated with the alert firing. These included two patients with anemia, and one each with hypotension from an antihypertensive medication, dialysis disequilibrium, post-operative pain, dehydration, and paroxysmal atrial fibrillation. The positive predictive value of the alert for detecting either a major or minor clinical event was 0.45 (95%CI: 0.31-0.60).

Twenty-two of 40 patients (55%) were not experiencing any identifiable acute illness that could explain the alert firing - these were considered false alerts. Aberrant vital signs triggered false alerts during dialysis (2), turning or sitting-up post-operative patients (2), an endoscopy procedure, and a family argument. Other false alerts were attributable to the pharmacological effect of calcium channel blocker, oximeter malfunction, error in vital sign entry, and widely discrepant blood pressures between right and left arms. The remaining false alerts were triggered by slightly abnormal vital signs with no identifiable cause.

Four patients (10%) did not survive to discharge – two had major clinical events, one a minor clinical event and one a false alert – in the later two cases, the cause of death was unrelated to the clinical event that triggered the alert.  

We examined alert triggering criteria to better understand how the discriminant ability of the alert might be improved. We noted that 15 of 40 (37%) alerts triggered with respiratory rates of 21 or 22 bpm, however these included six alerts associated with major clinical events. Twelve of 40 (30%) alerts triggered with heart rates in 91-95 bpm range, including two alerts associated with major clinical events. Laboratory results contributed to 31 of 40 alert firings – but in 12 cases they were stable or improving at the time they triggered the alert. In no case was a stable or improving laboratory value associated with a major clinical event.

Discussion

It’s important to study the effects of any quality improvement project in order to determine whether it is having the desired results. Our small pilot study suggests that our first-generation SIRS/organ dysfunction alert has a low positive predictive value for severe sepsis, and generates many false alerts. This is partially a reflection of the low specificity of SIRS criteria for sepsis (6). The high number of false positive alerts has led to alert-fatigue among physicians and nurses providing bedside patient care – a phenomenon which is not unique to our institution (7).  

Our alert demonstrated greater potential utility to detect acute clinical deterioration than to detect sepsis. Buck and colleagues (7) used an EMR-based logic system to activate a sepsis alert similar to ours, and observed similar results in that only 17% of alert patients had a sepsis-related discharge diagnosis, but 40% had a major illness which required urgent intervention. We have used the results of our study to re-task future iterations of our alert to detect acute clinical deterioration rather than sepsis.

Other researchers provide guidance in this regard. Vital sign and laboratory result criteria similar to the ones used in our study have been previously shown to predict in-hospital cardiac arrest (8), predict 30-day mortality (9), generate early warning scores to detect acute clinical deterioration (9), and activate medical emergency teams (8,10). A recent large study by Churpek and colleagues (11) validated a risk stratification tool that utilized vital signs, laboratory findings and demographics to predict the combined outcome of cardiac arrest, ICU transfer or death on the wards. The model yielded notable discriminant accuracy with an area under the receiver operating curve (AUROC) of 0.77.

We are currently investigating revisions in our alert logic to improve detection of acute clinical deterioration. The alert logic now trends laboratory values associated with organ dysfunction. We are studying whether adding a reflex serum lactate to the automatic alert response might help identify patients who are acutely deteriorating (12).   

Our study has many apparent weaknesses, but it should be noted that it was carried out originally only to provide data to help guide local efforts to improve patient safety. In this regard, it succeeded in guiding our (and perhaps other’s) future efforts in what will more likely be a useful direction.

We failed to clearly determine what role our automated alert played in bedside decision-making. In most cases, clinicians were already evaluating or treating the clinical event that triggered the alert before the alert fired. However, we feel that a safety net is a wise precaution even in a high-reliability system. It should also be noted that our institution has medicine and surgery residency teaching programs, a critical care fellowship, 24/7 in-house intensivist coverage, and remote video ICU coverage. The benefit of EMR-based automated alerts is likely to be amplified in less well-staffed institutions. Refined versions of EMR-based automated alerts, such as the ones we are currently investigating, have potential to enhance patient safety. 

References

1. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303-10. [CrossRef] [PubMed]
2. Rivers E, Nguyen B, Havstad S, Ressler J, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345:1368-77. [CrossRef] [PubMed]
3. The ProCESS investigators. A randomized controlled trial of protocol-based care for early septic shock. N Engl J Med. 2014;370(18):1683-93. [CrossRef] [PubMed]
4. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, Cohen J, Opal SM, Vincent JL, Ramsay G. International Sepsis Definitions Conference. Crit Care Med. 2003;31(4):1250-6. [CrossRef] [PubMed]
5. Dellinger RP, Levy MM, Rhodes A, et al, Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580. [CrossRef] [PubMed]
6. Pittet D, Range-Frausto S, Tarara LN, et al. Systemic inflammatory response syndrome, sepsis, severe sepsis and septic shock: incidence, morbidities and outcomes in surgical ICU patients. Intensive Care Med. 1995;21:302-9. [CrossRef] [PubMed]
7. Buck KM. Developing an early sepsis alert program. J Nurs Care Qual. 2014;29(2):124-32. [CrossRef] [PubMed]
8. Hodgetts TJ, Kenward G, Ioannis G, et al. The identification of risk factors for cardiac arrest and formulation of activation criteria to alert a medical emergency team. Resuscitation. 2002;54:125-31. [CrossRef] [PubMed]
9. Goldhill DR, McNarry AF. Physiological abnormalities in early warning scores are related to mortality in adult inpatients. Br J Anaesth. 2004;92:882-4. [CrossRef] [PubMed]
10. Kenward G, Castle N, Hodgetts T, Shaikh L. Evaluation of a medical emergency team one year after implementation. Resuscitation. 2004;61:257-63. [CrossRef] [PubMed]
11. Churpek MM, Yuen TC, Winslow C, et al. Multicenter development of validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med. 2014;190:649-55. [CrossRef] [PubMed]
12. Bakker J, Nijsten MWN, Jansen TC. Clinical use of the lactate monitoring in critically-ill patients. Ann Intensive Care. 2013;3:12-20. [CrossRef] [PubMed] 

Reference as: Raschke RA, Owen-Reece H, Khurana H, Groves RH Jr, Curry SC, Martin M, Stoffer B, Uppalapu S, Seth H, Menon N. Clinical performance of an automated systemic inflammatory response syndrome (sirs) / organ dysfunction alert: a system-based patient safety project. Southwest J Pulm Crit Care. 2014;9(4):223-9. doi: http://dx.doi.org/10.13175/swjpcc121-14 PDF