CICM Online CCR Journal logo CICM logo

Full Text View

Original Article

ISARIC-4C Mortality Score overestimates risk of death due to COVID-19 in Australian ICU patients: a validation cohort study

Matthew L Durie, Ary Serpa Neto, Aidan JC Burrell, D Jamie Cooper, Andrew A Udy, for the SPRINT-SARI Australia Investigators

Crit Care Resusc 2021; 23 (4): 403-413

Correspondence:andrew.udy@monash.edu

https://doi.org/10.51893/2021.4.OA5

  • Author Details
  • Competing Interests
    All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf. Ary Serpa Neto reports personal fees from Drager, outside the submitted work. Matthew Durie, Aidan JC Burrell, D Jamie Cooper and Andrew A Udy declare no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years. All authors declare that they have no relationships or activities that could appear to have influenced the submitted work.
  • Abstract
    OBJECTIVE: To assess the performance of the UK International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC) Coronavirus Clinical Characterisation Consortium (4C) Mortality Score for predicting mortality in Australian patients with coronavirus disease 2019 (COVID-19) requiring intensive care unit (ICU) admission.
    DESIGN: Multicentre, prospective, observational cohort study.
    SETTING: 78 Australian ICUs participating in the SPRINT-SARI (Short Period Incidence Study of Severe Acute Respiratory Infection) Australia study of COVID-19.
    PARTICIPANTS: Patients aged 16 years or older admitted to participating Australian ICUs with polymerase chain reaction (PCR)-confirmed COVID-19 between 27 February and 10 October 2020.
    MAIN OUTCOME MEASURES: ISARIC-4C Mortality Score, calculated at the time of ICU admission. The primary outcome was observed versus predicted in-hospital mortality (by 4C Mortality and APACHE II).
    RESULTS: 461 patients admitted to a participating ICU were included. 149 (32%) had complete data to calculate a 4C Mortality Score without imputation. Overall, 61/461 patients (13.2%) died, 16.9% lower than the comparable ISARIC-4C cohort in the United Kingdom. In patients with complete data, the median (interquartile range [IQR]) 4C Mortality Score was 10.0 (IQR, 8.0–13.0) and the observed mortality was 16.1% (24/149) versus 22.9% median predicted risk of death. The 4C Mortality Score discriminatory performance measured by the area under the receiver operating characteristic curve (AUROC) was 0.79 (95% CI, 0.68–0.90), similar to its performance in the original ISARIC-4C UK cohort (0.77) and not superior to APACHE II (AUROC, 0.81; 95% CI, 0.75–0.87).
    CONCLUSIONS: When calculated at the time of ICU admission, the 4C Mortality Score consistently overestimated the risk of death for Australian ICU patients with COVID-19. The 4C Mortality Score may need to be individually recalibrated for use outside the UK and in different hospital settings.
  • References
    1. Armstrong R, Kane A, Cook T. Outcomes from intensive care in patients with COVID-19: a systematic review and meta-analysis of observational studies. Anaesthesia 2020; 75: 1340-9.
    2. Armstrong RA, Kane AD, Cook TM. Decreasing mortality rates in ICU during the COVID-19 pandemic. Anaesthesia 2021; 76 (Suppl): 10.
    3. Tan E, Song J, Deane AM, Plummer MP. Global impact of coronavirus disease 2019 infection requiring admission to the ICU: a systematic review and meta-analysis. Chest 2021; 159: 524-36.
    4. Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020; 369: m1328.
    5. Gupta R, Marks M, Samuels THA, et al. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study. Eur Respir J 2020; 56: 2003498.
    6. Levy T, Richardson S, Coppa K, et al. Development and validation of a survival calculator for hospitalized patients with COVID-19 [preprint]. medRxiv 20075416; 2 June 2020. doi: 10.1101/2020.04.22.20075416 (viewed Oct 2021).
    7. Dong YM, Sun J, Li YX, et al. Development and validation of a nomogram for assessing survival in patients with COVID-19 pneumonia. Clin Infect Dis 2021; 72: 652-60.
    8. Liang W, Liang H, Ou L, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med 2020; 180: 1081-9.
    9. Knight SR, Ho A, Pius R, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ 2020; 370: m3339.
    10. Wellbelove Z, Walsh C, Perinpanathan T, et al. Comparing the 4C mortality score for COVID-19 to established scores (CURB65, CRB65, qSOFA, NEWS) for respiratory infection patients. J Infect 2021; 82: 414-51.
    11. van Dam P, Zelis N, van Kuijk SMJ, et al. Performance of prediction models for short-term outcome in COVID-19 patients in the emergency department: a retrospective study. Ann Med 2021; 53: 402-9.
    12. Yildiz H, Castanares-Zapatero D, Hannesse C, et al. Prospective validation and comparison of COVID-GRAM, NEWS2, 4C mortality score, CURB-65 for the prediction of critical illness in COVID-19 patients. Infect Dis (Lond) 2021; 53: 640-2.
    13. Covino M, De Matteis G, Burzo ML, et al. Predicting in-hospital mortality in COVID-19 older patients with specifically developed scores. J Am Geriatr Soc 2021; 69: 37-43.
    14. Burrell AJ, Pellegrini B, Salimi F, et al. Outcomes for patients with COVID-19 admitted to Australian intensive care units during the first four months of the pandemic. Med J Aust 2021; 214: 23-30.
    15. Intensive Care National Audit and Research Centre. ICNARC report on COVID-19 in critical care: England, Wales and Northern Ireland — 25 September 2020. London: ICNARC, 2020. https://www.icnarc.org/DataServices/Attachments/Download/baa7de02-3f00-eb11-912b-00505601089b (viewed Oct 2020).
    16. Knaus WA, Wagner DP, Draper EA, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest 1991; 100: 1619-36.
    17. Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med 1996; 22: 707-10.
    18. Paul E, Bailey M, Pilcher D. Risk prediction of hospital mortality for adult patients admitted to Australian and New Zealand intensive care units: development and validation of the Australian and New Zealand Risk of Death model. J Crit Care 2013; 28: 935-41.
    19. ISARIC-4C Consortium. 4C Mortality Score 2020. https://isaric4c.net/risk/ (viewed Oct 2020).
    20. Grasselli G, Greco M, Zanella A, et al. Risk factors associated with mortality among patients with COVID-19 in intensive care units in Lombardy, Italy. JAMA Intern Med 2020; 180: 1345-55.
    21. Zangrillo A, Beretta L, Scandroglio AM, et al. Characteristics, treatment, outcomes and cause of death of invasively ventilated patients with COVID-19 ARDS in Milan, Italy. Crit Care Resusc 2020; 22: 200-11.
    22. Grasselli G, Zangrillo A, Zanella A, et al. Baseline characteristics and outcomes of 1591 patients infected with SARS-CoV-2 admitted to ICUs of the Lombardy Region, Italy. JAMA 2020; 323: 1574-81.
    23. Mårtensson J, Engerström L, Walther S, et al. COVID-19 critical illness in Sweden: characteristics and outcomes at a national population level. Crit Care Resusc 2020; 22: 312-20.
    24. Monti G, Cremona G, Zangrillo A, et al. Home ventilators for invasive ventilation of patients with COVID-19. Crit Care Resusc 2020; 22: 266-70.
    25. Ling L, So C, Shum HP, et al. Critically ill patients with COVID-19 in Hong Kong: a multicentre retrospective observational cohort study. Crit Care Resusc 2020; 22: 119-25.
    26. Docherty AB, Harrison EM, Green CA, et al. Features of 20 133 UK patients in hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: prospective observational cohort study. BMJ 2020; 369: m1985.
    27. Intensive Care National Audit and Research Centre. ICNARC report on COVID-19 in critical care: England, Wales and Northern Ireland — 3 June 2021. London: ICNARC, 2020. https://www.icnarc.org/DataServices/Attachments/Download/483d4e84-3fc5-eb11-9131-00505601089b (viewed Oct 2020).
    28. RECOVERY Collaborative Group; Horby P, Lim WS, Emberson JR, et al. Dexamethasone in hospitalized patients with COVID-19. N Engl J Med 2020; 384: 693-704.
    29. Horby P, Lim WS, Emberson J, et al. Effect of dexamethasone in hospitalized patients with COVID-19: preliminary report [preprint]. medRxiv 20137273; 22 June 2020. doi: 10.1101/2020.06.22.20137273.
    30. Burrell AJ, Serpa Neto A, Trapani T, et al. Rapid translation of COVID-19 preprint data into critical care practice. Am J Respir Crit Care Med 2021; 203: 368-71.
    31. Gupta RK, Harrison EM, Ho A, et al. Development and validation of the ISARIC 4C deterioration model for adults hospitalised with COVID-19: a prospective cohort study. Lancet Respir Med 2021; 9: 349-59.
    32. Paul E, Bailey M, Van Lint A, Pilcher V. Performance of APACHE III over time in Australia and New Zealand: a retrospective cohort study. Anaesth Intensive Care 2012; 40: 980-94.
    33. Coombes ID, Dooley MJ, Townsend S, et al. Physician drug prescribing preferences and availability for ventilation of patients with COVID-19. Crit Care Resusc 2020; 22: 271-4.
    34. COVID-19 National Incident Room Surveillance Team. COVID-19 Australia: epidemiology report 28: fortnightly reporting period ending 25 October 2020. Commun Dis Intell (2018) 2020; 44. doi: 10.33321/cdi.2020.44.84.
    35. Millar JE, Busse R, Fraser JF, et al. Apples and oranges: international comparisons of COVID-19 observational studies in ICUs. Lancet Respir Med 2020; 8: 952-3.
    36. Nguyen T, Gupta S, Raman J, et al. Geolocated Twitter-based population mobility in Victoria, Australia, during the staged COVID-19 restrictions. Crit Care Resusc 2020; 22: 293-4.
    37. Landoni G, Moro M, Belletti A, et al. Recent exposure to smoking and COVID-19. Crit Care Resusc 2020; 22: 253-6.
    38. Plummer MP, Pellegrini B, Burrell AJ, et al. Smoking in critically ill patients with COVID-19: the Australian experience. Crit Care Resusc 2020; 22: 281-3.
    39. Ma X, Vervoort D. Critical care capacity during the COVID-19 pandemic: global availability of intensive care beds. J Crit Care 2020; 58: 96-7.
Mortality after admission to the intensive care unit (ICU) with coronavirus disease 2019 (COVID-19) varies widely, from initial reports in high prevalence areas of up to 90% to more recent international estimates between 30% and 50%. 1, 2, 3 Many prognostic tools have been proposed for use in patients with COVID-19. 4 These may help guide clinical decision making, improve the allocation of scarce resources and allow comparisons between different cohorts. In a meta-analysis during the early phase of the COVID-19 pandemic, the performance of these was only moderate (area under the receiver operating characteristic curve [AUROC], < 0.6–0.8), 5  although subsequent models have claimed performance above an AUROC of 0.8. 6, 7, 8 One of the largest to be validated is the International Severe Acute Respiratory and Emerging Infection Consortium (ISARIC) Coronavirus Clinical Characterisation Consortium (4C) Mortality Score, which used three demographic, three clinical, and two laboratory parameters to calculate a score between 0 and 21 points at the time of hospital admission among patients with COVID-19 in the United Kingdom. 9  This score demonstrated reasonable performance within a UK context in predicting mortality among a validation cohort of 22 361 patients (AUROC, 0.77). In subsequent independent UK validation cohorts, it performed favourably compared with existing in-hospital mortality prediction scores (eg, CURB-65 [confusion, uraemia, respiratory rate, blood pressure, age ≥ 65 years], qSOFA [Quick Sequential Organ Failure Assessment], and NEWS [National Early Warning Score]) 10 as well as in cohorts from the Netherlands 11  and Belgium 12 and in older Italian patients. 13 However, the performance of the ISARIC-4C Mortality Score in other regions and at the time of ICU admission is unknown.
 
Mortality from COVID-19 in Australia has been lower than that reported elsewhere. To the end of June 2020, Australian COVID-19 ICU mortality was around 15% overall, and 22% among invasively ventilated patients. 14 This was less than half of the ICU mortality for COVID-19 patients admitted to UK ICUs before 31 August 2020 (39%). 15 The reasons for this difference between Australian and UK outcomes remain unclear. Existing severity of illness and risk prediction scores (eg, Acute Physiology and Chronic Health Evaluation [APACHE], 16 SOFA 17 or, locally, the Australian and New Zealand Risk of Death [ANZROD]) 18 are not specific to COVID-19. We hypothesised that the 4C Mortality Score, while developed for use at the time of hospital admission, may offer a way of estimating prognosis at the time of ICU admission for patients with COVID-19. We also hypothesised that applying the UK validated 4C Mortality Score to an Australian context might assist in understanding the relatively low COVID-19 mortality rate in Australia compared with the UK. This study, therefore, sought to externally validate the performance of the 4C Mortality Score among patients with severe COVID-19 who required admission to Australian ICUs.

Methods

Study design, participants, and data collection

SPRINT-SARI (Short Period Incidence Study of Severe Acute Respiratory Infection) Australia is a multicentre, prospective, observational study of patients with COVID-19 admitted to participating ICUs in Australia. Details concerning the study design, partners, and data collection have been published elsewhere. 14  In brief, patients aged 16 years or over with a confirmed positive polymerase chain reaction (PCR) test for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) who had an index COVID-19-related admission to a participating ICU were included. Biological samples for PCR testing could be taken from the nasopharynx, trachea or lower airways via bronchoscopy. Patients without a positive PCR test were excluded. Research staff at each ICU were responsible for screening all admissions for patients with COVID-19. De-identified data were submitted by participating sites using a standardised case report form via REDCap (Vanderbilt University). The study was coordinated by the Australian and New Zealand Intensive Care Research Centre (ANZIC-RC), Monash University.
 
Data collected included baseline demographic and clinical characteristics before and during ICU admission. Of the acute components of the 4C Mortality Score, oxygen saturation and respiratory rate were collected at hospital admission, and oxygen saturation was re-measured on the first day of ICU admission. The case report form did not include respiratory rate at the time of ICU admission, so the value recorded at the time of hospital admission was used. Urea, C-reactive protein (CRP) and Glasgow Coma Scale data began on the first day of ICU admission. Comorbidities were recorded based on a modified Charlson Comorbidity Index. The APACHE II score for the first 24 hours of ICU admission was calculated. Data were recorded until death, hospital discharge or truncated at time of data extraction. In the event of an interhospital transfer, data from the index hospital or ICU admission were used to calculate the 4C Mortality Score and data from the receiving institution were used to determine outcomes.
 

Outcomes

Observed and predicted hospital mortality were compared using the 4C Mortality Score calculated at the time of ICU admission. Second, predictive performance was compared with an existing ICU scoring system (APACHE II). A sensitivity analysis was also performed comparing patients admitted to ICUs on or before 30 June 2020 with those admitted after this date.
 

Statistical analysis

The 4C Mortality Score and the predicted mortality were calculated according to original published formulae using all available data at the time of ICU admission. 9, 19 Patients with data missing for one or more of the eight 4C Mortality Score parameters were managed with the following analyses: excluded from the analysis (complete case analysis), best value imputation (best case scenario), worst value imputation (worst case scenario), median value imputation, and multiple imputation. Additional information is provided in the Online Appendix.
 
Data are reported as number (percentage) or median (interquartile range [IQR]). The total number of patients contributing data is provided for all analyses. Discriminatory performance of 4C Mortality Score was assessed by AUROC of the absolute score against observed mortality. Calibration was assessed through calibration plots and belts, and through the Brier score. Performance was further assessed comparing observed and predicted mortality by risk band as previously defined. 9  ICU and hospital length of stay were truncated at the date of dataset extraction (22 October 2020) for patients still admitted to the hospital. All analyses were performed using R v.4.0.2 (R Core Team, 2019).
 

Ethics approval

SPRINT-SARI Australia received Human Research Ethics Committee (HREC) and governance approval for data collection, with a waiver of patient informed consent through the Alfred Hospital (HREC/16/Alfred/59), or by separate applications to individual sites. 14

Results

Between 27 February and 10 October 2020, a total of 461 patients were admitted to 53 Australian ICUs with PCR-confirmed COVID-19 (Table 1). Due to variable disease prevalence, 25/78 (32%) participating ICUs did not receive any COVID-19 admissions, while 59% of patients were admitted to ICUs in the state of Victoria. The median age was 61.0 years (IQR, 51.0–70.0 years) and about two-thirds were male (64%). Patients were admitted to the ICU a median of 0.4 days (IQR, 0.1–2.0 days) after hospitalisation. Mechanical ventilation was required in 35.3% of patients during the first day of ICU admission and in 55.0% of patients at any time in the ICU. On the first day of ICU admission, 125/441 (28.3%) patients received neither advanced respiratory nor inotropic support. Additional intervention and complication data are included in Table 1 and in the Online Appendix.
 
Complete data for 4C Mortality Score calculation were present for 149/461 (32%) patients at ICU admission (Table 1). The most common missing values within the first 24 hours of ICU admission were CRP (present in 178/461 patients, 38.6%) and urea (present in 218/461 patients, 47.3%). For the 149 patients with complete data available, the median 4C Mortality Score at ICU admission was 10.0 (IQR, 8.0–13.0) (Table 2 and Figure 1), corresponding to a median predicted mortality of 22.9% (IQR, 14.4–40.1%). Imputing for missing values, the median 4C Mortality Score (risk of death percentage) for the total cohort (= 461) ranged from 8.0 (14.4%) with best case scenario to 13.0 (40.1%) with worst case scenario imputation (Table 2). At the time of data extraction, 446/461 (96.7%) patients had completed their hospital admission. The observed mortality was lower than the median predicted mortality across all risk categories (Table 3 and Figure 2), with 101/149 (67.7%) patients with complete data available falling in the high or very high risk bands. In total, 61/461 (13.2%) patients died in hospital: 57 in the ICU and a further four in hospital wards following ICU discharge (Table 4). A further five patients (1.1%) remained in the ICU and ten patients (2.2%) in hospital. Death occurred in 24/149 (16.1%) patients, for whom data for all 4C Mortality parameters were available.
 
The performance of the 4C Mortality Score on the first day of ICU admission in predicting hospital mortality by AUROC was 0.79 (95% CI, 0.68–0.90) for cases with complete data (= 149), and 0.75 (95% CI, 0.68–0.82) using multiple imputation for the whole cohort (= 461) (Table 2 and Figure 3). The 4C Mortality Score had similar performance to APACHE II in predicting hospital mortality (AUROC, 0.81; 95% CI, 0.75–0.87). Score performance across the range of mortality estimates is shown in calibration belts in Figure 3. Performance using other imputation methods is included in the Online Appendix.
 
A sensitivity analysis was performed comparing patients with complete data admitted on or before (= 71) versus after 30 June (= 78). The median 4C Mortality Score and predicted mortality did not change (median, 10.0 [IQR, 8–12] v 10.0 [IQR, 7–13] respectively; and predicted risk of death, 22.9% in both time periods) and observed mortality was comparable (11/71 [15.5%] v 13/78 [16.7%] patients). The discriminatory performance of the 4C Mortality Score was similar across both time periods (AUROC, 0.79 [95% CI, 0.63–0.96] v 0.78 [95% CI, 0.63–0.94]) (Online Appendix).

Discussion

Key findings

In an Australian context, the ISARIC-4C Mortality Score, calculated at the time of ICU admission, overestimated mortality by 6.8% in patients with complete data, a relative excess of 42%. Despite overestimating mortality, there was no evidence that the 4C Mortality Score was inferior in its predictive performance in this cohort compared with the original ISARIC-4C cohort (AUROC, 0.79 v0.77 respectively), 9  suggesting that recalibration of the risk estimate for Australia may be possible. Overall mortality (13.2%) was 16.9% lower than in the original UK validation cohort of patients admitted to hospital (30.1%). This is despite comparable proportions of patients in the high and very high risk 4C Mortality Score categories (68 v 71%), suggesting that the risk of death for a given severity of COVID-19 may have been lower in Australia than in the UK during this time period.
 

Relationship to previous work

Mortality following admission to the ICU with COVID-19 varies between regions. 1, 2, 3, 20, 21 High prevalence regions reported that a lack of ICU resources in early 2020 was associated with a rapid expansion of ward-based non-invasive ventilation 21  and delays between hospital presentation and ICU admission for patients with COVID-19. 20 A high proportion of patients in these regions required invasive mechanical ventilation following ICU admission (88% and 100% in two cohorts from Lombardy and Milan, Italy, 21, 22 and 85% in a cohort from Sweden 23 ), and there were concerns about exhausting available ICU ventilators. 24 Conversely, at the same time, an early case series from non-resource-limited settings (eg, Hong Kong) with much better outcomes suggested system strain may have been a factor. 25
 
Data from ISARIC and the Intensive Care National Audit and Research Centre (ICNARC) suggest that the mortality of patients admitted to the ICU with COVID-19 was higher in the UK, where the 4C Mortality Score was derived and validated, than in Australia. 14, 15, 26 The predictive performance of the 4C Mortality Score in our cohort was similar to other 4C Mortality Score validation studies, with AUROC values between 0.77 and 0.84. 9, 10, 11, 12 However, these studies did not report a bias between observed and predicted mortality as we have observed. The median age of patients in our cohort was younger than in the ISARIC-4C cohort (61 v73 years), but this factor is included in the 4C Mortality Score, and the median age of their separately reported ICU cohort appears close to that of our cohort of ICU patients. 26  The proportion of patients invasively ventilated within the first 24 hours of ICU admission in Australia (35.3%) is likely lower than that in the ISARIC-4C validation cohort, given that ICNARC data including the same UK population (to 31 August 2020) reported that 54.1% of patients were invasively ventilated on day 1 of ICU admission. 27  The ISARIC-4C cohort only included patients to the end of June 2020; however, we found no difference in performance or bias in our cohort before or after 30 June despite an increase in steroid use associated with the publication of the RECOVERY trial preprint. 28, 29, 30 Of note, the rate of invasive ventilation for patients admitted to ICUs in the UK has fallen markedly since 31 August 2020, to around 30% on day 1 or 55% at any time in ICU (similar to our cohort), while ICU mortality has remained unchanged at around 38%. 27
 

Implications

The ISARIC authors describe the 4C Mortality Score as a “rule in test” to identify those at risk of developing severe disease at the time of hospital admission. 9  Subsequently, a 4C Deterioration Score has been proposed specifically to identify patients at risk of deterioration. 31  Given our cohort included patients already admitted to the ICU, the 4C Mortality Score was thought to be more informative, particularly as a potential tool to identify patients at increased risk of dying and to inform clinical decision making. However, our findings imply that, as currently calibrated, the 4C Mortality Score cannot be used to accurately predict mortality among patients with COVID-19 in Australian hospitals who are admitted to the ICU. While the validity of the 4C Mortality Score for patients admitted to Australian hospitals but not the ICU remains unknown, most patients in our cohort were admitted to the ICU within 24 hours of hospital presentation. In addition, in Australia, the 4C Mortality Score did not perform any better than APACHE II, the latter being known to overestimate mortality in ICU patients locally. 18, 32 These findings also suggest that compared with the UK, the lower rate of mortality among Australian ICU patients with COVID-19 is not simply due to less severe illness. Indeed, it is unclear whether this difference is due to host or disease factors, or systematic factors such as comparative resource limitation in the UK as, by comparison, Australia has had much fewer COVID-19 admissions. Exploratory studies in the Australian context suggest resource limitation could have been quickly encountered with a relatively small increase in patients with COVID-19 in the ICU. 33
 

Strengths and weaknesses

SPRINT-SARI Australia is the largest source of COVID-19 ICU admission and outcome data nationally, and our cohort is estimated to have included more than 99% of ICU admissions due to PCR-confirmed COVID-19 in Australia during this time period. 34  The study design, case report form, and protocol were developed with ISARIC allowing direct comparison of our data with the UK ISARIC-4C data. While the ISARIC-4C article truncated outcomes at 28 days, 9  we report up-to-date outcomes for all patients including those 15 (3.3%) still in hospital at the date of data extraction. Incomplete outcome data may bias mortality estimates, 35  but it is unlikely we have under-reported mortality compared with the ISARIC-4C UK cohort.
 
Our relatively small number of patients compared with the UK ISARIC-4C cohort reflects the low prevalence of COVID-19 within Australia during the study period. Furthermore, over half of all patients in this study were admitted from one state, Victoria, due to a higher burden of disease in that region. This region was also subject to prolonged activity restrictions during the study period, 36  and other ICU activity was not captured in our data. The small cohort size limits the precision of the estimates and, therefore, we have reported mortality rates in risk bands. The retrospective nature of the data means there was a high proportion of patients with at least one missing 4C Mortality Score parameter (particularly CRP). We have corrected for missing data using different imputation methods, and demonstrated that, even using a best case scenario, the score did not underestimate mortality. As the case report form collected most data at the time of ICU admission, we did not have adequate data to calculate a score at hospital admission as originally described. It is possible that using data from a later time point has resulted in overestimation of predicted mortality; however, the median time to ICU admission was short, less than 24 hours in most cases.
 

Future study

The role of mortality prediction scores for COVID-19 remains uncertain. The 4C Mortality Score, at least in a UK context, appears to perform adequately when measured at the time of hospital admission. However, validation in other settings and with a larger group of patients will be required before it can be more widely recommended. Revalidation of the 4C Mortality Score at the time of ICU admission using the original ISARIC-4C data would help to better understand the differences observed between our cohort and the original article, as would validation in other regions that experienced a high burden of COVID-19 ICU admissions similar to the UK. It remains unclear whether the differences we have observed between our cohort and the UK are due to unmeasured patient-level factors that influence severity, as data on comorbidities or other respiratory risk factors such as smoking 37, 38 are not captured by the 4C Mortality Score. Likewise, practices around patient selection into ICU or strain on each health care system at the time may have contributed to the observed differences. 39  Critically, these populations may offer an opportunity to further explore these questions.
 

Conclusion

When calculated at the time of ICU admission, the 4C Mortality Score consistently overestimated the risk of death for patients with COVID-19 admitted to Australian ICUs. This was despite adequate discrimination, suggesting the score may need to be individually recalibrated for use outside the UK and at the time of ICU rather than hospital admission.

Acknowledgements: SPRINT-SARI Australia is funded by the Department of Health, Australian Government; Standing Deed SON60002733. D Jamie Cooper reports grant funding (Practitioner Fellowship) from the Australian National Health and Medical Research Council (Grant number: GNT1142215).
 
The SPRINT-SARI Australia participating sites: Albury Wodonga Health, Alice Springs Hospital, Angliss Hospital, Austin Hospital, Ballarat Base Hospital, Bankstown-Lidcombe Hospital, Barwon Health, Bendigo Hospital, Box Hill Hospital, Bunbury Hospital, Bundaberg Hospital, Caboolture Hospital, Cabrini Hospital Malvern, Cairns Hospital, Calvary Mater Newcastle, Campbelltown Hospital, Canberra Hospital, Casey Hospital, Concord Hospital, Dandenong Hospital, Epworth Richmond, Fiona Stanley Hospital, Flinders Medical Centre, Frankston Hospital, Gold Coast University Hospital, Hervey Bay Hospital, Ipswich Hospital, John Hunter Hospital, Joondalup Health Campus, Launceston General Hospital, Lismore Base Hospital, Liverpool Hospital, Logan Hospital, Lyell McEwan Hospital, Maroondah Hospital, Mater Hospital Brisbane, Mildura Base Hospital, Monash Children’s Hospital, Monash Medical Centre, Nepean Hospital, Northeast Health Wangaratta, Northern Hospital, Perth Children’s Hospital, Port Macquarie Base Hospital, Prince of Wales Hospital, Princess Alexandra Hospital, Queensland Children’s Hospital, Redcliffe Hospital, Rockingham Hospital, Royal Adelaide Hospital, Royal Brisbane and Women’s Hospital, Royal Children’s Hospital, Royal Darwin Hospital, Royal Hobart Hospital, Royal Melbourne Hospital, Royal North Shore Hospital, Royal Perth Hospital, Royal Prince Alfred Hospital, Sir Charles Gairdner Hospital, St George Hospital, St John of God Hospital Midland, St John of God Hospital Murdoch, St Vincent’s Hospital Melbourne, St. Vincent’s Hospital Sydney, Sunshine Coast University Hospital, Sydney Children’s Hospital Randwick, The Alfred Hospital, The Children’s Hospital at Westmead, The Prince Charles Hospital, The Queen Elizabeth Hospital, Toowoomba Hospital, Warrnambool Base Hospital, Werribee Mercy Hospital, Western Health (Footscray), Western Health (Sunshine), Westmead Hospital, Wollongong Hospital, and Women’s and Children’s Hospital Adelaide.

TOP