Full Text View
Comparative effectiveness research in critically ill patients: risks associated with mischaracterising usual care
Willard N Applefeld*, Jeffrey Wang*, Harvey G Klein, Robert L Danner, Peter Q Eichacker, Charles Natanson, (*equal first authors)
Crit Care Resusc 2020; 22 (2): 110-118
- Willard N Applefeld* 1
- Jeffrey Wang* 1
- Harvey G Klein 2
- Robert L Danner 1
- Peter Q Eichacker 1
- Charles Natanson 1
- (*equal first authors) 3
Comparative effectiveness research can help guide the use of common, routine medical practices. However, to be safe and informative, such trials must include at least one treatment arm that accurately portrays current practices. While comparative effectiveness research is widely perceived as safe and to involve no or only minimal risks, these assumptions may not hold true if unrecognised deviations from usual care exist in one or more study arms. For critically ill subjects in particular, such practice deviations may increase the risk of death or injury and undermine safety monitoring. Furthermore, unrecognised unusual care seems likely to corrupt informed consent documents, with underappreciated risks shrouded under the reassuring "comparative effectiveness" research label. At present, oversight measures are inadequate to ensure that research subjects enrolled in comparative effectiveness trials are actually receiving usual and not unusual care. Oversight by governmental and non-governmental entities with appropriate expertise, empowered to ensure that current clinical practice has been properly represented, could help prevent occurrences in clinical trials of unusual care masquerading as usual care.
- Macklin R, Natanson C. Misrepresenting “usual care” in research: an ethical and scientific error. Am J Bioeth 2020; 20: 31-9.
- SUPPORT Study Group of the Eunice Kennedy Shriver NICHD Neonatal Research Network; Carlo WA, Finer NN, Walsh MC, et al. Target ranges of oxygen saturation in extremely preterm infants. N Engl J Med 2010; 362: 1959-69.
- NICHD Neonatal Research Network. The Surfactant Positive Airway Pressure Pulse Oximetry Trial in Extremely Low Birth Weight Infants: the SUPPORT trial; 2005. https://www.citizen.org/wp-content/uploads/SUPPPORT-Protocol-Revised_NIH-Posted.pdf (viewed Feb 2020).
- Lockwood C, Riley L, Blackmon L, Lemons JA; editors. Guidelines for perinatal care, 6th ed. American Academy of Pediatrics, American College of Obstetrics and Gynecologists, 2007
- Tin W, Milligan DW, Pennefather P, Hey E. Pulse oximetry, severe retinopathy, and outcome at one year in babies of less than 28 weeks gestation. Arch Dis Child Fetal Neonatal Ed 2001; 84: F106-10
- Cortés-Puch I, Wesley RA, Carome MA, et al. Usual care and informed consent in clinical trials of oxygen management in extremely premature infants. PLoS One 2016; 11: e0155005
- Hagadon JI, Furey AM, Nghiem TH, et al. Achieved versus intended pulse oximeter saturation in infants less than 28 weeks’ gestation: the AVIOx study. Pediatrics 2006; 118: 1574-82.
- US Department of Health and Human Services. 45 CFR 46.111 Criteria for IRB approval of research. https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html#46.111 (viewed Feb 2020).
- Schmidt B, Whyte RK, Asztalos EV, et al. Effects of targeting higher vs lower arterial oxygen saturations on death or disability in extremely preterm infants: a randomized clinical trial. JAMA 2013; 309: 2111-20
- BOOST II United Kingdom Collaborative Group; BOOST II Australia Collaborative Group; BOOST II New Zealand Collaborative Group; Stenson BJ, Tarnow-Mordi WO, Darlow BA, et al. Oxygen saturation and outcomes in preterm infants. New Engl J Med 2013; 368: 2094-104.
- Slutsky AS. Mechanical ventilation: American College of Chest Physicians’ Consensus Conference. Chest 1993; 104: 1833-59.
- Acute Respiratory Distress Syndrome Network; Brower RG, Matthay MA, Morris A, et al. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. N Engl J Med 2000; 342: 1301-08.
- Carome MA. Human research subject protections under Multiple Project Assurance (MPA) M-1106 [letter from the Office for Human Research Protections]. Rockville, MD: OHRP, 2002. http://wayback.archive-it.org/4657/20150826185738/http://www.hhs.gov/ohrp/detrm_letrs/YR02/feb02a.pdf (viewed Feb 2020).
- Carome MA. Human research subject protections under Multiple Project Assurance (MPA) M-1025 [letter from the Office for Human Research Protections]. Rockville, MD: OHRP, 2002. http://wayback.archive-it.org/4657/20150826184950/http://www.hhs.gov/ohrp/detrm_letrs/YR02/jan02n.pdf (viewed Feb 2020).
- Carome MA. Human research subject protections under Multiple Project Assurance (MPA) M-1183 [letter from the Office for Human Research Protections]. Rockville, MD: OHRP, 2002. http://wayback.archive-it.org/4657/20150826185205/http://www.hhs.gov/ohrp/detrm_letrs/YR02/feb02g.pdf (viewed Feb 2020).
- Eichacker PQ, Gerstenberger EP, Banks SM, et al. A meta-analysis of acute lung injury and acute respiratory distress syndrome trials testing low tidal volumes. Am J Respir Crit Care Med 2002; 166: 1510-14.
- Deans KJ, Minneci PC, Suffredini AF, et al. Randomization in clinical trials of titrated therapies: unintended consequences of using fixed treatment protocols. Crit Care Med 2007; 35: 1509-16.
- Deans KJ, Minneci PC, Banks SM, et al. Tidal volumes in acute respiratory distress syndrome — one size does not fit all. Crit Care Med 2006; 34: 264-7.
- Deans KJ, Minneci PC, Cui X. Mechanical ventilation in ARDS: one size does not fit all. Crit Care Med 2005; 33: 1141-3.
- Carmichael LC, Dorinsky PM, Higgins SB, et al. Diagnosis and therapy of acute respiratory distress syndrome in adults: an international survey. J Crit Care 1996; 11: 9-18.
- Hébert PC, Wells G, Blajchman MA, et al. A multicenter, randomized, controlled clinical trial of transfusion requirements in critical care. N Engl J Med 1999; 340: 409-17.
- Hébert PC, Wells G, Martin C, et al. A Canadian survey of transfusion practices in critically ill patients. Crit Care Med 1998; 16: 482-7.
- Consensus conference. Perioperative red blood cell transfusion. JAMA 1988; 260: 2700-03.
- Deans KJ, Minneci PC, Klein HG, Natanson C. The relevance of practice misalignments to trials in transfusion medicine. Vox Sang 2010; 99: 16-23
- Cortés-Puch I, Wiley BM, Sun J, et al. Risks of restrictive red blood cell transfusion. Strategies in patients with cardiovascular disease (CVD): a meta-analysis. Transfus Med 2018; 28: 335-45.
- Jaswal DS, Leung JM, Sun J, et al. Tidal volume and plateau pressure use for acute lung injury from 2000 to present: a systematic review. Crit Care Med 2014; 42: 2278-89.
- Writing group for the PReVENT Investigators; Simonis FD, Serpa Neto A, Binnekade JM, et al. Effect of low vs intermediate tidal volume strategy on ventilator-free days in intensive care unit patients without ARDS: a randomized clinical trial. JAMA 2018; 320: 1872-80.
- Patient-Centered Outcomes Research Institute. About our research. Washington, DC: PCORI, 2019. https://www.pcori.org/research-results/about-our-research (viewed Feb 2020).
The three case studies presented here challenge the widespread belief that randomised trials of putative usual care, particularly those enrolling critically ill subjects, are invariably safe. Properly designed head-to-head comparisons of contemporary care can improve clinical decision making by better quantifying relative risks and benefits. However, for such research to be informative, at least one arm must be truly representative of current medical practice. Some trials purporting to compare usual care practices may not accurately reflect those practices and instead provide unusual care, either to all enrolled subjects or to important subgroups within a cohort. If subjects are critically ill with a high baseline risk of death, this unusual care may have grave consequences that include but are not limited to the following:
- Inadequate informed consent: for research to be ethical and consent to be informed, subjects must understand how their care will change and the potential risks of study enrolment.
- Suboptimal care: unusual therapies that are inferior to usual care can put trial subjects at increased risk, particularly when administered to critically ill subjects.
- Inability to monitor safety: when a trial compares two unusual therapies, the benchmark for safety monitoring is lost. Without a usual care comparator, harm from either or both unusual care arms may go unrecognised.
- Inaccurate and harmful conclusions: without a usual care comparator arm, if both therapies studied are inferior to usual care, there is a risk that the least harmful intervention will displace usual care in practice guidelines, putting future patients at risk.
Three case studies
Misapplication of a guideline as representative of usual care: the Surfactant, Positive Pressure, and Pulse Oximetry Randomized Trial (SUPPORT)Premature, high risk infants have underdeveloped lungs and often require supplemental oxygen therapy at the time of delivery, and sometimes for weeks to months afterwards. However, both the upper and lower ends of the oxygen concentration range used in these infants can have different but potentially serious adverse effects. With too much oxygen, these infants can develop blindness or retinopathy of prematurity; with too little oxygen, they can have neurological injury or death. SUPPORT randomised 1316 premature, low birth weight, high risk infants to a high (91–95%) or low (85–89%) target range of oxygen saturation measured by pulse oximetry (Spo2) in an attempt to find an Spo2 target range that minimises these competing risks. 2
The methodology employed for the design of study groups in SUPPORT failed for two main reasons. First, the two Spo2 target ranges at either end of the AAP guideline systematically underrepresented the middle of the recommended range, which better reflected most, if not all, of usual care at the time. Second, the Tin and colleagues’ study 5
To determine usual care at the time of SUPPORT, we performed a comprehensive review of all contemporaneous, medical literature that was available before the trial began describing Spo2 targets. 6
Notably, an observational study of usual care could have been performed before or during the design of SUPPORT or the same information could have been obtained by conducting a thorough literature search to ascertain usual care practices. Furthermore, a survey of practices at the more than 21 NICUs enrolling subjects in SUPPORT would have most likely determined that the low Spo2 range to be studied was unusual care. Instead, expert opinion, a very general guideline recommendation, and a single, decade-old study seem to have been the major sources of information for the design of the SUPPORT trial. Because of this, the experimental nature of the low Spo2 range arm and the potential risks it posed to enrolled subjects were not recognised at the time of the trial. The informed consent documents from the 21 enrolling centres described both Spo2 ranges targeted in SUPPORT as “standard of care”, “standard care”, “normal”, “routinely used”, “currently used”, “desired approach”, “best approach” or “already being used”, without any discussion of the risks of using an overall lower target Spo2 range than currently used in US NICUs. 6
Unfortunately, two subsequent trials, started after SUPPORT, began enrolling subjects using the same methodology and examining similar extreme ranges of oxygen treatment in high risk, premature infants: one predominantly in Canada (Canadian Oxygen Trial [COT]), 9
Employing outdated rather than current practices as usual care: Acute Respiratory Distress Syndrome Network Lower Tidal Volume (ARMA) trialThe early stages of acute respiratory distress syndrome (ARDS) are characterised by diffuse alveolar damage for which patients frequently require mechanical ventilatory support. Starting in the late 1980s, concerns grew that excessive tidal volumes and airway pressures associated with mechanical ventilation would aggravate alveolar injury with ARDS. 11
“One [purpose of the study] is to compare two ways of inflating your lungs while on the machine. Doctors currently use both methods to breathe for patients, but it is not known if one method is better [than the other].” 13
“Presently doctors use different size breaths of oxygen-rich air to inflate the patient’s lungs. It is unknown whether it is better to use large (12 mL/kg) or small (6 mL/kg) [tidal volumes]. Both ways of inflating the lungs are acceptable methods that are commonly used to treat patients with [acute lung injury] and [ARDS].” 14
“Presently doctors use varying volumes of oxygen-enriched air to inflate the lungs. It is unknown whether it is better to use a large or small volume of oxygen-enriched air to inflate [the] lungs of patients with lung injury. Both ways of inflating a patient’s lungs are considered acceptable methods and are commonly used [in] medical practice.” 15
An analysis of tidal volumes administered to patients before their enrolment and randomisation in the ARMA trial showed that usual care was not commonly or predominantly 6 mL/kg PBW or 12 mL/kg PBW but actually varied over a wide, normally distributed range related to the level of lung injury and underlying lung compliance (Figure 2 and Figure 3). 16, 17, 18, 19
For about 80% of ARMA subjects assigned to the “traditional” 12 mL/kg PBW arm, ventilator tidal volumes had to be increased after randomisation to reach this level. Accordingly, airway pressures also increased in a substantial number of these participants beyond levels believed safe. 16
Ignoring usual care, resulting in harm to subgroups receiving care opposite to usual care treatment: Transfusion Requirements in Critical Care (TRICC) trialThe TRICC trial 21
Randomising subjects to two fixed haemoglobin thresholds meant that transfusion strategies were no longer based on patient individual needs, comorbidities or underlying physiology. Instead, patients were randomly assigned to transfusion regimens chosen primarily to conserve blood supplies. Before beginning the trial, investigators surveyed 193 physicians to ascertain existing practices 22
The TRICC trial did not include a usual care group with transfusion triggers based on an individualised assessment of patient needs. The design of the trial heeded neither the results of the investigator survey nor the existing consensus guidelines on transfusion practices, instead making both arms unusual care for important patient subgroups by setting arbitrarily fixed triggers. 22, 23
We performed an analysis using data from the original TRICC trial publication combined with subsequently published trial data and demonstrated that subjects with pre-existing severe cardiovascular disease had a significantly different response to the two fixed transfusion thresholds compared with patients without that condition. 17, 24, 25
Randomisation to high and low haemoglobin thresholds in the TRICC trial ensured that one subgroup (younger, stable patients) in the high trigger arm would receive transfusions when not necessarily clinically indicated. 17, 21, 24