Reversing Evidence Hierarchies for Risk Assessment in Medicine

Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? Barbara Osimani Catholic University of Milan Evidence and Causality in the Sciences Canterbury, 5-7 September 2012

Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? In their comparative analysis of RCTs and observational studies, Papanikolau et al. (2006) assert: it may be unfair to invoke bias and confounding to discredit observational studies as a source of evidence on harms (p. 640, my emphasis).

Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? Recent contributions by philosophers and health scientists have acknowledged the role of so called "lower level" evidence as a valid source of information contributory to assessing the risk profile of medications both on theoretical (Aronson and Hauben, 2006; Howick et al. 2009) and on empirical grounds (Benson and Hartz, 2000; Golder et al. 2011). Nevertheless current practices have difficulty in assigning a precise epistemic status to this kind of evidence and in amalgamating it with standard methods of hypothesis testing.

Topics Recent proposals to amend evidence hierarchies especially Vandenbroucke s suggestion to reverse hierarchies when addressing the issue of risk discovery and assessment; The requirement of total evidence as applied to this specific context: how it is and how it should be interpreted (vs. lexicographic rule implicit in ranking); I will illustrate some of these points by reference to the recent debate on the causal association between paracetamol and increased asthma prevalence/exhacerbation.

Hierarchy reversal for risk assessment Hierarchy of study designs for intended effects of therapy Hierarchy of study designs for discovery and explanation i. Randomised controlled trials i. Anecdotal: case report and series, findings in data, literature ii. Case-control studies ii. Prospective follow-up studies iii. Retrospective follow-up studies iii. Retrospective follow-up studies iv. Case-control studies iv. Prospective follow-up studies v. Anecdotal: case report and series v. Randomised controlled trials Vandenbroucke J.P. (2008) Observational Research, Randomised Trials, and Two Views of Medical Science. Plos Medicine, 5 (3): 339-43

Vandenbrouckes defence of hierarchy reversal (I) 1. Methodological point: Observational studies concerning adverse reactions will not suffer from confounding in the same way asobservational studies for intended effects do. selection bias is less likely to affect observational studies with respect to adverse reactions. This because unintended effects, qua unintended, are not known in advance, and thus also not known by the drug prescriber, who cannot take them into consideration and thus bias treatment allocation. Ignorance of possible effect = natural masking

Vandenbrouckes defence of hierarchy reversal (II) 2. Epistemological point: Context of discovery vs. context of evaluation: Discovery is focused on explanation and hypothesis generation; Evaluation instead on hypothesis testing/confirmation. And research methods differ in the opportunities they offer with respect to either of these goals.

Vandenbrouckes defence of hierarchy reversal (III) Vandenbroucke (2008) formalizes the contrast between the context of evaluation and the context of discovery in terms of different priors assigned to hypotheses of benefits and of adverse reactions. High priors for intended effects Low priors for unintended ones

Vandenbrouckes defence of hierarchy reversal (III) 1. It is the higher priors which make the results more robust, not the method (Vandenbroucke, 2008: 16-17). 2. The reason why we accept uncertain results for risks rather than for benefits is that evaluation and discovery studies are associated with different loss functions: 1. evaluation is related to the approval of health technologies and is required to assure stakeholders about their efficacy and safety, 2. whereas discovery is more related to the context of research for its own sake, which might explain why certain study designs are preferred to others in different circumstances.

Vandenbrouckes defence of hierarchy reversal (III) 1. Priors are quickly swamped by data 2. Stakes are not lower for detecting risks than for testing the drug benefit: adverse drug reactions might be so severe as to reverse the safety profile of the drug and determine its withdrawal.

Prior knowledge about drugs general capacity to produce unintended adverse reactions The acceptability of anecdotal evidence or of uncontrolled studies for assessing risk has to do with a high prior about the general capacity of the drug to bring about side-effects. Whereas there is total ignorance as to some specific side effects which might be possibly caused by the drug, still there is almost certainty about the fact that the drug will indeed cause side-effects beyond the ones already detected in the pre-marketing phase. This high prior derives from historical knowledge and past experience with pharmaceutical products and is also strongly reflected in the regulation which introduced the notion of development (or potential) risk , the pharmacosurveillance system, and the precautionary principle.

Vandenbrouckes defence of hierarchy reversal on abductive grounds For discoveries, the original case reports, lab observations, data analysis, or juxtaposition in literature may be so convincing that they stand by themselves, either because of the magnitude of the effect or because the new explanation suddenly and convincingly makes the new finding fall into place with previous unexplained data or previous ideas . (Vandenbroucke, 2008: 6).

Case Study: hypothesis of causal connection between paracetamol and asthma Asthma increase in the United States and in Western countries in the last 3 decades: up to a 75% increase among adults and to a 160% among children in the same period. (Burr et al., 1989; Eneli et al., 2005, Ninan and Russel, 1992; Mannino et al., 1998, 2002, Seaton et al. 1994).

Explanatory hypotheses for asthma epidemic 1) increased exposure to outdoor and indoor pollutants; 2) decreased exposure to bacteria and childhood illnesses during infancy (the hygiene hypothesis ); 3) increased obesity incidence and prevalence; 4) changes in diet and oxidant intake; 5) cytokine imbalance as a reaction to environmental allergens in early childhood leading to lifelong T-helper type 2 (allergic) dominance over T-helper type 1 (nonallergic) reactions, thus increasing the risk for atopic disease Eneli et al., 2005; Seaton et al. 1994, Shaheen et al. 2000.

How suspicion fell upon paracetamol Varner and colleagues (1998) detected a precise correspondence between increase of asthma incidence and increased paracetamol use as a substitute for aspirin (following the recognition of an association between aspirin and Reye s syndrome). The trend levelled off in the 1990s, i.e. at a time when paracetamol had already become one of the most widespread analgesics. Varner and colleagues tentative explanation was however that asthma increase was due to aspirin avoidance, for the reason that aspirin may protect from asthma through inhibition of prostaglandins. However, this hypothesis was soon discounted on grounds that, if this had been the case, then one should have observed a decrease of asthma incidence when aspirin was first introduced (Shaheen et al. 2000). Thus the suspicion finally fell upon paracetamol itself and subsequent investigations explicitly aimed to examine the hypothesis of causal connection between paracetamol and asthma.

Evidence for causal association between paracetamol and asthma Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative: 1) consistency of the association across geography, culture and age; 2) strength of the association (comparative studies); 3) the dose-response relationship between paracetamol exposure and asthma; 4) the coincidence of the timing of increasing asthma prevalence and increasing paracetamol use; 5) the relationship between per-capita sales of paracetamol and asthma morbidity across countries; 6) our inability to identify any other abrupt environmental change that could explain this increase in asthma morbidity; 7) plausible mechanism: glutathione depletion in airway mucosa caused by paracetamol . McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).

Consistency of the association across geography, culture and age (I) Source Year study 2008 of Study objective Population Results Beasley et al. Cross-cultural study Examine the risk of asthma rhynoconjunctivitis and eczema in children using paracetamol 122 centers in 54 countries 200,000 children 6-7 yr Dose dependent increase in prevalence and severity of asthma > once per year: OR 1.61 (95% CI 1.46-1.77) once per month: OR 3.23 (95% CI 2.91-3.60) Association identified at almost all sites regardless of geography, culture, stage of development Dose dependent increase in prevalence and severity of asthma > once per year: OR 1.43 (95% CI 1.33-1.53) once per month: OR 2.51 (95% CI 2.33-2.70) Association identified at almost all sites regardless of geography, culture, stage of development Pooled odds ratio (OR) for asthma among subjects using acetaminophen was 1.63 (95% CI, 1.46 to 1.77). The risk of asthma in children among users of acetaminophen in the year prior to asthma diagnosis and within the first year of life was elevated (OR: 1.60 [95% CI, 1.48 to 1.74] and 1.47 [95% CI, 1.36 to 1.56], respectively). Only one study reported the association between high acetaminophen dose and asthma in children (OR, 3.23; 95% CI, 2.9 to 3.6). There was an increase in the risk of asthma and wheezing with prenatal use of acetaminophen (OR: 1.28 [95% CI, 1.16 to 41] and 1.50 [95% CI, 1.10 to 2.05], respectively). Beasley et al. Cross-cultural study 2011 Examine the risk of asthma rhynoconjunctivitis and eczema in adolescents using paracetamol 122 centers in 54 countries 320,000 children 13-14 yr old Systematic review and meta-analysis of epidemiol ogic studies Etminan et al. 2009 Quantify the association between acetaminophen use and the risk of asthma in children and adults. Thirteen cross-sectional studies, four cohort studies, and two case- control studies comprising 425,140 subjects

Consistency of the association across geography, culture and age (II) Longitudinal birth- cohort study Amberbir et al. 2011 Investigate the independent effects of paracetamol and geohelminth infection on the incidence of wheeze and eczema in a birth cohort. investigate the associations between infant and childhood paracetamol use and atopy and allergic disease at 5-6 years. population-based cohort of 1,065 pregnant women from Butajira, Ethiopia, Paracetamol use was significantly associated with a dose-dependent increased risk of incident wheeze (adjusted odds ratio = 1.88 and 95% confidence interval 1.03-3.44 for one to three tablets and 7.25 and 2.02-25.95 for 4 tablets in the past month at age 1 vs. never), but not eczema. Paracetamol exposure before the age of 15 months was associated with atopy at 6 years [adjusted odds ratio (OR)=3.61, 95% confidence interval (CI) 1.33-9.77]. Paracetamol exposure between 5 and 6 years showed dose-dependent associations with reported wheeze and current asthma but there was no association with atopy. Compared with use 0-2 times, the adjusted OR (95% CI) were wheeze 1.83 (1.04- 3.23) for use 3-10 times, and 2.30 (1.28-4.16) for use >10 times: current asthma 1.63 (0.92- 2.89) for use 3-10 times and 2.16 (1.19-3.92) for use >10 times: atopy 0.96 (0.59-1.56) for use 3- 10 times, and 1.05 (0.62-1.77) for use >10 times. Dose response association of paracetamol use and asthma (adjusted odds ratio, 1.20; 95% CI, 1.12 1.28; p value for trend 0.001). Wickens et al. Birth cohort study 2011 New Zealand Paracetamol exposure between birth and 15 months in Christchurch (n=505) and between 5 and 6 years for all participants (Christchurch and Wellington) (n=914). Outcome data collected at 6 years for all participants. Logistic regression models were adjusted for potential confounders Data from the Third National Health and Nutrition Examination Survey (U.S.) Participants aged between 20 and 80 years, with complete data for relevant exposures, outcomes Cross-sectional analysis McKeever et al. 2005 To investigate the associations between use of pain medication, particularly paracetamol, and asthma, COPD, and FEV1 in adults.

Consistency of the association across geography, culture and age (III) Shaheen et al. Case-control study 2000 To investigate whether frequent use in humans was associated with asthma. Adults aged 16 49 years registered with 40 general practices in Greenwich, South London. Frequency of use of paracetamol and aspirin was compared in 664 individuals with asthma and in 910 without asthma. The network compared 521 cases with a diagnosis of asthma and reporting of asthma symptoms with 507 controls with no diagnosis of asthma and no asthmatic symptoms across 12 European centres. After controlling for potential confounding factors OR for asthma, compared with never users, was 1.06 (95% CI 0.77 to 1.45) in infrequent users (<monthly), 1.22 (0.87 to 1.72) in monthly users, 1.79 (1.21 to 2.65) in weekly users, and 2.38 (1.22 to 4.64) in daily users (p (trend) = 0.0002). This association was present in users and nonusers of aspirin. Weekly use of paracetamol, compared with less frequent use, was strongly positively associated with asthma after controlling for confounders. OR 2.87 95% CI 1.49-5.37 No association was seen between use of other analgesics and asthma. Shaheen et al. Multicentric case- control study 2008 To examine whether or not frequent paracetamol use is associated with adult asthma across Europe.

Evidence for causal association between paracetamol and asthma Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative: 1) consistency of the association across geography, culture and age; 2) strength of the association (comparative studies); 3) the dose-response relationship between paracetamol exposure and asthma; 4) the coincidence of the timing of increasing asthma prevalence and increasing paracetamol use; 5) our inability to identify any other abrupt environmental change that could explain this increase in asthma morbidity; 6) the relationship between per-capita sales of paracetamol and asthma morbidity cross countries; 7) plausible mechanism: glutathione depletion in airway mucosa caused by paracetamol . McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).

Comparative studies Source Year of study 2000 Study objective Population Results Case-control study Shaheen et al. Determine if frequent paracetamol use is a risk factor for asthma. Adults aged 16-51 yr in South London Cases: n = 720 (51% response rate) Controls: n = 980 (49% response rate). 9400 women Never users: OR 1.06 (95% CI 0.77-1.45); Monthly users: OR 1.22 (95% CI 0.87-1.72); Weekly users: OR 1.79 (95% CI 1.21-2.65); Daily users: OR 2.38 (95% CI 1.22-4.64); P value for trend = 0.0002 Increased risk of wheezing before 6 mo for offspring of frequent paracetamol users over 20-32 wk prenatally: OR 2.34 (95% CI 1.24-4.40). Prospective cohort study Shaheen et al. 2002 Examine the relationship between prenatal paracetamol use and wheezing in offspring at 6 mo. Examine the relationship between paracetamol use and new onset of asthma Compare the incidence of adverse reactions among children administered paracetamol or ibuprofen Prospective cohort study Barr et al. Nurses Health Study 2004 73,321 women (44-69 yr) Increased risk of diagnosis of new-onset asthma with frequency of use Adjusted RR 1.63, 95% CI 1.11-2.39 Dose dependence: p value for trend = 0.006 Among 1879 children with pre-existing asthma, outpatient visits for asthma were lower in the ibuprofen arm than the paracetamol arm (RR 0.56 95% CI 0.34- 0.95); + dose-dependence Hospitalizations were nonsignificantly lower (RR 0.63 95% CI 0.25-1.60). Randomized double blind trial without placebo Boston University Fever Study 2002 84,000 febrile children Age 12 yr Randomly assigned paracetamol or low dose ibuprofen, or high dose ibuprofen

Evidence for causal association between paracetamol and asthma Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative: 1) consistency of the association across geography, culture and age; 2) strength of the association (comparative studies); 3) the dose-response relationship between paracetamol exposure and asthma; 4) the coincidence of the timing of increasing asthma prevalence and increasing paracetamol use; 5) the relationship between per-capita sales of paracetamol and asthma morbidity and across countries; 6) our inability to identify any other abrupt environmental change that could explain this increase in asthma morbidity; 7) plausible mechanism: glutathione depletion in airway mucosa caused by paracetamol . McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).

Coincidence of the timing of increasing asthma prevalence and increasing paracetamol use (4) Varner et al. Systematic review 1998 Investigate relationship between substitution of aspirin with paracetamol and increased asthma prevalence in developed countries. Systematic review of U.S. studies Epidemiologic trends, known biologic effects of cytokines and PGE2 on allergic sensitization, and a potentially relevant pharmacologic effect of aspirin used to explain a component of the increasing prevalence of childhood asthma in the United States. Although other changes in the environment have been suggested that might explain an increase in childhood asthma, none so easily explains the rapid increase in asthma in the 1980s and the subsequent levelling off of asthma prevalence over the last 15 years. Furthermore, the prevalence of childhood wheezing in 36 countries around the world is predicted by each country s per- capita sales of paracetamol . McBride (2011) (5)

Relationship between per-capita sales of paracetamol and asthma morbidity and across countries (5) Ecologic Study Newson et al. 2000 Examine the rate of Asthma and aggregate consumption of acetaminophen in 1994-95. English speaking countries in the ECHRIS study. Prevalence of wheeze increased by 0.52% for 13-14 yr olds; By 0.26% for young adults, For each gram increase in per capita paracetamol sales. Prevalence of childhood wheezing in 36 countries around the world is predicted by each country s per-capita sales of paracetamol.

Possible mechanisms Paracetamol Lack of suppression of cyclooxigenase pathway Low glutathione level **IgE mediated immune response Inability to counteract oxydative stress **Toxic effects of n.acetyl-p- benzoquinone mine Defective antigen processing Lung Injury with Bronchoconstriction Source: Eneli et al. 2005

Proof onus McBride (2011) explicitly warns against the use of Paracetamol in children with asthma or at risk for asthma and claims that if further evidence is required, then this is for documenting product safety rather than the contrary. This explicitly addresses the reluctance of sceptical commentators to accept such evidence as a sufficient basis for practice change and for establishing a causal relationship between paracetamol and asthma, on grounds that it does not result from randomized clinical trials (Eneli et al. 2005, Allmers et al. 2009, Johnson and Ownby, 2011; Karimi et al., 2006, Wickens et al. 2011, Chang 2011).

Hierarchy reversal = chronological 1. Non-controlled observational studies (e.g. ecological studies) 2. comparative studies (case-controlled retrospective) 3. prospective studies + evidence coming from cellular and molecular studies.

However, the justification of this hypothesis is neither exclusively focused on the exclusion of confounders (Vandenbroucke s point 1), nor based on utility functions(Vandenbroucke s point 2). Instead, several independent pieces of evidence jointly support a given hypothesis and are considered to do at least as good a job as the one-shot proof presumably provided by an RCT (or a meta-analysis of RCTs).

1. Both according to Vandenbrouckes point 1, as well as to the recent contribution by Aronson and Haube, (or Howick s contribution to the topic) case reports and observational data are considered sufficient evidence for causal claims to the extent that possible confounders (i.e. alternative causes for the experimental result) can be confidently excluded. This kind of reasoning also guides the general framework of evidence hierarchies: The higher the likelihood that the study design rules out more confounders than others, the higher it is settled in the ranking. And it is a straightforward consequence of the method of hypothesis testing, which is an hypothetico/deductive mode of investigation, for which the evidence is supposed to refute the null hypothesis. 2. Vandenbroucke s call on priors instead (point 2) invokes a Bayesian epistemology, where hypotheses are assigned probabilities and these are updated in the light of new data. 3. In McBride s, the causal hypothesis is assessed abductively, by putting things together and inferring the implications of their joint occurrence. The hypothesis of causal connection provides a unified explanation of the different pieces of evidence, which would otherwise need a series of distinct explanatory facts.

Epistemological paradigms Epistemology Method Assumptions Justification of lower level evidence Unificationist (qualitative abduction) Connection of data in light of explanatory hypothesis Connectedness (ontological) Explanatory power of hypothesis in light of data. (see also recent proposal of integration of Bradford-Hill criteria: Stegenga, 2011; Howick et al. 2009). Inductive-Bayesian (quantitative abduction) Bayes theorem Principle of total evidence - coherence Probability of hypothesis given evidence Hypothetico- deductive (statistical mode) Hypothesis testing: likelihood of evidence if H0 = true (p-value) Homogeneous populations with regard to all possible relevant causal factors (randomization) Only if alternative hypotheses (confounders) can be safely excluded, or treatment effect swamps them by a statistically significant amount (Howick, 2011).

Epistemological paradigms The unificationist paradigm regards hypotheses as explanatory factors for the observed data. In order to be really explanatory they must accommodate as many data as possible. Data which fail to be taken into account are left unexplained, thus making the hypothesis less virtuous from a theoretical point of view. In the Bayesian paradigm hypotheses can be associated with any probability in the unit interval. The main requirement is coherence (in the mathematical sense of standard probability calculus) and that all available evidence is used: this because, bayesian epistemology tracks inductive uncertainty and all non-deductive logics are non-monotonic.

Non-monotonicity Principle of total evidence Nonmonotonicity means that an addition to the premises may invalidate some previous conclusion. The principle of total evidence responds to this issue and is an essential principle of uncertain inference (Carnap, 1947). Linda is getting out of the bank It is 4 pm. Linda is getting out of the bank and she is going to an antinuclear demonstration It is earlier than 4 pm.

Hypothetico-deductive method (statistical mode) Instead statistical hypothesis-testing is a kind of approach which admittedly follows a Popperian hypothetico-deductive method of scientific enquiry. And being this paradigm inherently deductive, it does not feel urged to address the issue of non- monotonicity. The very idea of hierarchies follows from the assumption that if you have a study which has the capacity to eliminates more confounders than another, than the former should trump the latter. Trumping means that higher level evidence discards any evidence of inferior ranking, and also makes it irrelevant.

Lexicographic rule for hierarchy implementation Higher level studies trump lower level ones: 1. when two studies of different levels deliver contradictory findings, then the higher in the evidence study is considered more reliable and is allowed to discard the lower level one 2. lower level evidence adds nothing to higher level one and thus it can be neglected without loss of information.

Lexicographic rule for hierarchy implementation The strongest way to interpret hierarchies is to claim that, because it is assumed that randomization provides a guarantee for causality, than it should follow that there is no guarantee of causality without randomization. No randomization in, no causes out Cause

I think all instruments of investigation play an important role in scientific investigation. Analytical approaches work with truth conditions (RCTs set strict desiderata for considering sthg to be a cause for sthg else) but have strong methodological (only certain kind of evidence is at all meaningful) and epistemological limitations (failure to account for important phenomena such as causal interaction - see Muller). Inductive approaches work with imperfect indicators (P(E/H); dose- response relationship; specificity of the association); or theoretical virtues such as the explanatory power), thus not only they can, but they must take into account all evidence (non-monotonicity). Nevertheless they can give you no sure-fire guarantee that the delivered result demonstrates the truth of the hypothesis.

1. Probabilistic Hypothesis of Causal Connection From the time a risk is not known, to the moment in which it is incontrovertibly proven to be causally associated with the drug, there is a period of evidence accumulation which constitutes a state of partial and imperfect (but continuously increasing) knowledge. In this period it cannot be claimed that there is a causal link between the drug and the detected risk; but neither can we behave as if we knew nothing about it. Still, the latter attitude is precisely the only possible policy allowed by an epistemology grounded on hypothesis rejection.

2. Implementation of the precautionary principle Following the precautionary principle, you are not supposed to wait for the causal connection between harm and suspected drug to be certain, before you take adequate countermeasures, but instead, you should act as soon as the probability of causal connection is high enough to recommend countermeasures because of a negative risk/benefit balance. This probability might be also very low, in case the risk magnitude is considerably big with respect to the expected benefit. The frequentist mode of summarizing statistical data, following which hypotheses may only be accepted or rejected, cannot be of any use to this purpose.

3. Problem of external validity (ontological The problem of external validity and causal interaction is dramatic in the case of side- effects.

Conclusion My aim is to raise awareness about the different epistemological paradigms underlying the distinct evidence policies we may intuitively endorse. The take-home lesson is that different epistemologies grant different methodological actions and impose different, which in turn bring about relevant practical implications. Thus it is worthy to bear in mind the criteria underlying our evidence constraints whether we want to rank it or not, or else to reverse rankings.