Real-World Evidence for Medical Device Safety and Efficacy

Real World Evidence for Medical Device Safety and Efficacy Prof. Bernard Black Northwestern University, Pritzker School of Law and Kellogg School of Management April 2021 1

Need for big datasets You can t study real world evidence if you don t have it. Registries are valuable, but not enough, even apart from cost They don t include a control group:people who didn t get the device Can only compare device A to device B Cannot compare to not-A. People who receive not-A are outside the registry Even for comparing device A to competing device B: Selection bias if different hospitals or physicians make different device choices. Which correlate with unobserved patient health, or hospital/physician skill We often need big data sets: Linked claims data and EHR data with device id captured somewhere For very large populations 2

US is terrible at data consistency across time and datasets Claims data: Medicare Fee-for-Service data is often valuable But hard to get commercial payer data Nearly impossible to link commercial and Medicare FFS data CMS does not even collect Medicare Advantage data EHR data No consistency across EHRs If you ve seen one Epic EHR, you ve seen one Epic EHR And have no idea what the next one looks like Data dictionaries often incomplete or don t exist CMS can try to impose common data model (PCORnet, for example) 3

US is also terrible at linkage across datasets With rare, slow, expensive exceptions Cannot link Medicare FFS claims to commercial claims Cannot link across commercial payors Every state Medicaid dataset is different Cannot link claims data to EHR data Cannot easily link either type of data to mortality data Linking to NCHS death index is so expensive that researchers can t afford it The IRB approval process is often: Hugely more painful than it needs to be Some IRBs are decent, some are terrible Picky, slow, hard to navigate 4

COVID as eye-opener We can run vaccine trials Faster than we thought we could. Huge success! A message there for devices, about agency-industry cooperation? But after that: US is huge laggard For real-world vaccine safety and efficacy If you want to know about Pfizer-BioNTech, ask Israel, not the U.S. If you want to know about Moderna, ask [well, no one] The only data we have is risk if vaccinated. With no comparative data for similar people if not vaccinated The same problem with comparing A to not-A faced by registries If you want to know about reinfection risk, ask [other countries, not U.S.] 5

Linked, quality datasets as Healthcare Research Infrastructure We re bad at it. Job 3 is to get much better Job 1 is to recognize how bad we are We cannot do population-based research at speed We often cannot do it at all. Job 2 is to recognize the urgent need to get much better Cannot happen privately; incentives are not there CMS, FDA, CDC can lead, if they choose to Some examples Government data should be free to academic researchers, or nearly so Absurd to get US government funding to obtain government data Use 1/3 to pay university overhead, and the rest to pay the government for its own data Datasets should routinely include geographic identifiers (5-digit zip code) Non-identifiable datasets in research institutions should survive project expiration Reusable much more easily; with periodic audits for data security 6

But ok, lets imagine good data exists I m tasked with commenting on: How to use real-world evidence For an interesting, but artificial example What data would we want, how could we use it? What research designs would make sense? 7

Need data on Medicare beneficiaries (age 65+) For the artificial pancreas example we need: Large sample, to detect low-frequency adverse events Trial sizes (200 each, 100 treated) are very small Want to study range of macro outcomes (death, AMI, stroke, diabetic retinopathy, etc.) Low frequency outcomes, so need large sample Long time frames, so we won t have answers quickly 8

Imagine a real world trial, with feasible data Dr. Farmer imagined a device; so I ll imagine a trial! Link claims data, EHR data (for HbA1c, glucose, cholesterol, blood pressure, etc.), mortality data [Medicare has, but data with cause of death from NCHS would be better], demographic data [ideally at individual level, from American Community Survey (ACS] Because results could vary by race/ethnicity, poverty, etc. Device numbers, to compare this device to competitors when they emerge CMS might say: We want a big trial. Everyone age 68-73 are eligible. But no one else. Once we see results, we might expand the age range. Can imagine other restrictions: Say BMI between 25 and 30. 9

Logic behind this artificial trial Why start at age 68? So we can use Medicare claims to measure comorbidities with 3-year lookback And compare age 68 to controls age 65-67 Why stop at age 73? Want an arbitrary cutoff. To compare younger elderly to older elderly Why limit the BMI range? Same idea, want sharp, arbitrary cutoffs. Two dimensions is better than one. Use experience to predict who is most likely to benefit. 10

This is a Regression Discontinuity (RD) design The observational study design that does best at replicating RCTs In a variety of research areas Core idea: people just outside the RD cutoffs are Observationally identical to those just inside Except for the running variable(s) [here age, BMI] Use regression to control for the running variables Inside the cutoffs: Intent to Treat Design Not everyone offered the device will take it Use inside the cutoffs as an instrumental variable (IV) for takeup Need high takeup, among those eligible, for study power 11

Two-dimensional RD Box RD design idea: Controls (outer box) are similar to treated in inner box, on all covariates except age, BMI Use regression to control for age, BMI Inner box: Intent to treat design Use IV to recover ATT (average treatment effect on the treated) for those who receive the device Can randomize within the inner box Age = 73 BMI = 25 BMI = 30 Age = 68 12

Results valid outside the box (just not too far) Whatever results we find for age 68-73: Will carry over to younger and older ages (just not too far) Whatever results we find for BMI from 25-30 Will carry over to lower and higher BMI (just not too far) Use variation within the box (by age, BMI) to predict variation outside Over time, CMS can expand the RD bounds In 2 years, expand upper bound to 75 In 4 years expand upper bound to 77 Same for BMI range. This is a variation on a stepped wedge design 13

Even better: Randomize within the RD Box Can be crude: By first letter of last name, say. Then you have a randomized controlled trial (RCT) within the box, and an RD design at the edges Block randomize: Within gender, by race/ethnicity If poor and minorities are hard to reach: stronger encouragement Can Also Use Crossover Design Make the controls within the box eligible after 2 years (say) Or 50% after 2 years, the rest after 4 years Can also exploit time variation Early versus later treated Do treatment effects vary with time since implantation? 14

Broad Idea: Design Design your observational study, up front Prospectively Don t just find some treated and control persons after the fact Propensity score matching/reweighting is better than nothing But you can do better Different types of devices will call for different study designs Goal for all causal inference: Come as close as you can to an RCT Multiple design ideas are available Get a good causal inference researcher to help with the design Epidemiologists and RCT people don t know this Register the study up front (as for a clinical trial) Design the observational trial while you in the clinical trial Work with FDA and CMS, get them to agree on the study design In parallel with trial-based FDA approval So when the device is approved, the observational study is ready to go 15