Examining Methods and Reporting in Voluntary Medical Male Circumcision Costing Studies

GHCC Stakeholder and Advisory Meeting Bill & Melinda Gates Foundation Boston Massachusetts, USA iHEA Boston 2017 Congress, Seattle, WA USA 8-11 November 2016 8-11 July 2017 Assessing the Quality of Voluntary Medical Male Circumcision Costing Literature Session: Introducing a Reference Case for Costing Global Health Interventions Benjamin Herzel, MS Institute for Health Policy Studies University of California, San Francisco (on behalf of the GHCC)

Introduction High-quality studies use appropriate methods and provide detailed reporting Appropriate methods produce precise and replicable results Detailed reporting allows results to be used for intended purpose and setting Objective: To examine the methods used and the frequency of reporting in a comprehensive dataset extracted from Voluntary Medical Male Circumcision costing studies.

Search Strategy Used Cochrane Collaboration methods HIV/AIDS term based on search filter developed by the Cochrane HIV/AIDS Group, used in >100 Cochrane reviews Cost term based on the best optimization of sensitivity & specificity for econ studies Developed by Wilczynski et al. (2004) and McKinlay et al. (2006) Study dates from 1990-2016

VMMC Dataset 29 studies identified and extracted 759 cost observations 11 countries represented 6 5 4 3 6 2 4 4 4 3 1 2 2 1 1 1 1 0

Reporting Scope 30 1 1 2 5 25 12 12 20 7 NUMBER OF STUDIES 23 N/A 22 NR 15 inferred explicit 10 17 16 15 5 6 6 0 geography time period target population incremental vs. full economic vs. financial

Reporting Sampling 30 5 7 25 16 17 20 NUMBER OF STUDIES N/A NR 15 20 inferred 19 explicit 10 8 9 5 2 5 1 3 2 2 0 Country Site Area Patient

Reporting Methods (Cost Level) 900 800 4 13 26 700 NUMBER OF COST OBSERVATIONS 227 269 600 373 N/A 500 7 NR inferred 400 17 337 explicit 300 479 200 343 100 182 0 Resource Allocation Top-down vs. Bottom-up Cost source

Reporting Inclusion of Costs 30 3 3 7 25 8 8 8 20 NUMBER OF STUDIES N/A 15 NR 23 inferred 19 explicit 21 21 10 18 5 3 3 0 Unrelated costs Research costs Above service delivery costs Overheads Cost omissions

Reporting Valuation 30 2 25 17 20 NUMBER OF STUDIES 21 17 N/A NR 15 inferred explicit 10 2 0 12 5 8 8 0 Currency year Discount rate Conversion method

Reporting Input Prices and Quantities 30 25 20 15 28 10 5 1 0 Ps and Qs reported Ps and Qs not reported

Economies of Scale 31% 38% analyzed discussed ignored 31%

Compared by Subgroup compared by SG 45% 55% not compared by SG

Methods Type of Cost economic 48% 52% financial only

Methods Discount Rate 3% 24% 0.03 0.06 4% NR N/A 69%

Methods Above service costs 7% 24% some costs included no costs included NR 69%

Methods Timing 7% cross-sectional 38% longitudinal 55% NR

Methods Sensitivity Analysis 24% comprehensive limited none 62% 14%

Conclusions VMMC costing studies exhibit heterogeneous reporting High-quality areas include costing purpose and inclusion/exclusion of overheads Low-quality areas include sampling methods, allocation methods, cost sources, and input prices and quantities. Methodological choices are varied, even in similar contexts (VMMC in SSA)

VMMC and TB studies compared Similar findings: Infrequent reporting on full vs. incremental and economic vs. financial costing Very sparse reporting on sampling methods Differences: 45% of VMMC studies report allocation method compared to approx. 4% of TB studies VMMC studies much more likely to discuss or analyze scale effects (69% vs 35% in TB studies)

Next Steps Developing similar datasets for all HIV interventions in LMICs. Using data on reporting and methods to construct an empirically-validated quality- rating system