Global Digital Health Network Valuation Framework Overview

Software Global Goods Valuation Framework Global Digital Health Network 27 February 2019

Project Approach 1 20170619_Digital_Health_Final_Slides_v6.pptx

Key Definition and Assumption Global Goods: Mature digital health global good software is software that is Free and Open Source Software(FOSS), is supported by a strong community, has a clear governance structure, is funded by multiple sources, has been deployed at significant scale, is used across multiple countries, has demonstrated effectiveness, is designed to be interoperable, and is an emergent standard application. Software global goods can be categorized within a maturity framework: https://wiki.digitalsquare.io/index.php/What_are_Global_Goods#Maturity_Model Assumption: The value of the software global goods we collectively have invested in and built over the last decade (or more) is very high. 2 20170619_Digital_Health_Final_Slides_v6.pptx

Backup: Existing TCO tools/methodologies informed tool design Tool Organization ness Useful- Notes Exhaustive detail of cost buckets and extensive intr0 to ICT procurement Incorporates build vs. buy vs. outsource comparison, illustrates differences at cost bucket level Health Metrics Network (HMN) ICT Procurement High HMN Highly detailed scoring of readiness for mHealth/eHealth along 5 axes Uses a Pilot, Scale, and Sustain paradigm, may want to incorporate into tool Optimize, Planning an information systems project High WHO and PATH Medium- High Coverage breadth, considers cost of holistic interlinked ICT solution Includes costs of governance (eg. workshops, external advisors) Asian Development Bank (ADB) Tool Asian Development Bank Medium- High Good coverage of GG deployment costing buckets Off the shelf and easy to use Dimagi CommCare TCO Tool Dimagi For both self-hosted on-premise and cloud options Granular breakdown of software layer costs (eg. database, OS, virtualization) Granular breakdown of IT labor types (eg. problem resolution, DBA) Medium- High Gartner Cloud vs. Hosted TCO tool Gartner The MAPS Toolkit mHealth Assessment and Planning for Scale Focused on scaling up existing pilot projects Scoring methodology for technology gaps useful UNF, JHU-GMI, WHO, HRP Medium Assesses regulatory compliance for eHealth Regulations may affect cost e.g. Right to delete, data protection, multiple languages # of facilities (primary/secondary/tertiary) may affect cost Global Observatory for eHealth Medium WHO Medium Helps assess maturity level for eHealth/mHealth project along 5 dimensions Maturity Model Dimagi WHO mHealth Technical Evidence Review Group Medium Costs for reporting and assessment should be factored in mERA Checklist Expansive technology costs section Links to existing reports / white papers. K4Health, The mHealth Planning Guide Medium USAID, JHU CCP Likely low Provides mobile connectivity ratings, could be an input into costing Mobile Connectivity Index GSMA Provides sustainable energy score, low likelihood that this will be key input into digital health costing Readiness for Investment in Sustainable Energy Likely low World Bank HealthEnabled and the WHO Low Does not yet exist Global Digital Health Index 3 20170619_Digital_Health_Final_Slides_v6.pptx

For focus global goods, separate cumulative development cost into retrospective vs. ongoing Cumulative Development Cost for Core GG Country / Project Cost Component Retrospective Development Cost Ongoing Cost Retrospective Project Cost Approach / COCOMO code-based method FTE-based method Revenue based method Recent incurred costs Projected budget Data requests from example projects Interviews with project heads Source 4 20170619_Digital_Health_Final_Slides_v6.pptx

For retrospective development cost, three distinct approaches utilized FTE Based Revenue Based Code Based Estimate historical headcount dedicated to core Global Good development Total grants amounts received for core development of global good Established effort estimation model (COCOMO 81). Approach Estimate code effort based on lines of code, development "style" and product/team attributes For organization, FTE easier to count and less sensitive than spend data. Accurate upper bound on development cost Widely used methodology Benefits Easily computable based on code May be easy to obtain if organization received a small # of grants May be challenging to estimate historical effort if detailed records / project plans not available Contributions to code base funded by project grants may be difficult to account for Expected to be overestimate: Evolution of software development practices since model developed May systematically overestimate open source effort given low overhead, self-selection of interest /talent Challenges Volunteer contributions difficult to value Volunteer contributions difficult to value 5 20170619_Digital_Health_Final_Slides_v6.pptx

Five criteria used to select three focus global goods Selection Criteria OpenMRS CommCare High, large range of use cases (HIV, antenatal care, malaria) enabled by self-serve capabilities Front line health worker apps; data entry / decision support 3 options: SaaS subscription model Implementation services from Dimagi / partners Open source (self- implement) High, highly adaptable based on core & community- maintained add-on modules Serve as EMR for different use cases and scales Medium Focused on HR (employee mgmt, recruiting, training, certification) Range of functionality Pure open source Implementation by partners Past implementations primarily supported by IntraHealth through Capacity/CapacityPlus projects Range of different business models Front line health workers Central MoH (eg. doctors, system admin, nurses, public health officials) High Adoption, over 60K active mobile users Mature with 300+ completed projects Health use cases comparable to OpenSRP Self-service capabilities unique Central MoH (eg. doctors, system admin, nurses, public health officials) Central MoH (eg. nurse manager, community health manager, doctor) Variety of end users High adoption > 1800 locations in 80 countries Mature, >20 implementations of sizable scale High Uptake/ Maturity Comparable to commercial EMR (HIMMS Stage 3) Comparable to commercial HR software (eg. Taleo) Comparability 6 20170619_Digital_Health_Final_Slides_v6.pptx

Total Cost of Ownership Cost Drivers Cumulative Development Cost for Core GG Country / Project Cost Major Releases Initial Product Development Infrastructure Development Maintenance Modification Community Deployment Adaptation Operations Overhead Overhead System Support Admin / Helpdesk Support Implementation Services Equipment Replacement Monitoring & Evaluation Software Customization Software Development Implementer Support Training / Academies Infrastructure Costs Software Licensing Software Licensing Documentation Requirements Requirements Development Development Maintenance Procurement Maintenance Localization Integration Equipment Hardware Overhead Software Training Training Testing Design Design Test Test combined in agile 7 20170619_Digital_Health_Final_Slides_v6.pptx

8 20170619_Digital_Health_Final_Slides_v6.pptx

9 20170619_Digital_Health_Final_Slides_v6.pptx

10 20170619_Digital_Health_Final_Slides_v6.pptx

The framework and its parts The framework that we released has a user s manual and two spreadsheets and information on them and how to use them can be found here: https://www.usaid.gov/cii/software-global-goods-valuation-framework Software Global Goods Valuation Framework- An excel-based tool with step-by-step instructions for users: https://www.usaid.gov/documents/1864/software-global-goods-valuation-framework User s Guide- A PDF document to help users understand the background, context, and methodology contained in this document, as well as the importance of software global goods in the changing global health landscape: https://www.usaid.gov/documents/1864/software-global-goods-valuation-framework-users-guide Budgeting for Country-Level Digital Health Implementations- An excel-based framework that helps countries and stakeholders plan and budget for digital health global goods implementations at the country level: https://www.usaid.gov/documents/1864/budgeting-country-level-digital-health-implementations 11 20170619_Digital_Health_Final_Slides_v6.pptx

OpenMRS Profile and Costing 12 20170619_Digital_Health_Final_Slides_v6.pptx

Quick Facts Description OpenMRS is an open source electronic medical record platform. Implementers can create an electronic medical record by combining a selection of OpenMRS components with bespoke code. OpenMRS is organized in a modular fashion, roughly divided into Platform (core data record), Reference App (example end user application) and other modules including integration modules (eg. SSO) and use case modules (eg. Ebola) Creation Date 2004 Open source, utilize partners for implementation Business Model Adoption 60+ countries (1800+ sites) # Countries OpenMRS has had high adoption in 1800+ locations in 60+ countries. Many for a particular time period or limited in scope (eg. HIV) There have been a number of significant large-scale LMIC deployments Both Kenya and Uganda have used OpenMRS core to build own EMRs (KenyaEMR, UgandaEMR) with wide rollout Mozambique, 6 clinical partners of CDC/USAID created local community of developers / implementers called eSaude, planning deployment across country Core EMR record storage + add ons Functionality Point of care health workers MoH employees End users Community Seconded OpenMRS work is performed primarily by Regenstrief Institute and PIH. Other contributing orgs include ThoughtWorks (IT consultancy pro bono) , SolDevelo (Polish IT firm, pro bono) A number LMIC companies perform implementation under supervision of ministries or NGOs In 2016, over 160 people contributed to OpenMRS repositories (not counting specific implementations) in over 100 repositories Originally HIV, broadened to full spectrum records Health Focus 1.8M LOC primarily Java, Javascript, HTML Programming Language(s) Source: Interviews with OpenMRS leadership, publicly available information 13 20170619_Digital_Health_Final_Slides_v6.pptx

OpenMRS value proposition consistent with general benefits of POC EMRs in developing countries Decision support for drug ordering including allergy and drug incompatibility warnings Warnings for abnormal laboratory results Improved legibility of clinical notes for future reference (potentially at cost of physician efficiency) Enhanced Patient Care Healthcare information reporting largely driven by funding agency / government requirements Point of Care EMRs can improve report quality and efficiency: Data quality improved because errors more easily recognized in context of individual patient More accurate aggregation compared to hybrid paper / data entry systems Cheaper recurring costs for report aggregation compared to hybrid paper / data entry systems Improved Reporting / Monitoring Increased support for clinical research, allowing for viewing of past medical records Improved supply chain with better inputs into drug restocking Other Source: Fraser; Biondich; Moodley; Choi; Mamlin; Szolovits, Implementing electronic medical record in developing countries 2005 14 20170619_Digital_Health_Final_Slides_v6.pptx

Cost summary Direct spend on development of OpenMRS estimated at $4-5M, though valuation of in-kind contributions suggests ~$8M is more reflective of total cost Retrospective Development Cost Cumulative Development Cost for Core GG COCOMO methodology implies market replacement cost of $76M Optimal annual run rate estimated at $2.8M $1.1M predicted to keep the lights on $1.7M additional funding required for further development Ongoing Cost Country / Project Project Cost Cost AMPATH project in Kenya estimated to cost $1.4M over 5 years 15 20170619_Digital_Health_Final_Slides_v6.pptx

Retrospective Development Cost FTE and revenue analysis are broadly aligned Direct cost estimated at ~$4-5M + value of in-kind contributions which is corroborated by revenue analysis Analysis of paid FTEs suggests ~$4M spent on labor Total cumulative grants allocated to OpenMRS core development estimated at $5-8M 37 FTEs from Regenstrief & PIH $100k fully loaded compensation x = $3.7M Backing out assumed indirect charge for funding through universities implies $3-5M in direct spend However, value of additional in-kind contributions estimated at $4.5M Estimated FTEs Market Salary In-Kind Valuation $25,0001 Secondees from SolDevelo 4 $100,000 Implementers4 $20,0002 14 $272,903 Independent contributors4 $100,0003 42 $4,165,879 Total 59 $4,538,782 1. Based on market rate for software developers in Poland 2. Based on average developer compensation in LMICs 3. Based on fair market value of fully loaded software developer compensation in HICs 4. Implementer and independent contributor commits estimated based on relative contribution in 2016 16 20170619_Digital_Health_Final_Slides_v6.pptx

Retrospective Development Cost FTE-based development cost FTE 11.9 12 11.6 11.3 11.3 Implementer (Est)1 Independent (Est)1 2.0 10.4 10.4 1.9 1.9 1.9 SolDevelo 10 PIH 1.7 1.7 Regenstrief 8 5.9 5.7 5.6 5.6 5.9 5.9 5.9 5.2 5.2 6 1.0 1.0 1.0 4.2 3.9 3.8 4 1.0 2.9 2.9 2.9 1.0 1.0 1.0 2.2 2.4 2.3 2 3.5 3.5 3.0 2.9 1.1 2.8 2.8 2.0 2.0 2.0 2.0 0.6 1.5 1.5 0.5 0 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 FTE-based development cost estimates 97 total man-years 1. Implementer, Independent total FTE-years estimated based on relative contribution in 2016, distributed over 2008-2016 Source: Regenstrief FTE numbers, SolDevelo Pro Bono report 17 20170619_Digital_Health_Final_Slides_v6.pptx

Retrospective Development Cost Code-based (COCOMO 81) development cost Small team, less than rigid requirements Development Style1 Basic estimate: 523 man-years Lines of Code Organic Semi-detached Embedded 1.8M Ratings Cost Drivers Very Low Low Nominal High Very High Product attributes Required software reliability Size of application database Complexity of the product Hardware attributes Run-time performance constraints Memory constraints Volatility of the virtual machine environment Required turnabout time Personnel attributes Analyst capability Applications experience Software engineer capability Virtual machine experience Programming language experience Project attributes Application of software engineering methods Use of software tools Required development schedule X 0.75 0.88 0.94 0.85 1.00 1.00 1.00 1.15 1.08 1.15 1.40 1.16 1.30 Core functionality, high product reliability/ complexity 0.70 1.00 1.00 1.11 1.06 1.30 1.21 EAF 0.87 0.87 1.00 1.00 1.15 1.07 1.30 1.15 Multiplier = 1.46 Server / PC hosted, low reqs 1.46 1.29 1.42 1.21 1.14 1.19 1.13 1.17 1.10 1.07 1.00 1.00 1.00 1.00 1.00 0.86 0.91 0.86 0.90 0.95 0.71 0.82 0.70 Assume average for team cost drivers = Intermediate estimate: 764 man-years Open source community, lower coordination 1.24 1.24 1.23 1.10 1.10 1.08 1.00 1.00 1.00 0.91 0.91 1.04 0.82 0.83 1.10 1. Organic projects - "small" teams with "good" experience working with "less than rigid" requirement Semi-detached projects - "medium" teams with mixed experience working with a mix of rigid and less than rigid requirements Embedded projects - developed within a set of "tight" constraints. It is also combination of organic and semi-detached projects.(hardware, software, operational, ...) 18 20170619_Digital_Health_Final_Slides_v6.pptx

Ongoing Cost Annual run rate for OpenMRS estimated at $1.1-2.8M Incurred/projected run rate ($M) Key takeaways 3.0 0.62 2.84 $670k operating costs in 2016 believed to be inadequate Lean operations not ideal for long-term sustainability Reliance on volunteers may create additional risk 0.41 2.5 0.44 0.22 0.28 2.0 1.69 0.24 0.16 0.03 1.54 0.21 1.5 $1.1M (40% of proposed budget) deemed necessary to "keep the lights on" 0.97 1.0 0.67 1.14 0.5 Additional $1.7M requested for ongoing development Majority of request comprises developer labor Additional costs include community support and overhead 0.56 0.0 Maintenance) Total Development (Major Releases/ Support Development Infrastructure Community 2016 Spend Overhead Actual spend Continued development "Keep the lights on" 1. Development spans both new releases and maintenance 19 20170619_Digital_Health_Final_Slides_v6.pptx

OpenMRS Contributor Analysis 20 20170619_Digital_Health_Final_Slides_v6.pptx

OpenMRS Contributor Analysis OpenMRS leadership aims to actively foster community involvement To facilitate analysis, OpenMRS leadership segmented development contributors and code base OpenMRS developers classified into Seconded (paid for contribution), Implementers (paid for implementation project) and Independent (volunteer) OpenMRS code base classified into Platform, Reference Application, Documentation and Tooling, and Other Quantitative assessment of code contribution reveals robust contribution by community For Docs and Tooling and Other repos (61 repos), >50% of contributions were made by independent volunteers Core Platform and Reference Application still dominated by seconded developers with >60% of contributions) Results roughly aligned with major open source projects For Docker project, 90% of contributors are volunteer and contribute >50% of code base Note: Contributions defined as code check ins 21 20170619_Digital_Health_Final_Slides_v6.pptx

3 groupings of OpenMRS developer contributors Seconded to Community ~10 contributors Implementers ~26 contributors Independent ~131 contributors Contributions by full/part- time employees funded by an organization to work on community-defined roadmap Contributions to code-base driven by specific implementation, typically compensated by implementation project Contributions based on interest / prior involvement without compensation. Include student participants in summer coding programs sponsored by Google Does not capture full community of non-developer contributors Note: Contributor #'s based on those who contributed code in 2016 Source: OpenMRS leadership team, 2016 Github commits 22 20170619_Digital_Health_Final_Slides_v6.pptx

Classified code repositories into 4 categories 102 repositories modified in 2016 considered in scope Core platform openmrs-core, openmrs-module-fhir, openmrs-module-owa, openmrs-module- webservices.rest, openmrs-standalone Platform (5 repos) Complete end user application Model for front end of other implementations Reference Application (35 repos) SDK for developers Developer manual User guide Documentation and Tooling (4 repos) All other modules Use case extensions eg. allergy, radiology, ebola Integration extensions eg. oauth, bamboo, sso Other (57 repos) 23 20170619_Digital_Health_Final_Slides_v6.pptx

Independent volunteers contributed >50% commits to Docs & Tooling and other repositories in 2016 Platform Reference App Docs & Tooling Other % Breakdown % Breakdown % Breakdown % Breakdown 82 712 70 1,590 15 878 81 2,319 100 100 100 100 180 (11%) 239 (34%) 256 (16%) 80 80 80 80 1,185 (51%) 8 527 (60%) (53%) 45 39 (5%) (64%) 59 61 (73%) (74%) 60 60 60 60 1 401 (17%) 2 (7%) 40 40 40 40 1,154 (73%) (0%) 434 (61%) 17 6 349 (40%) (24%) 13 12 20 20 20 20 (40%) 733 (32%) (16%) (15%) 8 9 9 (11%) (11%) (11%) 0 0 0 0 Contributors Contributions Contributors Contributions Contributors Contributions Contributors Contributions Robust community participation in Docs/Tooling/Other Independent Implementer Seconded to the Community 1. All github repositories other than "openmrs-core" Note: Contributions measured as # commits Source: Github commits 24 20170619_Digital_Health_Final_Slides_v6.pptx

Backup: Data on four open source communities shows mix of compensated vs. volunteer contributions Apache Spark KDE % Contributor Breakdown 100 10 34 36 40 44 Compensated 56 Volunteer 71 10% paid contribute > 50% of code changes 86 50 90 66 64 60 56 44 29 14 0 Core/ Platform Main Desktop Secondary Desktop Core/ Platform Main Desktop Secondary Desktop Apache Spark Docker Linux desktop environment Linux desktop environment Big data processing Application containerization Functionality TBD TBD ~200 ~1000 Community Size Year Begun 1997 1996 2014 2013 Novell Sun Microsystems Red Hat Novell Klar lvdalens Datako Google IBM Twitter Quantcast Docker Inc. Red Hat IBM Google Cisco Major Paid Contributors Note: Data for Gnome and KDE from 2007 study; Source: Apache Spark repository, https://gist.github.com/arun-gupta/7d5a373099ff831d7213; Managing the Bazaar: Commercialization and Peripheral Participation in mature, community-led Free/Open Source Software projects http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.462.6158&rep=rep1&type=pdf 25 20170619_Digital_Health_Final_Slides_v6.pptx