Implementing Fault-Tolerant Systems in Health IT

Installation and Maintenance of Health IT Systems Creating Fault-Tolerant Systems, Backups, and Decommissioning Lecture b This material (Comp 8 Unit 9) was developed by Duke University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC000024. This material was updated in 2016 by The University of Texas Health Science Center at Houston under Award Number 90WT0006. This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/.

Creating Fault-Tolerant Systems, Backups, and Decommissioning Learning Objectives 1. Define availability, reliability, redundancy, and fault tolerance (Lecture a) 2. Explain areas and outline rules for implementing fault tolerant systems (Lecture a) 3. Perform risk assessment (Lecture a) 4. Follow best practice guidelines for common implementations (Lecture b) 5. Develop strategies for backup and restore of operating systems, applications, configuration settings, and databases (Lecture c) 6. Decommission systems and data (Lecture c) 2

Creating Fault Tolerance: Computer Hardware Redundant and fault tolerant hardware costs more Computers are workstations and servers Workstations need little fault tolerance o No critical data used interchangeably Servers need redundancy and fault tolerance o Hot-swap hard drives o Hot-plug expansion cards o Error checking and correcting, hot-add memory o Redundant and hot-swap fans o Redundant power supply (PSU) Multiple servers o Clustered systems are complex but highly available o Mirrored servers less complex but highly available o Hot spare simplest configuration but requires effort after failure (Tulloch, 2005) 3

Creating Fault Tolerance: Data Storage Store data redundantly, so that single failures cause no loss RAID (Redundant Array of Independent Disks) for hard drives RAID 0 provides no fault tolerance! Speed increase only RAID 1 (disk mirroring): Fast reading, simple, easy RAID 5 (disk striping with distributed parity): increased speed & reliability with relatively few disks, complex o Critical systems should include a hot spare RAID 6 (disk striping with double distributed parity): increased speed & additional reliability with relatively few disks, similar to RAID 5 in complexity (Tulloch, 2005; RAID, 2012) 4

Example RAID Arrays 5

Creating Fault Tolerance: Data Storage (cont d) Store data redundantly, so that single failures cause no loss Distributed file system running over a network Distributed File System (DFS) for Windows o Used with File Replication Service (FRS) to duplicate data Others will depend on platform, can include ZFS (Solaris), AFS (general UNIX), GFS (RedHat) SAN (Storage Area Network), NAS (Network Attached Storage) o EMC2 and NetApp are large vendors Cloud or Hosted storage uses the Internet o Let someone else worry about drives! o Dropbox o iCloud o Amazon S3 o Windows Azure Storage (Tulloch, 2005) 6

Creating Fault Tolerance: Virtualization Types of Virtualization Storage virtualization discussed previously Server virtualization - virtual machines (VMs) o Virtual Machine = Software emulation of physical environment o Server running VMs called a VM host multiple VMs run on single host o Advantages Easy upgrading and scalability Simplified hardware management and fault tolerance Easy to integrate existing systems and infrastructure o Disadvantage is a slight performance hit and more systems down with failure o Some services, e.g. Databases not perfectly suited for virtualization Best practices for each service are available from the service vendor. Infrastructure virtualization o Everything accessed through remote interfaces o Contracted level of service is important to specify o Simple devices + Internet access = Infrastructure as a Service (IaaS) (Sanford, 2010) 7

Creating Fault Tolerance: Off-Site Hosting and Access Hosted servers are similar to hosted storage, but can maintain an entire environment. Web server hosting is early example Virtual Servers in the cloud System hardware extremely reliable and fault tolerant, backed by service guarantees. Ensure availability for servers with: Redundancy & fault tolerance in network infrastructure: o Switches with Spanning Tree o Routers with secondary or backup links o Multiple Internet connections: Multihoming Uninterruptible Power Supply (UPS) & backup power in key areas, e.g. server rooms, wiring closets, critical PCs 8

Creating Fault Tolerance: Software as a Service (SaaS) SaaS, also known as Application Service Provider (ASP) or Cloud provider Benefits: No local hardware admin costs (except network access) Service contract guarantees very high fault tolerance Accessible from PCs, tablets potentially anything with a web browser Drawbacks Cost grows as usage grows not a fixed cost Network access can fail whose fault is it? o Internet Access Provider o SaaS Host Provider o SaaS Company or software 9

Creating Fault-Tolerant Systems, Backups, and Decommissioning Summary Lecture b Best Practices for providing fault tolerant computer hardware, data storage, virtualization, remote hosting, and network access 10

Creating Fault-Tolerant Systems, Backups, and Decommissioning References Lecture b References RAID [cited 2012 January 31]. Retrieved from: http://en.wikipedia.org/wiki/RAID Sanford, R. (April 2010) Electronic Health Records Need a Fail-Proof Foundation to Deliver on Quality, Economy Promises. Health News Digest. Available from: http://www.healthnewsdigest.com/news/Guest_Columnist_710/Electronic_Health_Records_Need_ a_Fail-Proof_Foundation_to_Deliver_on_Quality_Economy_Promises_2_printer.shtml Shackhow, T. et al. (June 2008). EHR Meltdown: How to Protect Your Patient Data. Fam Pract Manag, 15(6), A3-A8. Available from: http://www.aafp.org/fpm/2008/0600/pa3.html Tulloch, M. (April 2005) Implementing Fault Tolerance on Windows Networks . Available from: http://www.windowsnetworking.com/articles_tutorials/Implementing-Fault-Tolerance-Windows- Networks.html Images Slide 5: RAID 0, RAID 1, RAID 5, RAID 6 [en:User:Cburnett]. c2006 [updated 2000 Jan 28; cited 2006 Feb 15]. Available from: http://commons.wikimedia.org/wiki/Redundant_array_of_independent_disks 11

Installation and Maintenance of Health IT Systems Creating Fault-Tolerant Systems, Backups, and Decommissioning Lecture b This material was developed by Duke University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC000024. This material was updated in 2016 by The University of Texas Health Science Center at Houston under Award Number 90WT0006. 12