Device Failure Prediction Service at University of Wisconsin-Madison

Device Failure Prediction - Ankit Maharia - Pulkit Kapoor - Sreyas Krishna Natarajan UNIVERSITY OF WISCONSIN-MADISON

Outline Problem Statement & Motivation Methodology/Implementation ML Results Future Scope Conclusion UNIVERSITY OF WISCONSIN-MADISON

Problem Statement Build a service to facilitate creation of open dataset of device health metrics. Failure prediction using public dataset. UNIVERSITY OF WISCONSIN-MADISON

Motivation Limited public dataset availability for researchers. Avoid degraded data redundancy or complete data loss. Tuning background scrub speed UNIVERSITY OF WISCONSIN-MADISON

Methodology UNIVERSITY OF WISCONSIN-MADISON

Service Dependencies Pecan: Python web framework for creating REST API. ElasticSearch: To store device metrics MongoDB: For storing host id and host secret on service side. UNIVERSITY OF WISCONSIN-MADISON

Flow - Registering a host - Done via POST api call to: /register-host - Returns a unique host_id and host_secret which are to be used when sending metrics - Client can store it in its persistent storage. For ceph: we have stored it in the manager store. UNIVERSITY OF WISCONSIN-MADISON

Flow - Sending Device Metrics Done via POST api call to: /store-device- metrics Client sets host_id and host_secret in the request headers and posts the payload like below Service stores the payload to elastic search, replacing device serial number under smartctl_json with a SHA-1 hash UNIVERSITY OF WISCONSIN-MADISON

DEMO - - Demo 1 - - Demo 2

CEPH INTEGRATION UNIVERSITY OF WISCONSIN-MADISON

UNIVERSITY OF WISCONSIN-MADISON

UNIVERSITY OF WISCONSIN-MADISON

FAILURE PREDICTION UNIVERSITY OF WISCONSIN-MADISON

SMART (Self-Monitoring, Analysis and Reporting Technology) SMART is a monitoring system supported by most drives that reports on various indicators of drive health, including various types of errors, but also operational data, such as drive temperature, and power on hours of the drive. UNIVERSITY OF WISCONSIN-MADISON

Features UNIVERSITY OF WISCONSIN-MADISON

Features: Interesting SMART 5 (S5) - count of reallocated sectors. When a read or a write operation on a sector fails, the drive will mark the sector as bad and remap (reallocate) it to a spare sector on disk. SMART 187 (S187) - read errors that could not be recovered using ECC SMART 197 (S197) - count of unstable sectors. Some drives mark a sector as unstable following a failed read, and remap it only after waiting for a while to see whether the data can be recovered in a subsequent read or when it gets overwritten UNIVERSITY OF WISCONSIN-MADISON

Challenges How do we define a failure? - User rarely(never) uploads data regarding failure. This leads to lack of failure signal. - If metrics for a device is not reported on a day and it was reported the previous day, it can be assumed as a failure. - If a device has moved from one host to another then it could be marked as a failure. (Fixed and attached back) UNIVERSITY OF WISCONSIN-MADISON

Prediction Pipeline Processing of metrics dumped to elasticsearch index on daily basis - Adding a failure signal - Checking multipath - Validated flow using backblaze We used Backblaze dataset from 2013 to 2016. - Data was sampled to 3:2 (label 0 : 1) - Train # failure samples: 3250 - Test # failure samples: 350 Negative Label(0): Device did not fail UNIVERSITY OF WISCONSIN-MADISON

Results Best Model- Random Forest Model Precision Recall F1 1 0.75 0.67 0.71 2 0.55 0.83 0.66 UNIVERSITY OF WISCONSIN-MADISON

Future Scope - Dockerize the service (Done!) - REST API for prediction - Better Machine Learning Models - Script to publicly release dataset UNIVERSITY OF WISCONSIN-MADISON

Questions? UNIVERSITY OF WISCONSIN-MADISON

Thank You! UNIVERSITY OF WISCONSIN-MADISON