Micro-Simulation Tool for Disease Modeling with Python Framework

MIST: Micro-Simulation Tool to Support Disease Modeling Jacob Barhak SciPy 2013 Austin TX, USA 26 Jun 2013

The Basics MIST stands for MIcro-Simulation Tool MIST is a Python framework that supports chronic disease modeling using computing power Disease models: Describe phenomenon observed in past trials Attempt to predict future disease progression Used to predict Costs / Quality of Life Jacob Barhak

From IEST to MIST MIST is a split from the Indirect Estimation and Simulation Tool (IEST) MIST is simplified: Simpler Installation using Python(X,Y) and Anaconda Simpler definitions avoiding complexity associated with estimation Simulation rules are simpler and more flexible MIST runs over the cloud! And on Sun Grid Engine (SGE) clusters Reproducibility and Traceability Simulations include extra Trace Back information Renewed test suite - using nose & some Bug fixes Jacob Barhak

Form Based User Interface Jacob Barhak

Simulation Language / Compiler Strict Expression language a subset of Python with extensions: Supported Types: Integer, Number, Expression, State Indicator, System Option Comparison: Eq, Ne, Gr, Ge, Ls, Le Boolean operators: Or, And, Not, IsTrue Special math: Inf, NaN, IsInvalidNumber, IsInfiniteNumber, IsFiniteNumber Mathematical functions: Exp, Log, Ln, Log10, Pow, Sqrt, Pi Other functions: Mod, Abs, Floor, Ceil, Max, Min Statistical: Bernoulli, Binomial, Geometric, Uniform, Gaussian Control and Data Access: Iif, Table Application specific: CostWizard MIST Python Script Features: Compiles into Python Syntax check upon expression definition Runtime Bound Checks Runtime recalculation due to out of bounds random error Compile Model Results Jacob Barhak

Monte Carlo Simulation Pop If random < OccurrenceProbability: If random < OccurrenceProbability: AffectedParameter = DefinedExpression AffectedParameter = DefinedExpression Mutli-Process State Transitions Process CHD Init Rules CHD Death No CHD MI Survive MI Process Stroke Death Pre State Transition Rules Post State Transition Rules Survive Stroke Stroke Death No Stroke Stroke Process Competing Mortality Other Death Alive Cost QoL Biomarkers Repeat Simulation Step Jacob Barhak

Monte Carlo Initialization: Distribution to Population Generation The system: Compiles distributions into initialization code before simulation Automatically resolves calculation order Can handle interdependencies more complicated than statistical functions Example: Age ~ 61+8.2*CappedGaussian3 Male ~ Bernoulli(803/1199) SBP ~ 133.4+16.4*CappedGaussian3 AgeAtDiagnosisOfDiabetes ~ Age - 8 Age Male 1 1 0 1 1 SBP AgeAtDiagnosisOfDiabetes 57.51415 45.76856 65.71445 37.79667 49.21742 65.51415 53.76856 73.71445 45.79667 57.21742 129.1721 137.4234 132.8542 147.5537 122.68 Good for: Using published aggregate data from clinical trial publications Avoiding using individual data that is typically restricted Allowing access to more population information Jacob Barhak

Reproducibility MIST stores random state of each simulation MIST can recreate a simulation from Trace Back upon request MIST records additional traceability information in compiled simulation files to help debugging Good For: Saving storage space Debugging Distributing results & publication Jacob Barhak

MIST Runs Over the Cloud! Anaconda drives MIST to run over the Amazon cloud! Batch mode MIST utilities allow: Submitting jobs to Sun Grid Engine (SGE) Running simulations Generating reports Combining reports from multiple repetitions/scenarios Star Cluster creates an SGE cluster on the Amazon Elastic Compute Cloud The Anaconda Amazon Machine Image (AMI) is used for the cluster master / nodes Good for: Cutting down computation time by renting computing power Saving initial and maintenance costs associated with a cluster MIST Cloud Jacob Barhak

Summary & Points to Remember MIST stands for MIcro Simulation Tool MIST runs over the cloud! MIST is free and available on GitHub: https://github.com/Jacob-Barhak/MIST The Reference Model for disease progression uses MIST Jacob Barhak

Acknowledgments Deanna J.M. Isaman - who is the spirit behind the great ideas. She taught me my first steps in disease modeling Morton Brown & William H. Herman for guidance, critical feedback, and growth environment Continuum Analytics and specifically: Benjamin Zeitler for creating the cloud AMI Ilan Schnell for his work on Anaconda. The very capable cluster builder & sys admin: Chris Scheller CAC / Cyber Infrastructure Team at U of M Michigan Python Users Group All those who developed free software used: including Python, Anaconda, Python(x,y), numpy, SciPy, nose, winpdb, Star Cluster, Ubuntu The IEST modeling framework was supported by the Biostatistics and Economic Modeling Core of the MDRTC (P60DK020572) and by the Methods and Measurement Core of the MCDTR (P30DK092926), both funded by the National Institute of Diabetes and Digestive and Kidney Diseases. The modeling framework was initially defined as GPL and was funded by Chronic Disease Modeling for Clinical Research Innovations grant (R21DK075077) from the same institute. The Reference Model and MIST were developed independently without financial support Jacob Barhak

The Reference Model Purpose: better understand disease progression by competition/comparison Process CHD CHD Death No CHD MI Survive MI Process Stroke Death Survive Stroke Stroke Death No Stroke Stroke Process Competing Mortality Other Death Alive Built from literature references and hence the name: The Reference Model A League / Consumers Report for disease models Uses Monte Carlo Micro-simulation, i.e. simulate each virtual individual Jacob Barhak

The Reference Model Ranks Model and Population Fitness Best Model Overall A1c changes BMI changes BP changes Lipid change Smoke changes MI Equation # Stroke Equation # CHD Death Equation # Stroke Death Equation # Treatment Improvement Correction FITNESS: LOW SCORE = GOOD FITNESS UKPDS33 Conventional UKPDS33 Intensive UKPDS33 Full ASPEN All Placebo ASPEN All Atorvastatin ASPEN Primary Placebo ASPEN Primary Atorvastatin ASPEN Secondary Placebo ASPEN Secondary Atorvastatin ASPEN Full ADVANCE Standard ADVANCE Intensive ADVANCE Asia Standard ADVANCE Asia Intensive ADVANCE EME Standard ADVANCE EME Intensive ADVANCE Eastern Europe Standard ADVANCE Eastern Europe Intensive ADVANCE Full ACCORD BP Standard Therapy ACCORD BP Intensive Therapy ACCORD BP Full 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 1 2 1 1 0 0 0 0 0 0 1 2 1 1 1 0 0 0 0 0 1 3 1 1 0 0 0 0 0 0 1 3 1 1 1 0 0 0 0 0 1 4 1 1 0 0 0 0 0 0 1 4 1 1 1 0 0 0 0 0 2 1 1 1 0 0 0 0 0 0 2 1 1 1 1 0 0 0 0 0 2 2 1 1 0 0 0 0 0 0 2 2 1 1 1 0 0 0 0 0 2 3 1 1 0 0 0 0 0 0 2 3 1 1 1 0 0 0 0 0 2 4 1 1 0 0 0 0 0 0 2 4 1 1 1 0 0 0 0 0 3 1 1 1 0 0 0 0 0 0 3 1 1 1 1 0 0 0 0 0 3 2 1 1 0 0 0 0 0 0 3 2 1 1 1 0 0 0 0 0 3 3 1 1 0 0 0 0 0 0 3 3 1 1 1 0 0 0 0 0 3 4 1 1 0 0 0 0 0 0 3 4 1 1 1 0 0 0 0 0 4 1 1 1 0 0 0 0 0 0 4 1 1 1 1 0 0 0 0 0 4 2 1 1 0 0 0 0 0 0 4 2 1 1 1 0 0 0 0 0 4 3 1 1 0 0 0 0 0 0 4 3 1 1 1 0 0 0 0 0 4 4 1 1 0 0 0 0 0 0 4 4 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 0 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 3 1 1 0 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 4 1 1 0 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1 2 1 1 1 0 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 2 2 1 1 0 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 2 3 1 1 0 1 1 1 1 1 2 3 1 1 1 1 1 1 1 1 2 4 1 1 0 1 1 1 1 1 2 4 1 1 1 1 1 1 1 1 3 1 1 1 0 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 3 2 1 1 0 1 1 1 1 1 3 2 1 1 1 1 1 1 1 1 3 3 1 1 0 1 1 1 1 1 3 3 1 1 1 1 1 1 1 1 3 4 1 1 0 1 1 1 1 1 3 4 1 1 1 1 1 1 1 1 4 1 1 1 0 1 1 1 1 1 4 1 1 1 1 1 1 1 1 1 4 2 1 1 0 1 1 1 1 1 4 2 1 1 1 1 1 1 1 1 4 3 1 1 0 1 1 1 1 1 4 3 1 1 1 1 1 1 1 1 4 4 1 1 0 1 1 1 1 1 4 4 1 1 1 Models 27.0 24.6 21.5 14.3 18.5 23.3 21.7 23.2 16.2 8.3 67.3 67.3 66.7 72.6 84.8 91.2 75.9 78.5 57.3 46.8 42.5 43.9 24.1 24.6 22.5 21.4 23.0 21.3 10.2 13.8 9.4 20.5 9.5 23.1 13.1 24.3 37.6 21.2 18.8 10.3 11.8 9.9 54.0 12.1 59.3 18.9 58.4 21.2 64.4 14.0 80.4 21.6 83.0 15.7 71.0 10.8 69.2 12.7 54.9 30.4 48.0 27.9 46.7 27.9 45.4 31.7 26.7 26.3 9.7 10.2 11.1 11.4 39.7 20.9 13.4 10.4 12.8 17.0 17.5 20.4 27.4 19.5 17.5 14.6 27.9 28.6 29.2 24.6 24.6 22.4 10.2 17.6 22.3 21.7 21.9 12.2 7.6 60.1 62.7 64.5 71.2 79.7 82.4 69.1 73.6 55.4 54.3 49.8 53.0 33.1 30.1 29.9 10.2 11.3 9.0 10.3 40.4 29.0 13.1 16.0 14.8 12.2 18.0 26.4 30.3 21.5 19.6 18.3 35.1 32.7 34.0 26.9 24.2 23.6 15.1 15.8 20.6 22.9 21.0 11.6 8.2 62.7 60.3 70.9 79.4 81.1 87.1 70.8 75.8 57.0 53.0 48.9 51.8 31.2 26.5 29.5 10.6 8.0 9.0 11.9 37.3 20.7 9.7 12.8 14.5 13.2 17.3 22.4 28.3 17.1 16.2 12.7 32.0 29.2 29.3 43.0 37.0 35.1 15.9 22.7 19.7 25.0 22.7 14.8 9.8 36.6 42.7 35.2 32.8 75.6 82.3 65.6 66.3 52.7 41.5 43.5 44.3 24.6 31.9 29.5 21.9 28.4 31.5 32.1 6.2 18.5 17.3 24.1 23.6 21.5 24.5 49.2 45.7 41.2 44.2 38.8 21.5 15.1 17.7 42.5 34.1 34.5 13.5 22.6 25.4 23.0 19.0 13.1 10.3 27.5 34.5 26.8 32.7 67.9 74.9 59.7 61.1 52.5 44.7 41.6 43.2 32.7 35.2 39.4 22.0 28.5 26.7 32.7 11.6 20.7 17.9 22.8 27.2 20.5 23.2 51.7 57.0 48.0 47.7 44.8 24.1 24.2 25.3 36.3 35.6 35.1 17.8 18.7 24.6 24.2 20.5 15.3 9.4 31.0 36.4 29.2 37.8 65.9 70.5 58.1 56.1 53.9 50.6 48.4 49.1 35.8 38.2 37.3 18.1 30.0 31.3 31.2 17.7 27.5 19.5 27.2 27.2 25.3 23.9 52.9 58.5 48.7 49.2 48.1 34.7 31.7 34.8 39.2 34.2 34.3 19.0 21.6 23.0 25.5 18.1 13.2 9.0 32.9 37.6 32.6 37.3 65.1 75.3 55.9 59.3 53.3 49.5 43.4 50.1 33.4 34.0 33.8 22.1 29.9 27.8 31.6 11.9 23.6 20.1 25.0 28.9 19.9 26.7 49.7 56.0 42.8 45.1 46.6 29.0 29.0 28.2 39.7 32.2 31.5 12.5 16.8 15.5 17.2 18.4 25.9 9.4 54.1 59.1 53.0 58.9 79.4 79.0 62.9 69.5 53.1 62.6 54.4 58.9 29.4 40.8 37.1 23.5 26.0 23.8 25.2 15.7 40.1 16.6 39.4 47.0 43.5 49.5 55.1 57.2 44.9 51.9 50.2 32.3 35.3 33.6 39.7 32.5 32.0 12.6 15.4 14.3 15.6 13.6 23.0 5.6 48.0 51.3 48.3 53.9 66.9 76.3 57.1 57.8 56.4 63.3 55.9 60.4 36.9 43.3 42.4 18.7 25.2 22.6 25.7 15.4 39.2 15.7 42.2 48.9 43.6 52.4 57.7 66.0 56.7 53.2 51.7 39.1 36.5 41.2 39.6 34.3 33.6 10.9 16.6 16.0 17.4 14.6 24.3 7.6 50.1 53.4 52.4 56.5 70.0 72.9 51.4 55.0 52.5 66.3 63.6 64.3 46.2 45.7 45.9 18.2 29.5 21.7 26.2 24.4 46.6 13.7 52.4 53.5 47.4 54.1 59.0 63.8 51.2 57.0 54.9 49.3 47.7 47.7 35.9 29.6 34.2 12.1 13.9 15.8 16.5 12.8 24.2 8.0 49.9 56.2 53.1 57.5 72.4 73.9 51.6 54.4 54.3 63.5 60.3 63.3 38.1 41.8 42.5 19.3 28.2 21.7 25.7 16.2 42.7 17.4 43.4 50.8 47.0 52.9 61.4 65.4 100.6 50.0 49.3 55.1 42.4 39.8 41.1 34.6 42.4 43.8 23.2 28.8 33.3 34.7 20.9 20.8 18.9 65.4 73.2 67.2 79.7 89.0 22.7 21.6 21.9 12.5 9.8 9.8 13.1 45.7 26.2 14.8 12.9 9.8 23.3 16.9 16.9 16.0 23.8 16.2 13.8 37.0 33.8 35.1 35.8 41.4 39.7 22.4 25.1 31.3 31.4 13.8 19.3 18.8 60.5 63.5 63.9 70.7 82.2 86.4 72.2 72.7 61.4 49.4 44.3 48.2 27.8 25.6 27.9 13.4 11.8 10.2 10.5 47.3 26.4 16.3 10.1 8.5 21.0 14.3 19.8 17.1 22.2 19.1 12.8 36.2 32.2 34.4 33.7 43.7 42.6 22.9 28.9 29.9 32.2 19.4 19.3 17.3 67.6 67.9 67.1 74.4 87.3 92.6 76.2 78.8 64.5 54.4 56.8 51.8 30.4 30.6 32.3 13.5 12.1 9.0 10.0 50.7 33.1 16.2 12.3 11.6 12.2 7.6 24.5 20.6 20.8 21.0 13.1 38.3 37.6 37.6 34.4 41.4 41.7 25.1 29.0 32.7 32.7 16.6 20.6 19.7 60.2 70.2 66.7 75.2 81.5 89.3 70.2 68.4 66.2 52.5 52.0 56.2 29.8 29.5 28.7 13.4 10.1 9.3 10.4 48.1 26.0 15.3 11.0 11.6 11.7 7.7 20.8 18.1 22.5 16.7 13.6 36.2 34.9 36.0 23.6 22.8 23.7 15.5 15.0 23.4 17.0 22.7 12.0 8.2 45.9 40.4 54.4 45.3 64.3 61.4 54.0 43.5 39.6 46.5 36.7 43.5 24.7 22.7 21.6 10.3 9.4 8.7 11.3 37.9 21.9 11.9 10.6 9.8 21.2 18.7 11.1 11.9 20.4 19.3 15.8 30.0 28.7 28.4 25.0 22.4 22.5 14.5 12.8 21.3 15.7 21.1 7.6 10.1 40.1 37.6 42.2 42.6 58.7 54.7 47.9 39.9 35.5 47.7 37.2 42.2 29.2 29.0 28.1 9.0 10.6 9.8 10.7 36.3 24.7 13.7 7.2 7.1 19.3 15.3 17.7 19.5 17.6 16.8 11.6 28.5 29.2 30.1 25.5 22.4 21.9 12.3 12.8 20.8 16.5 19.5 12.5 10.4 43.5 39.2 47.3 42.2 64.6 57.3 48.5 40.9 38.2 46.9 43.6 48.0 33.0 28.6 29.6 9.3 11.3 9.0 9.8 39.7 28.3 13.7 9.1 8.4 12.4 10.7 21.5 18.2 19.0 15.1 10.9 34.8 31.4 34.2 26.7 22.0 23.0 14.5 9.1 22.4 16.0 20.2 14.2 9.3 44.0 41.8 48.7 41.8 62.0 56.0 56.1 38.0 37.6 50.9 41.3 46.2 33.1 29.0 27.6 9.9 8.7 9.1 12.0 38.7 25.9 12.4 9.6 8.8 11.1 9.7 19.1 19.5 16.6 14.2 7.2 31.9 32.3 30.3 40.4 34.6 35.6 13.7 17.1 23.4 21.6 24.8 10.3 11.0 27.7 30.3 22.3 29.9 64.2 70.2 52.6 52.9 45.3 42.2 36.1 39.6 24.7 33.7 30.0 21.1 22.3 30.1 24.9 7.9 13.0 19.8 18.1 20.2 22.7 21.9 41.8 40.1 34.9 36.1 36.6 20.0 13.9 16.9 39.9 31.6 33.8 17.5 17.8 24.3 21.4 19.2 7.3 10.2 24.6 25.7 23.8 25.8 60.7 66.8 46.0 51.6 47.3 41.7 36.9 40.6 30.3 32.6 34.5 25.7 26.9 30.5 26.7 10.2 11.8 17.3 19.6 18.8 20.2 20.4 45.8 46.7 39.7 37.8 40.6 23.1 20.8 22.3 40.3 32.0 36.3 14.6 15.8 23.5 22.0 20.9 11.2 9.0 26.9 31.1 24.0 25.5 62.4 61.0 45.7 48.0 45.0 49.3 40.6 46.0 34.2 38.3 37.4 21.1 24.9 29.5 26.8 18.2 21.7 17.3 23.7 23.1 20.2 19.1 50.2 47.1 43.8 39.1 42.1 31.7 28.4 31.1 38.6 33.2 37.5 16.7 16.0 26.5 24.0 18.3 11.7 8.9 29.6 28.7 24.2 28.8 59.7 64.4 46.3 44.5 48.7 48.3 45.1 44.4 30.4 36.1 34.5 22.5 23.1 31.1 28.6 11.5 16.8 16.6 23.2 24.8 22.2 21.5 43.4 48.7 40.5 43.0 40.2 28.8 28.7 27.1 37.7 34.0 34.1 13.7 14.9 14.6 15.6 18.7 21.3 7.6 45.2 40.2 42.5 47.0 65.6 62.3 49.9 47.6 44.0 58.3 44.2 52.4 33.7 39.4 37.7 21.6 24.6 25.1 22.0 16.0 30.5 16.0 37.6 31.6 37.7 35.1 46.9 43.9 36.0 35.2 37.7 28.8 25.5 30.5 36.0 34.2 33.9 11.4 13.3 16.1 15.9 13.8 18.1 7.2 34.7 34.3 39.1 39.1 59.1 54.0 46.9 44.7 45.7 56.0 46.9 51.7 35.0 45.7 44.8 24.3 23.8 25.1 23.8 18.2 31.8 13.4 33.6 38.7 41.4 37.7 49.7 46.0 44.6 37.9 46.6 39.6 31.9 34.9 40.2 32.9 36.4 12.3 14.5 15.0 13.3 14.9 22.3 8.0 40.9 36.5 44.5 38.5 57.2 56.3 48.9 38.8 47.4 63.1 53.3 57.5 41.1 47.1 47.3 19.1 21.2 21.5 23.5 25.8 37.0 15.7 40.5 40.7 34.8 38.6 53.7 49.1 45.3 35.7 45.3 45.7 35.9 39.2 38.6 32.3 34.3 14.5 15.6 17.3 15.1 13.6 20.9 10.1 41.6 39.1 41.0 45.3 61.1 58.7 42.1 43.6 44.7 60.7 54.6 57.5 37.6 45.0 41.0 23.4 21.3 24.3 24.4 18.3 35.7 16.3 37.7 37.2 35.4 40.1 55.1 49.9 41.7 41.1 46.5 43.0 34.9 38.6 35.8 43.4 42.6 22.0 17.0 29.4 23.1 23.2 11.7 18.7 50.7 52.0 48.5 50.8 74.1 68.8 60.6 62.4 46.5 45.9 30.7 37.4 22.3 21.9 23.3 14.6 11.8 9.4 11.8 46.9 29.9 16.7 19.9 12.8 25.8 22.5 19.0 17.1 29.1 25.2 19.0 38.9 38.3 40.0 38.4 42.9 40.5 24.0 16.5 31.1 25.4 14.9 6.1 14.2 40.7 45.8 48.0 49.4 67.9 60.4 56.2 52.2 44.8 47.0 37.8 37.6 29.3 27.4 28.3 13.3 12.8 10.2 10.3 49.7 29.3 18.2 15.5 12.6 21.9 19.9 21.0 20.6 27.2 21.5 13.2 37.2 35.7 37.5 34.3 44.1 45.1 20.8 18.4 32.7 23.2 17.3 9.2 17.9 50.2 47.7 51.4 56.6 70.0 63.5 57.1 55.8 48.3 52.5 41.2 44.2 33.2 31.2 28.7 12.8 11.9 8.8 8.7 48.1 34.8 16.0 14.4 11.5 17.5 9.7 20.5 16.8 26.0 20.7 13.3 38.6 39.2 40.3 34.3 43.7 41.2 20.9 17.9 32.2 22.0 18.0 10.2 16.8 45.1 46.6 55.2 56.6 64.4 64.7 53.9 54.5 51.9 54.0 40.8 47.9 31.0 27.5 27.4 13.9 11.8 7.9 8.5 47.4 32.9 15.7 13.7 12.4 14.7 11.9 19.0 17.9 26.1 24.1 13.6 38.1 39.0 39.3 FITNESS SCORE 7.9 81.8 79.0 64.6 52.8 47.5 49.9 Fitness score matrix compares simulation result to reported trial results: Green = good fit, Red = bad fit Rows = populations , Columns = models Each matrix entry represents: 1000 individuals X 10 repetitions (X 10 time steps) In this run 34 populations x 64 models ~ 20M parallel computations ~ 2 weeks on 8 core machine OVERALL MODEL RANKING RESULTS Method_A1c Method_BMI Method_BP Method_Lipids Method_Smoke Method_MI Method_Stroke Method_DeathCHD Method_DeathStroke Method_TimeImprove Weighted Mean MODELS 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 0 0 0 0 0 4 1 1 1 1 1 1 1 1 1 1 4 1 1 1 0 0 0 0 0 4 2 1 1 1 1 1 1 1 1 1 3 1 1 1 0 0 0 0 0 1 2 1 1 1 0 0 0 0 0 1 4 1 1 1 0 0 0 0 0 4 4 1 1 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1 1 4 4 1 1 1 0 0 0 0 0 4 3 1 1 1 1 1 1 1 1 4 2 1 1 1 1 1 1 1 1 4 3 1 1 1 0 0 0 0 0 1 3 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 2 2 1 1 1 0 0 0 0 0 2 1 1 1 1 1 1 1 1 1 2 4 1 1 1 0 0 0 0 0 2 2 1 1 1 1 1 1 1 1 2 3 1 1 1 0 0 0 0 0 2 4 1 1 1 1 1 1 1 1 2 2 1 1 0 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 2 1 1 0 0 0 0 0 0 2 3 1 1 1 1 1 1 1 1 2 3 1 1 0 1 1 1 1 1 2 1 1 1 0 1 1 1 1 1 1 3 1 1 0 1 1 1 1 1 2 4 1 1 0 1 1 1 1 1 1 4 1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 3 2 1 1 0 1 1 1 1 1 3 2 1 1 1 0 0 0 0 0 2 2 1 1 0 1 1 1 1 1 3 4 1 1 1 0 0 0 0 0 3 1 1 1 1 0 0 0 0 0 2 3 1 1 0 0 0 0 0 0 2 4 1 1 0 1 1 1 1 1 3 3 1 1 0 1 1 1 1 1 3 3 1 1 1 1 1 1 1 1 3 4 1 1 0 1 1 1 1 1 3 1 1 1 0 0 0 0 0 0 2 1 1 1 0 0 0 0 0 0 3 2 1 1 1 1 1 1 1 1 4 2 1 1 0 0 0 0 0 0 3 4 1 1 1 0 0 0 0 0 1 2 1 1 0 1 1 1 1 1 4 4 1 1 0 1 1 1 1 1 4 1 1 1 0 0 0 0 0 0 3 2 1 1 0 1 1 1 1 1 4 3 1 1 0 0 0 0 0 0 3 4 1 1 0 0 0 0 0 0 1 3 1 1 0 0 0 0 0 0 3 3 1 1 0 0 0 0 0 0 3 3 1 1 1 0 0 0 0 0 1 4 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 3 1 1 1 0 0 0 0 0 0 4 2 1 1 0 0 0 0 0 0 4 4 1 1 0 0 0 0 0 0 4 3 1 1 0 0 0 0 0 0 4 1 1 1 0 RANK 18.51 18.92 19.78 20.04 20.51 20.72 20.73 20.92 21.48 21.66 22.5 22.63 22.88 23.2 23.57 23.77 25.67 27.65 27.91 30.56 31.15 31.41 32.09 33.88 33.89 34.14 34.43 35.32 35.69 35.7 36.1 36.12 36.34 37.7 37.9 38.1 38.96 39.15 39.45 39.65 39.92 40.18 40.52 40.76 41.61 42.15 42.85 43.02 43.71 44.55 44.57 44.99 45.38 45.46 46.3 46.87 47.07 47.24 47.62 47.7 51.37 53.41 55.47 55.58 Best Model Overall Jacob Barhak

Possible Future Directions Optimization Using a database for results Simulation language improvements Population generation improvements GUI extensions Jacob Barhak