Principles of Software Development - Testing and Telemetry Insights

Testing Kevin Scannell, David Ferry CSCI 5030 Principles of Software Development Saint Louis University St. Louis, MO 63103 1

Testing Case Study: Tweets Business Problem: Twitter needs to parse and validate tweets Business Problem: Twitter wants to make sure other people can parse, validate, and submit tweets correctly Solution: Publish a set of libraries in multiple languages that perform tweet parsing and validation Development Problem: How to ensure uniform behavior across many different code bases? See https://github.com/twitter/twitter-text Look in conformance, then validate.yml, autolink.yml, etc. CSCI 5030 Principles of Software Development 2

Testing Case Study: Tweets One shared (plain text) test set, each language implements the test set in its own way These tests define how Twitter works, what a hashtag is, what is linkable, etc. Most of this functionality is not explicitly documented anywhere else, and even if it were, the code determines what actually happens CSCI 5030 Principles of Software Development 3

Test Coverage Full test coverage is usually impossible Why? Alternatives: Equivalence partitioning inputs belong to a small number of equivalence classes E.g. 100 char tweet vs. 101 char tweet E.g. triangle( 2,2,2 ) vs. triangle ( 3,3,3 ) If one member of class works, every member should work (hopefully) Identify boundary conditions, if program accepts 1- 100, at least test 0, 1, 100, 101 What many of us do automatically still hard CSCI 5030 Principles of Software Development 4

Test Coverage Contd Random testing: automate testing on many inputs, can never test everything, but can build confidence Pure randomness usually not an approximation of a realistic input space E.g. Consider randomly generating triangles E.g. Consider randomly generating tweets by picking 140 characters at random Fuzz testing (fuzzing)- start with a set of valid inputs and generate permutations Can you build a model of what real inputs look like? Data mine system logs, user opt-in to data collection Can you collect actual instances of data? E.g. Twitter can store everything sent as a tweet, even if some of those are rejected. E.g. bug report tools CSCI 5030 Principles of Software Development 5

Feedback & Telemetry Many programs ask you to participate in sending feedback, automatic feedback is telemetry What makes users happy, sad, frustrated? What do users spend their time doing? What functions are commonly used? What code is most frequently executed? E.g.: https://youtu.be/fkjY_R7zQsM?t=835 (Start at 13:55, go to 18:10) Pareto principle: theory is only 20% of features are used Focus testing effort based on risk-reward CSCI 5030 Principles of Software Development 6

Modelling User Input Personal experience: finite-element modelling of structures Software simulates buildings subject to an earthquake Outputs have real physical meaning- acceleration, velocity, displacement of building elements Random structures? Not helpful CSCI 5030 Principles of Software Development 7

Modelling User Input Personal experience: finite-element modelling of structures Software simulates buildings subject to an earthquake Outputs have real physical meaning- acceleration, velocity, displacement of building elements Random structures? Not helpful Solution: Recursively subdivide existing structure to scale the computational load while maintaining most of simulated dynamics CSCI 5030 Principles of Software Development 8

Numerical Performance Measures Not all tests are pass/fail Even if they are pass/fail, a large corpus of test sets can result in a percentage E.g. Twitter hashtag detection in foreign languages E.g. machine learning and classification Regression testing then comes down to making sure that we re not doing any worse than we used to Make sure you have a valid baseline- a safe self driving car only needs to be as safe as a human driver would be, not perfect Again, machine learning classification is prime example CSCI 5030 Principles of Software Development 9

Code Hygiene Clean code helps eliminate errors: Testable and self-testing code asserts its own correctness Test all assumptions and preconditions, as well as output postconditions Observable systems are transparent Should be obvious what is happening in system when something goes wrong Log system state and interactions Data and input validation should not be ignored Make interfaces should be simple CSCI 5030 Principles of Software Development 10

Common Types of Tests Volume tests feed program very big files, long strings, etc. Program 2 Program 1 4096 byte capacity Load tests heavily stress a system e.g. make lots of connections, make lots of queries CSCI 5030 Principles of Software Development 11

Common Types of Tests Security tests subject system to known vulnerabilities before attackers do SQL injection, buffer overflows, etc. Performance tests timing benchmarks and microbenchmarks Portability tests different OSes, OS versions, libraries, library versions, Intel vs. AMD vs. etc, different browsers CSCI 5030 Principles of Software Development 12

Open Problem: Concurrency and Races #define iters 5000 int global = 0; Thread 1: for( i=0; i<iters, i++){ global++; } Thread 2: for( i=0; i<iters; i++){ global--; } Race condition on two cores No race condition on one core Race condition on one core and iters = 1,000,000 Difficult / impossible to rule out that bug that only happens once in a blue moon test often and long! CSCI 5030 Principles of Software Development 13