Software Testing Overview and Common Errors

SVVRL @ IM.NTU Software Testing: An Overview Yih-Kuen Tsay Dept. of Information Management National Taiwan University Based on [Patton 2006, Kaner et al. 2002] SDM 2023

SVVRL @ IM.NTU Infamous Software Error Cases The Lion King Animated Storybook, 1994-1995 Didn t run on untested but common systems Intel Pentium Floating-Point Division Bug, 1994 NASA Mars Polar Lander, 1999 A control bit to shut off the thrusters, which is triggered by touched down , but also by mechanical vibration. The Y2K (Year 2000) Bug, circa 1974 Two digits used to represent the year Facebook s user data-leak, 2018 A security vulnerability exploited by the hacker Yih-Kuen Tsay SDM 2023: Software Testing 2 / 28

SVVRL @ IM.NTU What Is a Software Bug/Error? The software doesn t do something that the product specification says it should do. The software does something that the product specification says it shouldn t do. The software doesn t do something that the product specification doesn t mention but should. The software does something that the product specification doesn t mention. The software is difficulty to understand, hard to use, slow, or in the software tester s eyes will be viewed by the end user as just plain not right. Yih-Kuen Tsay SDM 2023: Software Testing 3 / 28

SVVRL @ IM.NTU Real Causes of Software Bugs Four categories: Specification Design Code Others (false positives, multiple ones from the same root cause, testing errors, ) How frequent? specification > design > code How costly to fix? code > design > specification Yih-Kuen Tsay SDM 2023: Software Testing 4 / 28

SVVRL @ IM.NTU What Does a Software Tester Do? The goal of a software tester is to find bugs. The goal of a software tester is to find bugs and find them as early as possible The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed. Yih-Kuen Tsay SDM 2023: Software Testing 5 / 28

SVVRL @ IM.NTU Traits of Good Software Testers They are explorers. They are troubleshooter. They are relentless. They are creative. They are (mellowed) perfectionists. They exercise good judgement. They are tactful and diplomatic. Know how to work with the programmers They are persuasive. Get the severe-enough bugs fixed Yih-Kuen Tsay SDM 2023: Software Testing 6 / 28

SVVRL @ IM.NTU Realities of Software Testing Some axioms of software testing: It s impossible to test a program completely. Software testing is a risk-based exercise. Testing can t show that bugs don t exist. The pesticide paradox: the more you test software, the more immune it becomes to your tests. Not all the bugs you find will be fixed. Software testers aren t the most popular members of a project team. Some terms and definitions: Precision and accuracy, verification and validation Yih-Kuen Tsay SDM 2023: Software Testing 7 / 28

SVVRL @ IM.NTU Lessons Learned: The Role of the Tester Beware of testing completely. (L. 10) Be clear about what tests are done and what are not. You don t assure quality by testing. (L. 11) The assurance results from the effort of the entire team. Never be the gatekeeper! (L. 12) It is the responsibility of the people who control the project. Beware of the not-my-job theory of testing. (L. 13) Take a more expansive view (be helpful). Yih-Kuen Tsay SDM 2023: Software Testing 8 / 28

SVVRL @ IM.NTU Testing Fundamentals Black-box vs. white-box testing In black-box testing, the tester only knows what software is supposed to do he can t look in the box to see how it operates. In white-box (or clear-box) testing, the tester has access to the program s code. Static vs. dynamic testing (analysis) Static testing refers to testing something that s not running examining and reviewing it. Dynamic testing is simply to run and use the software (to see if it works as expected). Yih-Kuen Tsay SDM 2023: Software Testing 9 / 28

SVVRL @ IM.NTU Testing/Examining the Specification It is a static black-box testing (why?) (what if your project doesn t have a spec?) High-level techniques: Pretend to be the customer Research existing standards and guidelines Review and test similar software Low-level techniques: Specification attributes checklist Specification terminology checklist Yih-Kuen Tsay SDM 2023: Software Testing 10 / 28

SVVRL @ IM.NTU Good Well-Thought-Out Specifications Complete Accurate Precise, unambiguous, and clear Consistent Relevant Feasible Code-free Not constraining the underlying design, architecture, or code. Testable Yih-Kuen Tsay SDM 2023: Software Testing 11 / 28

SVVRL @ IM.NTU Problem Words in a Specification (1/2) Always, every, all, none, never Look for exceptions Certainly, therefore, clearly, obviously, evidently Be aware of accepting something as a given Some, sometime, often, usually, ordinarily, customarily, most, mostly Too vague to test Etc., and so forth, and so on, such as Incomplete listing, not testable Yih-Kuen Tsay SDM 2023: Software Testing 12 / 28

SVVRL @ IM.NTU Problem Words in a Specification (2/2) Good, fast, cheap, efficient, small, stable Unquantifiable terms and not testable Handled, processed, rejected, skipped, eliminated Hidden functionalities If then (but missing else) Ask yourself what will happen if the if doesn t happen Yih-Kuen Tsay SDM 2023: Software Testing 13 / 28

SVVRL @ IM.NTU Test Cases and Testing Techniques To do black-box testing effectively requires some definition of what the software does. Test cases are the specific inputs that you will try and the procedures that you will follow. A mechanism for determining whether a test has passed or failed is usually called a test oracle. Selecting test cases is the single most important task that software testers do. Testing techniques teach you how to select good test cases. Yih-Kuen Tsay SDM 2023: Software Testing 14 / 28

SVVRL @ IM.NTU Testing Levels Unit testing To verify the functionality of a specific section of code Integration testing To verify the interfaces between components against a software design System testing Testing a completely integrated system to verify that the system meets its requirements Acceptance testing (user acceptance testing) To determine if the requirements of a specification/contract are met Yih-Kuen Tsay SDM 2023: Software Testing 15 / 28

SVVRL @ IM.NTU Two Fundamental Approaches Test-to-pass (smoke/sanity testing) Assure only that the software minimally works. You don t push its capabilities. Test-to-fail (destructive testing) Design and run test cases with the sole purpose of breaking the software. Try things that should force the bugs out. Yih-Kuen Tsay SDM 2023: Software Testing 16 / 28

SVVRL @ IM.NTU Testing Techniques Equivalence Partitioning To systematically get a smaller, but equally effective, set of test cases Examples: Addition of small numbers vs. large numbers Filenames with valid characters vs. invalid characters Short vs. long filenames Divide test work along the two parts of software The data The program Yih-Kuen Tsay SDM 2023: Software Testing 17 / 28

SVVRL @ IM.NTU Data Testing Boundary conditions The first element of an array Sub-boundary (or internal boundary) conditions Neighbors of boundary values, e.g., in powers of two or the ASCII table Default, Empty, Blank, Null, Zero, and None Invalid, Wrong, Incorrect, and Garbage Data Yih-Kuen Tsay SDM 2023: Software Testing 18 / 28

SVVRL @ IM.NTU Program State Testing Testing the software s logic flow Create a state transition map Reduce the number of states and transitions to test Testing the states to fail Race condition (bad timing) Repetition testing (same operation again and again) Stress testing (under less-than-ideal conditions) Load testing (with largest possible data sets) Yih-Kuen Tsay SDM 2023: Software Testing 19 / 28

SVVRL @ IM.NTU Code Coverage A technique for white-box testing A percentage measure of the degree to which the source code of a program is executed when a particular test suite is run. Criteria: Function coverage Statement coverage Branch coverage: each control branch Condition coverage: both values of each Boolean sub- expression Yih-Kuen Tsay SDM 2023: Software Testing 20 / 28

SVVRL @ IM.NTU Lessons Learned: Testing Techniques People-based techniques focus on who does the testing. (L.49) alpha testing, beta testing, eat your own dogfood, Coverage-based techniques focus on what gets tested. (L. 50) Problems-based techniques focus on why you re testing (the risks you re testing for). (L. 51) Activity-based techniques focus on how you test. (L. 52) regression testing, exploratory testing, Evaluation-based techniques focus on how to tell whether the test passed or failed. comparison with saved results, oracle-based testing, Yih-Kuen Tsay SDM 2023: Software Testing 21 / 28

SVVRL @ IM.NTU Applying Your Testing Techniques Configuration Testing Compatibility Testing Foreign-Language (Internationalization) Testing Usability Testing Testing the Documentation Testing for Software Security (discussed earlier) Website Testing (discussed earlier) Yih-Kuen Tsay SDM 2023: Software Testing 22 / 28

SVVRL @ IM.NTU Configuration Testing Configuration testing is the process of checking the operation of the software with various types of hardware. System boards + BIOS, input/output cards, peripherals, device drivers, etc. Be prepared that standards aren t always followed. Apply equivalence partitioning to reduce the usually very large set of possible configurations. Yih-Kuen Tsay SDM 2023: Software Testing 23 / 28

SVVRL @ IM.NTU Compatibility Testing Software compatibility testing means checking that your software interacts with and shares information correctly with other software. Issues: Platform and application versions Backward compatibility and forward compatibility Compatibility standards and guidelines Data Sharing File save and file load File export and file import Cut, copy, and paste Yih-Kuen Tsay SDM 2023: Software Testing 24 / 28

SVVRL @ IM.NTU Usability Testing Usability is how appropriate, functional, and effective the interaction between the software and its end user is. What makes a good user interface (UI)? Follows standards and guidelines, Intuitive, Consistent, Flexible, Comfortable, Correct, Useful Usability testing includes accessibility testing (or testing for the disabled). Yih-Kuen Tsay SDM 2023: Software Testing 25 / 28

SVVRL @ IM.NTU Testing the Document Assuring that the documentation is correct is the tester s job, too. Packaging text and graphics Marketing material, ads, and other inserts Warranty/registration EULA (End User License Agreement) Labels and stickers Installation and setup instructions User s manual, etc. Good software documentation improves usability and reliability and lowers support costs. Yih-Kuen Tsay SDM 2023: Software Testing 26 / 28

SVVRL @ IM.NTU Benefits of Automation and Tools Speed Efficiency More time for test planning and designing new tests Accuracy and precision Resource reduction Simulation and emulation Relentlessness Tools never tire or give up Tools for Web app.: Chrome DevTools, Selenium Yih-Kuen Tsay SDM 2023: Software Testing 27 / 28

SVVRL @ IM.NTU Test Case Specification Test case specification documents the actual values used for the input along with the anticipated outputs. That is, the details of a test case should explain exactly what values or conditions will be sent to the software and what result is expected. Other important information: Test item Environmental needs Intercase dependencies Etc. Yih-Kuen Tsay SDM 2023: Software Testing 28 / 28