Hierarchy-Aware Regression Test Prioritization Techniques
Explore hierarchy-aware regression test prioritization techniques to improve testing efficiency and fault detection. Learn about prioritization strategies, common challenges, and solutions for optimizing test suites. Discover the impact of test hierarchies on test duration and the importance of integrating previous work for ranking test scores.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Hierarchy-Aware Regression Test Prioritization Hao Wang1, Pu (Luke) Yi2, Jeremias Parladorio3 Wing Lam4, Darko Marinov5, Tao Xie6 1. University of California at Berkeley 2. Stanford University 3. National University of Rio Cuarto 4. George Mason University 5. University of Illinois Urbana-Champaign 6. Peking University CCF-1763788 CCF-1956374 CCF-2338287
Regression test prioritization 101 Original test suite Original test suite (executed) T1 T1 T2 T2 T3 T3 T4 T4 Prioritization Prioritized test suite Prioritized test suite (executed) Repo T3 T3 T2 T2 T4 T4 T1 T1 Find faults sooner! 2
Common prioritization Original test suite (executed) T1 T2 T3 T4 Globally apply prioritization technique Problematic Original test suite (executed) Repo T3 T2 T4 T1 3
Test hierarchies Costly! Module 1 Test Module switch Module 2 T1 T2 T3 T4 T3 T3 T4 T4 T1 T1 T2 T2 Execution T3 T3 T2 T2 T1 T1 T4 T4 CoCoCostly! In reality, 5.8x switching! RUN TIME INCREASES! 4
Test hierarchies (cont.) Start T3 T3 T2 T4 T2 T1 T4 T1 Switching can also greatly affect test duration! (Caching, true randomness creation, etc. ) RUN TIME INCREASES! In reality, 1.5x runtime! 5
Why is the problem not reported before - Expensive to run the test orders! - Difficulties in compiling old repos - Machine cost of running > 10k test orders - Simulation rather than execution - Reusing history time and test outcome 6
Solution: Hierarchy-aware - Hierarchy-aware: run each test module as a whole! Previous T3 T2 T4 T1 Prioritize within modules Module 1 Module 2 Module 2 Module 1 Ours T1 T2 T3 T2 T1 T4 T3 T2 T4 T1 Prioritize the modules Minimizes run time increase 7
Integrating previous work Rank Test Score 1 T3 100 Use existing technique within modules 2 T2 80 Module 1 Module 2 Module 2 Module 1 3 T4 20 T1 T2 T3 T2 T1 T4 T3 T2 T4 T1 4 T1 10 Use score summarization for the modules Rank Module Avg 1 M2 60 2 M1 45 sum, avg, max, min, 8
Evaluation metric The most practically important: average time to fault (ATTF) ATTF = 77.5s 100 Problem: Hard to calculate average for multiple test suites. Found % of Faults Found 50 O T1 T1 T2 T2 T3 T3 T4 T4 20 60 90 11 0 120 Time (s) 9
Evaluation metric (cont.) cost-cognizant Average Percentage of Faults Detected (APFDc) 100 % of Faults Found 50 APFDc = 0.354 O T1 T1 T2 T2 T3 T3 T4 T4 20 60 90 11 0 120 Time (s) 10
Problem of APFDc APFDc fails when run time increases! ATTF = 77.5s ATTF = 97.5s 100 50 APFDc = 0.458 APFDc = 0.354 O T1 T1 T2 T2 T3 T3 T4 T4 T3 T2 T4 T1 20 160 90 20 60 90 11 0 120 50 120 130 180 Time (s) 11
Hierarchy-aware APFDc (HAPFDc) ATTF = 77.5s ATTF = 97.5s 100 50 APFDc = 0.354 HAPFDc = 0.569 HAPFDc = APFDc = 0.458 T 4 O T1 T1 T2 T2 T3 T3 Extended T3 T2 T4 T1 T4 20 60 90 11 0 120 180 20 13 0 50 90 12 0 160 180 Time (s) 12
Realistic evaluation For the benchmark: - Systematic filtering from one of the largest RTP dataset (Java) - All orders are run and perfed without flakiness - Use Tukey s test to ensure statistical significance We select seven techniques from Peng et al. [1] as the baselines. [1] Q. Peng, A. Shi, and L. Zhang, Empirically revisiting and enhancing IR-based test-case prioritization, in ISSTA, 2020. 13
Evaluation result: hierarchy-awareness - HAPFDc score significantly surpasses previous techniques - Statistically significantly better - Achieves .99x-1x total run time, whereas previous technique is 1.5x 1.5x Previous techniques Ratio of total run time Comparing to default Default Hierarchy-aware techniques 1x O HAPFDc score 14
Evaluation result: misleadingness - Metric: APFDc vs. HAPFDc - APFDc has large inconsistencies comparing to HAPFDc results - HAPFDc matches ordering by ATTF but APFDc does not - Simulation - Matches results reported in previous work, hierarchy-unaware being the best - Not consistent with ranking based on real execution 15
Discussions - Why run each test modules as a whole? - Simple, reliable and improves on previous techniques by a lot - Execution is so hard to do - Experiment shows that execution is the right way to go - Can also use small-scale execution to verify validity - Hierarchy is not limited to modules - Test modules, sub-projects, directories 16
Takeaway - - - - For the exact same test suite, run time increases for different orders! Use hierarchy-aware RTP in real life is better than using previous techniques! HAPFDc more truthfully reflects the performance of test orders Run actual orders for future RTP study (even only for small-set validation) 17
