Exploring Bug Fix Choices and Developer Decision-Making

The Design Space of Bug Fixes and How Developers Navigate It IEEE Transactions on Software Engineering, Vol. 41, No. 1, January 2015 Emerson Murphy-Hill, Thomas Zimmermann, Christian Bird, and Nachiappan Nagappan Kadie Clancy CS2310: Seminar Fall 2017

Bug Fix Choices Bug Fix Choices The fix they choose has many implications What is the risk of introducing other bugs during the fix? Is the bug fix in the same code that caused the bug? Is the change fixing the cause or just covering a symptom?

Bug Fix Choices Bug Fix Choices Traditional Approach: assume that the location in the software at which an engineer fixes a bug is the location at which the error was made Quickly assume the actual error was equated to the correction made Consequences of this misunderstanding: Researchers studying bug prediction and bug localization models of how developers have fixed bugs in the past may only capture workarounds Software Engineers evaluated based on how many bugs they fix, may not accurately reflect that engineer s effect on software quality Without teaching the contextual factors that go into deciding which fix to apply, future engineers may choose inappropriate fixes

Motivations Motivations What evidence is there that the same bug can be fixed in multiple ways? Browser based learning environment called Pex4Fun where programmers try to solve programming challenges Given some code, a method with parameters and a default return value Asked to modify the program until it produces the same output as a secret solution Browsing the 260 submitted solutions to the problem, found that no two appeared exactly alike Substantial design differences between many solutions

Bug Fix Choices Bug Fix Choices Define the design of bug fixes as the human process of envisioning several ways to fix the same bug Judging which fixes to apply Engineer must deal with a number of competing forces when choosing what change to make Not always straightforward What are the different ways that bugs can be fixed? What factors influence which fix an engineer chooses?

Design Space of Bug Fixes Design Space of Bug Fixes Research Methods Opportunistic Interviews Firehouse Interviews Triage Meeting Observations Surveying

Research Methods Research Methods Opportunistic Interviews, Firehouse Interviews Goal: rapidly obtain qualitative answers to research questions in a way that was minimally intrusive to interviewees Outcomes: Identified symptoms of the bug, the fix that was applied, alternative fixes, and the reasons for discriminating between fixes based on the interviews Most reported multiple possible fixes for the bug that they discussed Interviewer was able to suggest an alternative fix In these cases, the engineer agreed that the fix was possible, but never consciously considered the alternative fix, due to external project constraints

Research Methods Research Methods Triage Meetings Goal: obtain qualitative answers to our research questions with respect to how engineers work together to find good bug fix designs Outcomes: little data was gathered Discussion of a bug fix was limited to whether or not it should be fixed Team was so tightly knit that when bug fixes were discussed, very little expansion was needed

Research Methods Research Methods Surveys & Replicated Surveys Goal: to quantify observations made during interviews Outcomes: Identified symptoms, the fix that was applied, alternative fixes, and the reasons for discriminating between fixes based on the interviews Descriptive statistics to quantify ideas from interviews

Design Space Design Space - - Dimensions Dimensions describes how much internal code is changed versus external code is changed as part of a fix describes how much error information is revealed to users, describes the degree to which code is restructured in the process of fixing a bug, while preserving its hardcodes data. On one end of the dimension, data is specified explicitly, and on the other, data is describes how far information is allowed to propagate across a piece of software whether that information is for end users or other engineers describes whether a fix is perceptible to the user of a feature is removed during a bug fix behavior. fixed with a simple one-line change entail significant code restructuring generated dynamically captures to what degree a fix describes how much captures the degree to which a fix utilizes accurate information

Dimension Dimension Possible Values Possible Values Data Propagation Across Components: how far information is allowed to propagate across a piece of software Data is corrected at the source Data corrected at furthest level from source Example: a bug where the user interface was reporting disk spaces too large; found that the bug could be traced back to the lowest level that was reporting values in KB when the UI was expecting values in MB

Dimension Dimension Possible Values Possible Values Error Surfacing: how much error information is revealed to users User is made aware of detailed error information Existence of error is not revealed to user Example: a bug where software crashed when a user deleted a file; the fix caught the exception then delete the file; had to consider whether or not to notify the user whether this situation occurred

Dimension Dimension Possible Values Possible Values Behavioral Alternatives: whether or not a fix is perceptual to a user Fix doesn t require the user to do anything different The user must significantly modify behavior Example: bug in a mobile app where the back button wasn t working; fix back button but had to completely remove another feature because it was impossible for the two to work simultaneously

Dimension Dimension Possible Values Possible Values Functionality Removal: how much of a feature is removed during a bug fix No code is removed at all Whole software product is eliminated Example: bug in which a crash occurred; the engineer considered removing the feature that the bug was in altogether

Dimension Dimension Possible Values Possible Values Refactoring: degree to which code is restructured in the process of fixing a bug, while preserving its behavior Fixed with one line Significant code restructuring Example: bug fix involved removing some copy and paste duplication within the code; not only fixes the bug but improves the quality of the code

Dimension Dimension Possible Values Possible Values Internal vs External: how much internal code is changed vs how much external code is changed Bug is fixed changing internal code Bug is fixed changing external code Example: a bug in which web application occasionally did not work correctly when the user entered info with an onscreen keyboard; one way to fix this would be to change the way a web browser worked with web applications and onscreen keyboards

Dimension Dimension Possible Values Possible Values Accuracy: degree to which a fix utilizes accurate information Uses heuristics and guesses Highly accurate information Example: fixing a race condition bug between two threads; accurate fix would be to introduce some explicit synchronization construct between the two threads. Heuristic approach would be to simply have one thread wait a few seconds until the other thread has probably completed

Dimension Dimension Possible Values Possible Values Hardcoding: degree to which a fix hardcodes data Data specified specifically Data generated dynamically Example: bug related to incorrect address information in a database. One hardcoded solution to the problem was to modify the records in the database. The dynamic solution was to modify the original code that produced that data in the database.

Design Space Design Space - - Example Example

Navigating Design Space Navigating Design Space - - Dimensions Dimensions variety including mandates from supervisors, ability to find chose how to design a bug fix is by considering the development phase of the project. existing practices to what degree a fix will be consistent with the existing software or ownership describes the effect of how users of knowledgeable people, and code degree to which a fix breaks existing interfaces the software behave on the fix how thoroughly an engineer understands why a particular bug occurs

Limitations Limitations An important limitation is that of generalizability beyond the population studied (external validity) Microsoft Strengthened by replicated survey, where results based on Microsoft developers were largely transferrable to developers at large Struggled with remembering the design decisions they made Generally unable to articulate implicit decisions Memory bias: inherent in most retrospective research methods Attempted to control memory bias by asking opportunistic interviewees to recall their most recently fixed bugs

Implications Implications Additional factors in bug prediction and localization Research has investigated several approaches to predicting future bugs based on previous bugs Results of these papers suggest factors that can cause a bug to be fixed in one way at one point and a completely different way at another time Future research may incorporate these factors into their models Limits of Bug Prediction and Localization Some factors are practically outside of the reach of what bug predictors can consider Like whether or not an engineer made a fix with full knowledge of why the bug occurred Refactoring while Fixing Bugs Results suggest that engineers frequently see code that should be refactored, yet still avoid refactoring Usage Analytics Engineers often made decisions about how to fix bugs without a data driven understanding of how real users use their software Envision a future where engineers, while deciding the design of a bug fix, can quickly query existing usage data with an easy-to-use tool Fix Reconsideration Ad-hoc methods like TODOs in code are difficult to keep track of Need a better mechanism to reconsider fixes in the future

Relationship to Patterns Relationship to Patterns Problem: the bug itself Context: dimensions of how we navigate the design space Solution: the fix itself (one fix out of many), defined by the dimensions of the design space

Conclusion Conclusion Authors have described a study that combined opportunistic interviews, firehouse interviews, meeting observation, and a survey Delivered a multi-dimensional design space for bug fixes, a space that engineers navigate by The authors have taken the critical first step towards understanding a practice that engineers have always engaged in Enable researchers, practitioners, and educators to better understand and improve bug fixes

References References E. Murphy-Hill, T. Zimmermann, C. Bird, and N. Nagappan, The design of bug fixes, in Proc. Int. Conf. Softw. Eng., 2013, pp. 332 341. http://ieeexplore.ieee.org/document/6606579/ E. Murphy-Hill, T. Zimmermann, C. Bird, and N. Nagappan. The design space of bug fixes and how developers navigate it. IEEE Transactions on Software Engineering, 41(1):65 81, Jan 2015. http://ieeexplore.ieee.org/document/6901259/