Framework for Writing Trigger-Action Todo Comments in Executable Format
A framework developed by Pengyu Nie, Rishabh Rai, Junyi Jessy Li, Sarfraz Khurshid, Raymond J. Mooney, and Milos Gligoric which allows developers to write trigger-action todo comments in an executable format. The framework aims to address the issues of tedious and error-prone manual checking and execution of trigger-action comments. It provides motivation from Google Guava and Apache Struts, highlighting the need for automation in handling trigger-action comments. TrigIt, the framework introduced, enables developers to specify triggers and actions in a DSL embedded in the host language (Java) and automates the execution process by checking triggers before execution.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
A Framework for Writing Trigger-Action Todo Comments in Executable Format Pengyu Nie, Rishabh Rai, Junyi Jessy Li, Sarfraz Khurshid, Raymond J. Mooney, and Milos Gligoric August 29, 2019 ESEC/FSE 2019 Tallinn, Estonia supported by
Motivation (1) From Google Guava protected AbstractStreamingHasher(int chunkSize, int bufferSize) { // TODO(kevinb): check more preconditions // (bufferSize >= chunkSize) if this is ever public checkArgument(bufferSize % chunkSize == 0); ...} The method is public checkArgument(bufferSize >= chunkSize); Trigger Action The comment is written in natural language, thus a developer needs to locate the comment check if the trigger evaluates to true execute the action Tedious and error-prone 2
2012/11: The comment was added 2015/5: The trigger was satisfied 2015/10: The action was executed 2017/10: The comment was removed Motivation (2) From Apache Struts void testDynamicAttributesSupport() throws Exception { ... // TODO : remove expectedJDK15 and if() after switching to Java 1.6 String expectedJDK15 = "<input type=\"text\" ..."; String expectedJDK16 = "<input type=\"text\" ..."; if (result.contains("foo=\"bar\" ...")) assertEquals(expectedJDK15, result); else assertEquals(expectedJDK16, result); ...} switch to Java 1.6 remove a variable and if statement Trigger Action 3
Motivation (3) Trigger-action comments are frequently used We have found hundreds of trigger-action comments in 10 open-source projects Trigger-action comments are 10+% of developer comments Like any other comments written in natural language, they frequently become outdated or forgotten We contacted 10 developers from large software companies and asked if they write trigger-action comments; 7 (out of 8 replies) answered yes They confirmed they have a lot of trigger-action comments, and lack of automation is a big issue 4
TrigIt A framework for writing trigger-action comments as specifications in executable format Developers write triggers and actions in the TrigIt DSL Embedded in the host language (Java) Utilizes native syntax (if-statements) with different semantics TrigIt checks the triggers after compilation and before execution If a trigger satisfies, TrigIt executes the associated actions TrigIt removes all its specifications before execution 5
Example TrigIt Specification (1) From Google Guava protected AbstractStreamingHasher(int chunkSize, int bufferSize) { // TODO(kevinb): check more preconditions // (bufferSize >= chunkSize) if this is ever public if (trigItIsPublic()) checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod boolean trigItIsPublic() { return TrigIt.getMethod("<init>", int.class, int.class).isPublic(); } 6
Example TrigIt Specification (2) From Apache Struts void testDynamicAttributesSupport() throws Exception { ... // TODO : remove expectedJDK15 and if() after switching to Java 1.6 if (TrigIt.getJavaVersion().ge(TrigIt.JAVA6)) { String expectedJDK16 = "<input type=\"text\" ..."; assertEquals(expectedJDK16, result); } else { String expectedJDK15 = "<input type=\"text\" ..."; String expectedJDK16 = "<input type=\"text\" ..."; if (result.contains("foo=\"bar\" ...")) assertEquals(expectedJDK15, result); else assertEquals(expectedJDK16, result); } ...} 7
Design Choices Preserves semantics of comments from the execution point Each TrigIt specification is local to its context similar to inline comments TrigIt specifications are removed from the compiled code (after TrigIt's processing), thus they do not cause side-effects during execution Java compiler checks partial correctness of TrigIt specifications Integrated as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true 8
TrigIt DSL Query Expression / Trigger Action Statement / Global Action Local Action TrigIt Method 9
TrigIt DSL (1/4) Query Expression / Trigger if ( <query expression> ) { // action } Examples AST: TrigIt.getClasses().findAny( C ).isPresent() Build: TrigIt.getJavaVersion().ge(TrigIt.JAVA8) Issue: TrigIt.isClosed( github issues/1897 ) Time: TirgIt.after(2019, 4, 2) 10
TrigIt DSL (2/4) Action Statement / Global Action @TrigItMethod void globalAction() { if (/* trigger */) { <action statement> } } Examples TrigIt.getMethod(m1()).remove() TrigIt.getField(f).setPrivate() 11
TrigIt DSL (3/4) Local Action if (/* trigger */) { /* code executed if trigger evaluates to true */ } else { /* code executed if trigger evaluates to false */ } 12
TrigIt DSL (4/4) TrigIt Method @TrigItMethod annotation void return type: specifies trigger + global action(s) @TrigItMethod void trigItDeprecateMakeChecked() { if (TrigIt.after(2018, 4, 30)) TrigIt.getMethodByName("immediateCheckedFuture").remove(); } boolean return type: specifies a trigger that can be reused @TrigtItMethod boolean trigItIsPublic() { return TrigIt.getField(f).isPublic(); } if (trigItIsPublic()) { /* action 1 */ } if (trigItIsPublic()) { /* action 2 */ } 13
TrigIt Workflow Test/ Execution .java Files Compiler .class Files 14
TrigIt Workflow TrigIt Test/ Execution .java Files Compiler .class Files 15
TrigIt Workflow TrigIt rewrite TrigIt Test/ Execution .java Files Compiler .class Files specifications Rewrite removes all other code and only keeps TrigIt specifications 16
TrigIt Workflow TrigIt evaluate rewrite TrigIt Test/ Execution .java Files Compiler .class Files specifications AST Build Configurations Issue Tracker System Time Rewrite removes all other code and only keeps TrigIt specifications Evaluate checks all triggers 17
TrigIt Workflow TrigIt evaluate rewrite TrigIt Test/ Execution .java Files Compiler .class Files specifications apply AST Build Configurations Issue Tracker System Time Rewrite removes all other code and only keeps TrigIt specifications Evaluate checks all triggers Apply executes actions by changing the bytecode, sends warning messages, or changing source code removes all TrigIt specifications Warning Messages Patch to .java Files 18
Evaluation RQ1: Can developers quickly implement correct triggers and actions using TrigIt? RQ2: What s the build overhead of TrigIt? RQ3: What s the overhead for writing TrigIt specifications over trigger-action comments, measured by number of tokens? 19
User Study Design Can developers quickly implement correct triggers and actions using TrigIt? Tasks Read a brief tutorial of TrigIt syntax (10min) Migrate 3 trigger-action comments to TrigIt specifications The first one is warm-up and not counted in results (without telling the participants) 2 groups of tasks: Warm-up, Task A from Google Guava, Task B from Apache Struts Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins Details available on our webpage: cozy.ece.utexas.edu/trigit 20
User Study Participants 2 groups of tasks: Warm-up, Task A from Google Guava, Task B from Apache Struts Warm-up, Task C from Apache Ignite, Task D from Jenkinsci Jenkins 20 participants 8 professionals (6 developers in large software companies, 2 researchers) 12 students (4 undergrads, 8 PhD students) Task A+B Task C+D Professionals 6 2 Students 6 6 21
User Study Results Time (min) Confidence Correctness Grouped By Avg. Median Avg. Median Trigger Action Syntax A 4.7 4.5 4.2 4.5 12/12 12/12 9/12 B 4.3 5.0 4.4 4.5 12/12 12/12 12/12 Task C 7.6 7.5 3.9 4.0 8/8 7/8 8/8 D 7.2 6.5 4.2 4.0 7/8 8/8 8/8 22
User Study Results Time (min) Confidence Correctness Grouped By Avg. Median Avg. Median Trigger Action Syntax A 4.7 4.5 4.2 4.5 12/12 12/12 9/12 B 4.3 5.0 4.4 4.5 12/12 12/12 12/12 Task C 7.6 7.5 3.9 4.0 8/8 7/8 8/8 D 7.2 6.5 4.2 4.0 7/8 8/8 8/8 Professionals 5.0 5.0 4.5 5.0 16/16 15/16 15/16 Role Students 6.1 5.5 4.0 4.0 23/24 24/24 22/24 All 5.7 5.0 4.2 4.0 39/40 39/40 37/40 Participants obtained correct solutions with very little training 23
Build Overhead RQ2: What s the build overhead of TrigIt? Subjects: We migrated 44 trigger-action comments from 10 projects to TrigIt specifications. Environment: Intel i7-6700U@3.40GHz, 16GB RAM, Ubuntu 18.04 LTS Results: average 8.85%, median 6.63%, min 1.91%, max 26.75% 24
Complexity of the TrigIt DSL RQ3: What s the overhead for writing TrigIt specifications over trigger-action comments, measured by number of tokens? Results: 58.74% Trigger-action comment: average 12.2, median 12.0, min 6, max 26 TrigIt specifications: average 18.0, median 16.5, min 11, max 36 Trigger: average 7.3, median 6.0, min 4, max 19 Action: average 7.4, median 5.0, min 2, max 27 Boilerplate code: average 3.3, median 3.0, min 3, max 7 Boilerplate code Trigger Action 25
Anecdotal Experience During our inspection of trigger-action comments in open source projects, we observed some comments are forgotten, i.e., the trigger is satisfied but the action is not executed We reported 6 cases and the developers responded quickly Some triggers and actions in comments are not specified enough: Unfortunately Java 8 still causes some issues with some Google internal infrastructure. Java 8 is allowed in tests but not in the main part of the code (yet). Knowing the details of the projects, e.g., internal testing reports, will help writing correct TrigIt specifications 26
Conclusion A framework for writing trigger-action comments in executable format Introduces DSL for triggers and actions Preserves semantics and developers workflow TrigIt is easy to learn and introduces small overhead TrigIt can run as a bot that sends a notification to developers or a code review with changes when a trigger evaluates to true Webpage: cozy.ece.utexas.edu/trigit Pengyu Nie <pynie@utexas.edu> 27
User Study Participants Prior Experience Programming experience: average 8.4 years Java skills (on a scale of 1-5): 3.7 None of the participants has contributed to the open-source projects where we selected the tasks from 29
User Study Error Analyze Trigger error: 1 & action error: 1 Misunderstanding the trigger-action comments Syntax errors: 3 Putting the local action in a wrong place We added additional checks in TrigIt to prevent this kind of errors protected AbstractStreamingHasher(int chunkSize, int bufferSize) { // TODO(kevinb): check more preconditions // (bufferSize >= chunkSize) if this is ever public checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod boolean trigItIsPublic() { if (TrigIt.getMethod("<init>", int.class, int.class).isPublic()) checkArgument(bufferSize >= chunkSize); } 30
Example TrigIt Specifications (3) From Elastic Elasticsearch public void testShadowReplicaNaturalRelocation() throws Exception { // TODO: uncomment the test below when https://github.com/elastic/elasticsearch/issues/17695 // is resolved. if (TrigIt.isClosed("https://github.com/elastic/elasticsearch/issues/17695")) assertIndicesDirsDeleted(nodes); } 31
Example TrigIt Specifications (4) From Google Guava // TODO(b/72241893): Remove by 2018-04 @Deprecated public static <V, X extends Exception> CheckedFuture<V, X> immediateCheckedFuture( @NullableDecl V value) { return new ImmediateSuccessfulCheckedFuture<>(value); } @TrigItMethod void trigItDeprecateMakeChecked() { if (TrigIt.after(2018, 4, 30)) TrigIt.getCurrentClass().getMethodByName("immediateCheckedFuture").remove(); } 32
Study of Trigger-Action Comments We inspected 471 trigger-action comments and classify the trigger/action s specificity high: migration feasible medium: migration could potentially be done low: hard to understand, migration is not feasible 28% have both specific trigger and specific action 70% have either specific trigger or specific action 33
TrigIt Implementation Implemented as a standalone Java library, can be easily integrated into existing build process Command line interface Maven plugin Implemented with Java bytecode manipulation, using ASM Extra checks for the correctness of triggers and actions TrigIt.getClass("C").getField("f").isPrivate() TrigIt will check if class C and field f exist Debug options: --assume-true, --no-action 34
Using Bytecode Analysis & Manipulation The bytecode format was more stable in recent years than the Java language syntax; this may lower the cost of maintenance. We wanted to avoid dependencies on IDEs and enable integration of TrigIt with any build process, even if the build is running on a continuous integration service. Future versions of TrigIt may have a tighter connection with IDEs based on the adoption and feedback that we received from the community. 35
Example Patch (Before Trigger Satisfies) From Google Guava protected AbstractStreamingHasher(int chunkSize, int bufferSize) { if (trigItIsPublic()) checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} @TrigtItMethod boolean trigItIsPublic() { return TrigIt.getMethod("<init>", int.class, int.class).isPublic(); } 36
Example Patch (The Patch) From Google Guava protected AbstractStreamingHasher(int chunkSize, int bufferSize) { - if (trigItIsPublic()) - checkArgument(bufferSize >= chunkSize); + checkArgument(bufferSize >= chunkSize); checkArgument(bufferSize % chunkSize == 0); ...} -@TrigtItMethod -boolean trigItIsPublic() { - return TrigIt.getMethod("<init>", int.class, int.class).isPublic(); -} 37
Related Work Pengyu Nie, Junyi Jessy Li, Sarfraz Khurshid, Raymond Mooney, and Milos Gligoric. 2018. Natural language processing and program analysis for supporting todo comments as software evolves. In Workshops of the the AAAI Conference on Artificial Intelligence. 775 778. Margaret-Anne Storey, Jody Ryall, R. Ian Bull, Del Myers, and Janice Singer. 2008. TODO or to bug. In International Conference on Software Engineering. 251 260. Giriprasad Sridhara. 2016. Automatically detecting the up-to-date status of ToDo comments in Java programs. In India Software Engineering Conference. 16 25. Dorsaf Haouari, Houari Sahraoui, and Philippe Langlais. 2011. How good is your comment? A study of comments in Java programs. In International Symposium on Empirical Software Engineering and Measurement. 137 146. Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*iComment: bugs or bad comments?*/. In Symposium on Operating Systems Principles. 145 158. Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc comments to detect comment-code inconsistencies. In International Conference on Software Testing, Verification, and Validation. 260 269. 38
