Combinatorial Test Design Model in Software Development

Building Combinatorial Test Input Model from Use Case Artefacts Preeti Satish, Milind B, Medhini S Narayan, Krishnan Rangarajan Department of Computer Science and Engineering Dayananda Sagar College of Engineering Bangalore, India preetisatish8@gmail.com, milindbasavaraja96@gmail.com, medhininarayans@gmail.com , krishnanr1234@gmail.com

Introduction to Combinatorial Testing (CT) State-of-art in CT practice Our Approach to Combinatorial Test Design Model (CTDM) Sample Case Study- Withdrawal Deriving CTDM from Use Case Specifications (UCS) Deriving CTDM from Use Case Diagrams (UCD) Combined results from UCS and UCD. Conclusion & Future Work

CT is a test design methodology that aims to reveal interaction faults CT is a test design methodology that aims to reveal interaction faults Well Researched area. Good open source tools like ACTS available. Focus of this paper is to automate this step. Not much research done and published in this area. Steps in CT Step 1 Combinatorial Test Design Model Derivation Step 2 Test case Generation Step 3 Test Case Prioritization Step 4 Test Execution Step 5 Regression and Fault Analysis IWCT-2017, Japan, March 13, 2017

The software to be developed shall work on two types of operating systems such as Windows and Linux. The software shall also be compatible with Mozilla Firefox and Internet Explorer browsers Requirements Document Analysis artifacts like Use Case Diagrams Domain knowledge & expertise Implemented CTD Tool Test designer Manually Automation Operating Systems Browser Parameters Values Windows Mozilla Firefox CTDM Linux Internet Explorer IWCT-2017, Japan, March 13, 2017

Software Requirement Specification Test Designer Inputs Use Case Diagram + Use Case Specification Output CTDM elements User Interface (Combined Results) Rule Based Processing UML Activity Diagram Paper accepted at IWCT 2013 Combinatorial Test Design Modelling Approach UML Sequence Diagram Paper accepted at IWCT 2014 IWCT-2017, Japan, March 13, 2017

Normal flow 1) A withdrawal transaction asks the customer to choose an account type to withdraw from savings-account or checking-account. 2) Choose an amount from a menu of possible amounts. 3) The system verifies that it has sufficient money on hand to satisfy the request 4) The transaction is sent to the bank for approval 5) Appropriate amount of cash is dispensed and recorded in ATM log. Alternative flow 2a) Customer can cancel the transaction 3a) Customer is informed to enter a different amount. 3b) Customer cannot withdraw amount>10000 from his saving-account. IWCT-2017, Japan, March 13, 2017

Input: UCS specifies the requirements in a predefined structure namely the precondition, normal-flow statements (NFS), alternative-flow statements (AFS) and post-conditions. The UCS Parser: The parser is implemented in Python using NLTK and ALCHEMY API. The rules for parameter and value identification and extraction are embedded in it. Text analysis is done using WATSON. Output: CTDM elements- parameters and values IWCT-2017, Japan, March 13, 2017

Rule 1: PARAM_KW Rule 2: PARAM_OKW Rule 3: PARAM_SYMBOL IWCT-2017, Japan, March 13, 2017

The PARAM_KW rule is based on the keyword extraction feature of Alchemy API. Keyword extraction feature extracts important topics in your content that are typically used when indexing data. A python script is coded to import Alchemy API and execute the keyword extraction function. ALGORITHM: Input: Normal Flow statements (NFS) from UCS Step 1: Begin Step 2: For each NFS, Initialize variables data = Input, myText= data Step 2.1: KW =alchemyapi.keywords (myText) Step 2.2: para [] = keywords obtained (KW) Step 3: End Output: Set of Parameters IWCT-2017, Japan, March 13, 2017

Manually derived reference CTDM Appropriate balance Parameters Account type Amount Sufficient amount Checking account Savings account Amount> 10000 Amount> balance Amount<= balance Value 1 Cancel Valid (sufficient money) Invalid (different amount) Value 2 Value 3 Amount<=10000 Parameter Extracted from PARAM_KW PARAM_KW withdrawal transaction, account-type, customer, savings account, checking account, possible amount, menu, sufficient money, hand, request, approval, transaction, bank, ATM log, cash, savings Red:- True Positive Black :- False Positive IWCT-2017, Japan, March 13, 2017

The PARAM_OKW rule is based on the research by Krishnan et al that nouns in the object section of the sentence are more likely to be parameters. Input Statement Output Parameters A withdrawal transaction asks the customer to choose an account type to withdraw from savings-account or checking-account, and to choose an amount from a menu of possible amounts. account-type, menu, possible, amounts The system verifies that it has sufficient money on hand to satisfy the request before sending the transaction to the bank. request, bank Red:- True Positive Black :- False Positive IWCT-2017, Japan, March 13, 2017

The PARAM_SYMBOL ruleis regular expression parser that scans statements for symbols like > , < , = and nouns on the Left Hand Side (LHS) of the symbols are extracted as parameters. Input Statement Output Parameters The customer cannot withdraw amount >10000 from his savings account in a single transaction. Amount Red:- True Positive Black :- False Positive IWCT-2017, Japan, March 13, 2017

Rule 1: VAL_CONJ Rule 2: VAL_SYMBOL Rule 3: VAL_JJ Rule 4: VAL_COND Rule 5: VAL_ACTOR IWCT-2017, Japan, March 13, 2017

The values are extracted from both the NFS and AFS of the UCS. 1. The VAL_CONJ: Sentences containing conjunctions and and or are expected to hold values. The rules scan for these conjunctions and the Left Hand Side (LHS) and Right Hand Side (RHS) words of these conjunctions considered as values. 2. The VAL_SYMBOL: Comparative and assignment statements containing symbols like < or less than , > or greater than , = or equal to hold values. The symbol along with its succeeding word together is considered as value. 3. The VAL_JJ: Adjectives in the sentence are fetched as values. 4. The VAL_COND: Conditional statements like the IF statements are expected to hold values. Grammar notation used for parsing is IF , NN , VB . The verb is extracted as value. Also, when one value of IF is found, then the other value is found by negating it 5. The VAL_ACTOR: Actors are roles played in the system. System is an actor that provides the service and user is an actor that accepts the service. The task performed by the actor becomes the value. The processing is carried out by scanning and extracting for a verb after the actor. IWCT-2017, Japan, March 13, 2017

Values Extracted Rules savings-account, checking-account, dispensed, recorded VAL_CONJ choose, verified, ~verified, cancel VAL_ACTOR possible, sufficient amount, different amount, VAL_JJ >10000 VAL_SYMBOL Red:- True Positive Black :- False Positive IWCT-2017, Japan, March 13, 2017

We tried our approach on various case studies like flight reservation, facebook, enrollment and car ordering. Since, complete case studies from SRS to all UML diagrams were difficult to find, students familiar with Object Oriented Modelling Concepts (OOMD) were given the requirements to come up with needed analysis artefacts. The original documents are marked in and the derived documents are marked in . SRS UCS UCD UAD USD Withdrawal Facebook Flight Reservation Enrolment Car Ordering

In comparison with the manual model, TP are the correctly reported parameters and values, FP are the reported parameters and values that are not correct and MSD are unidentified parameters and values. Recall % (R%) = (TP/(TP+MSD))*100 Precision % (P%) = (TP/(TP+FP))*100 PARAMETERSAND VALUESOBTAINEDFROM UML UCS R% P% Case Study Evaluation focus TP FP MSD parameter Values parameter Values Parameter Values Parameter Values Parameter Values 6 11 4 6 7 6 2 5 7 2 13 0 16 6 6 2 10 10 9 6 0 2 0 3 0 5 1 2 0 8 100 85 100 67 100 55 67 71 100 20 39 100 20 50 53 75 17 33 44 25 Car Withdrawal Flight Facebook Enrollment

UML UCD depicts how the user interacts with the system. UCD represents SUT from functionality point of view, and the function is encompassed within a use case in a UCD. The main elements of UCD are actors, use cases and interactions between them. ArgoUml an open source UML modeling tool is used to draw the use case diagrams. IWCT-2017, Japan, March 13, 2017

The UML UCD Parser is implemented in python programming language to parse the XMI file, embedded with rules necessary to identify the CTDM elements. The rules are for: 1)Rule for parameter identification and extraction. 2)Rule for value identification and extraction. IWCT-2017, Japan, March 13, 2017

PARAM_UC rule states that use case associated with the actor holds the parameter information. VAL_EXTEND:. The extended use-case in extend association holds the value information. Extending Use case Extended Use case <extends> Parameter Value Actor UCD Skeleton depicting the parameters and values IWCT-2017, Japan, March 13, 2017

Red:- Values Blue:- Parameters IWCT-2017, Japan, March 13, 2017

Parameters Extracted Account-type, amount, sufficient-money, appropriate amount Values Extracted Amount>10000 IWCT-2017, Japan, March 13, 2017

In comparison with the manual model, TP are the correctly reported parameters and values, FP are the reported parameters and values that are not correct and MSD are unidentified parameters and values. Recall% (R%) = (TP/(TP+MSD))*100 Precision% (P%) = (TP/(TP+FP))*100 PARAMETERSAND VALUESOBTAINEDFROM UML UCD Evaluation focus parameter 5 Values 0 parameter 4 Values 1 Parameter 7 Values 3 Parameter 3 Values 1 Parameter 6 Values 0 Case Study R% P% TP FP MSD 0 0 0 0 1 0 3 0 0 1 1 83 0 100 11 100 27 100 14 86 0 100 0 100 100 87 100 50 100 100 0 Car 13 0 8 0 8 0 6 1 10 Withdrawal Flight Facebook Enrollment

The results obtained from individual input sources UCS and UCD are combined to arrive at a single set of CT parameters and values. The combined list of parameters and values are then compared with the manual model to note the TP, FP and MSD and compute the recall percentage and precision percentage metrics. The combined effect brings the best from individual input sources. With respect to parameters, as seen from the column R% of table we could achieve 100% recall factor for parameters, from all case studies, which suggest that automation is successful. With respect to values, the combined results show a slight improvement. As seen from column R% of table, except for Enrollment case study, the recall is above 60%. R% P% Case Study Evaluation focus TP FP MSD 6 11 4 6 7 7 3 6 7 2 13 0 16 6 7 2 10 10 9 7 0 2 0 3 0 4 0 1 0 8 100 85 100 67 100 63 100 86 100 20 32 100 20 50 50 78 23 38 44 22 parameter Values parameter Values Parameter Values Parameter Values Parameter Values Car Withdrawal Flight Facebook Enrollment

An approach for deriving CT parameters and values from UCS and UCD has been designed, implemented and evaluated. The results show promise in providing automated support to the test designer in modelling the CT input space. The parameters can be extracted from use case specification and diagrams and we could achieve 100% recall from the combined results. The recall of values was found to be low as not all the values of interest tend to appear in UCD and UCS. This points to the need for looking into other analysis artefacts like Activity Diagram, Sequence diagram and the requirements document itself. IWCT-2017, Japan, March 13, 2017

The rules are not limited to what has been identified in this paper and more such rules can be formulated making it more complete. In our work under progress we have planned a set of initial thoughts on linking the parameters and values and on constraints extraction. Our next step is to build a CT modelling tool, which can process UML activity diagrams, sequence diagrams, use case diagrams and specifications and combine the results to build a more precise combinatorial model. Our vision is to build a combinatorial test design work bench that will be of immense value to the test designer. IWCT-2017, Japan, March 13, 2017

Any Questions??

Thank You!!