Optimal Spacing Approach for Sampling Small Datasets
Software effort estimation is critical for project success. This study introduces an optimal spacing approach for sampling small datasets in software effort estimation. The Eubanks Optimal Spacing Theorem is utilized to estimate parameters for censored order statistics. A classification system is proposed to define small-sized datasets using a threshold of project instances. Empirical analysis and dataset descriptions are provided for better understanding and application of the approach.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
An Optimal Spacing Approach for Sampling Small-sized Datasets for Software Effort Estimation Samuel Abedu, Solomon Mensah, Frederick Boafo, Eva Bushel and Elizabeth Akuafum
Introduction Software effort estimation (SEE) deals with predicting the effort in relation to project cost and resource allocation to enable timely production and delivery of software projects within budget [1]. When project effort is overestimated or underestimated, it can result in devastating consequences for the given company.
Introduction Estimate effort for new project Instances Historical dataset Train model X
Can it work best for relatively small-sized dataset(s)? How can we define small-sized data from a given dataset?
Eubanks Optimal Spacing Theorem The Eubank s optimal spacing approach makes use of a quantile function over a given interval, [p,q] [0,1] to estimate the location and scale parameters for a censored set of order statistics.
Classification of Dataset Class 1 is regarded as the first-class (Q1), class 2 as the second-class (Q2 and Q3), and class 3 as the third-class (Q4). A threshold of 43 project instances for a small-sized dataset.
Small-Sized Datasets 6Datasets Albrecht, Atkinson, Cosmic, Kemerer, Finnish, Telecom1
Empirical Analysis Dataset Data Pre-processing Model Selection Experimental Setup
Empirical Analysis Dataset Data Pre-processing Model Selection Experimental Setup
Datasets Description Instances Features Dataset Albrecht 24 7 IBM DP Services Project Builds to a large telecommunications product at U.K. company X N/A Data collected by the TIEKE organization from IS projects from nine different Finnish companies Large business application Enhancement to a U.K. telecommunication product Cross-organisational projects compiled by the ISBSG Data collected from various software firms Atkinson Cosmic 16 42 12 10 Finnish Kemerer Telecom1 ISBSG (R10) China 38 15 18 4106 499 6 5 2 105 17
Empirical Analysis Dataset Data Pre-processing Model Selection Experimental Setup
Data Pre-processing Resolve issues of missing data values, outliers, and influential data
Empirical Analysis Dataset Data Pre-processing Model Selection Experimental Setup
Model Selection Selected 7 Models for the empirical analysis The Transformed Linear Model (ATLM), Bayesian Network (BN), ElasticNet (ENR) regression, Support Vector Machine (SVM), Artificial Neural Network (ANN), Deep Neural Network (DNN) and Long-Short Term Memory (LSTM). models are the Automatically
Empirical Analysis Dataset Data Pre-processing Model Selection Experimental Setup
Mean Absolute Error ? 1 ? ?=1 ??? = ??? ???
Cliffs Delta Effect Size ? =????? ??> ?? ????? ??< ?? ??
Results The results show that the deep learning models (DNN and LSTM) recorded the best prediction accuracy for five datasets. The ANN, which is a shallow neural network recorded the best prediction accuracy for two datasets and the baseline ATLM also recorded the best prediction accuracy for one dataset.
Dataset Metric ATLM LSTM DNN ANN SVM ENR BN 0.352 0.014 0.024 0.239 0.248 0.186 0.006 MAE Albrecht 0.004 0.017 0.010 0.031 0.010 0.010 0.004 Cliff's 0.052 0.002 0.004 0.038 0.034 0.031 0.002 MAE Atkinson 0.000 0.010 0.000 0.010 0.010 0.010 0.010 Cliff's 0.549 0.020 0.013 0.516 0.475 0.467 0.008 MAE Cosmic Results 0.001 0.005 0.001 0.002 0.002 0.001 0.007 Cliff's 0.012 0.009 0.013 0.001 0.002 0.001 0.001 MAE Finnish 0.002 0.002 0.007 0.002 0.002 0.009 0.002 Cliff's 0.206 0.024 0.014 0.196 0.173 0.161 0.009 MAE Kemerer 0.000 0.031 0.031 0.020 0.020 0.020 0.000 Cliff's 0.206 0.009 0.003 0.216 0.199 0.128 0.003 MAE Telecom 0.004 0.004 0.013 0.022 0.004 0.013 0.031 Cliff's 0.014 0.005 0.004 0.014 0.014 0.350 0.004 MAE ISBSG 0.001 0.000 0.000 0.000 0.001 0.001 0.006 Cliff's 0.006 0.005 0.007 0.007 0.007 0.174 0.005 MAE China 0.000 0.000 0.002 0.000 0.000 0.000 0.000 Cliff's
Conclusion A threshold of 43 project instances for a small-sized dataset. Deep learning models should be adopted for software effort estimation. However, it recommends techniques like dropout and early stopping to reduce overfitting in the deep learning models.
Future Work The empirical study will be extended to evaluate the computational cost of running the deep learning and conventional machine learning models in the software engineering field.
References [1] [2] [3] [4] [5] [6] [7] [8] [9] S. Mensah and P. K. Kudjo, A classification scheme to improve conclusion instability using Bellwether moving windows, Journal of Software: Evolution and Process, vol. 34, no. 9, Sep. 2022, doi: 10.1002/smr.2488. R. L. Eubank, A Density-Quantile Function Approach to Optimal Spacing Selection, The Annals of Statistics, vol. 9, no. 3, pp. 494 500, 1981, doi: 10.1214/AOS/1176345454. Y. Mahmood, N. Kama, A. Azmi, A. S. Khan, and M. Ali, Software effort estimation accuracy prediction of machine learning techniques: A systematic performance evaluation, Softw Pract Exp, 2021, doi: 10.1002/SPE.3009. L. Song, L. L. Minku, and X. Yao, A novel automated approach for software effort estimation based on data augmentation, in ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Oct. 2018, pp. 468 479. doi: 10.1145/3236024.3236052. X. Y. Jing, F. Qi, F. Wu, and B. Xu, Missing data imputation based on low-rank recovery and semi-supervised regression for software effort estimation, in Proceedings - International Conference on Software Engineering, May 2016, vol. 14-22-May-2016, pp. 607 618. doi: 10.1145/2884781.2884827. F. H. Yun, China: Effort Estimation Dataset, Apr. 2010, doi: 10.5281/ZENODO.268446. S. Mensah, J. Keung, S. G. MacDonell, M. F. Bosu, and K. E. Bennin, Investigating the Significance of the Bellwether Effect to Improve Software Effort Prediction: Further Empirical Study, IEEE Trans Reliab, vol. 67, no. 3, pp. 1176 1198, 2018, doi: 10.1109/TR.2018.2839718. S. Hochreiter and J. Schmidhuber, Long Short-Term Memory, Neural Comput, vol. 9, no. 8, pp. 1735 1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735. P. A. Whigham, C. A. Owen, and S. G. MacDonell, A baseline model for software effort estimation, ACM Transactions on Software Engineering and Methodology, vol. 24, no. 3, 2015, doi: 10.1145/2738037.
