A Defect is Being Born: How Close Are We? A Time Sensitive Forecasting Approach - Online Appendix

A Defect Is Being Born: How Close Are We? A Time-Sensitive Forecasting Approach Online Appendix Mikel Robredo∗, Matteo Esposito∗, Fabio Palomba†, Rafael Pe˜ naloza‡, Valentina Lenarduzzi∗§, ∗University of Oulu — †University of Salerno — ‡University of Milano-Bicocca — §University of Southern Denmark {mikel.robredo, matteo.esposito}@oulu.fi, [email protected], [email protected], [email protected] I. MODELS A. Time Series Analysis (TSA) forecasting models We will adopt four distinct time series analysis (TSA) models to forecast future effectiveness, ARIMA and SARIMA, as well as their multivariate counterparts, ARIMAX and SARIMAX. We define them as follows: ARIMA: As version control data is inherently nonstationary, i.e. denotes a trend across time, we will use all the four models which are state-space variants of the Auto-Regressive Integrated Moving Average (ARIMA) model, which determines the best autoregression model parameters for the effectiveness to be predicted, and also identified the best moving average parameter while determining the level of differencing required for the time series of the effectiveness probability to achieve stationarity. ARIMAX: Previously used in the literature [1], [2], it is the multivariate variant of the ARIMA model, in which a set of independent variables included in the model training. This model offers a further multivariate approach to the ARIMA model by introducing additional independent variables into the model and, therefore, helping to explain the evolution of the probability on the defectiveness of projects. SARIMA: The SARIMA model is a similar approach to treat, and therefore forecast, time series data as in the ARIMA model. However, the ARIMA model considers a theoretical trend and seasonality values within the model. However, SARIMA model allows for randomness in the seasonal pattern of the time series data, which is continuously updated from one forecasting step to the next. SARIMAX: he Seasonally adjusted Auto-Regressive Integrated Moving Average model stands as the seasonally adjusted extension of the ARIMAX model. It addresses the seasonality of the data within model training through model parameter tuning [3]. More information on the adjustment of the model parameters in the TSA model will be provided in full version of the manuscript. Similarly, the SARIMAX model provides a multivariate approach compared to its univariate counterpart, SARIMA, by including independent variables in the model. B. Bayesian forecasting models There is empirical evidence that for some case scenarios, Bayesian statistics can perform better inference than classical frequentist statistics [4], [5]. Bayesian methods allow for model building with less data, and rely on the Markovian theory of continuous model parameter update while training models [5]. Said this, Bayesian statistics can be implemented as a different model-building approach given a prediction task [6]. Consequently, we select four classically adopted Bayesian time series forecasting models. Bayesian Damped Local Trend (BDLT): This bayesian structural time-series method models trend and seasonal behaviour while controlling for persistent trend growth. For that, it applies a damping factor that gradually reduces the influence of the trend component over time, reflecting the realistic assumption that long-term upward or downward movements eventually slow [7]. Bayesian Exponential Smoothing (BETS): One of the formal models on bayesian time series forecasting, Bayesian Exponential Smoothing models time series data based on three central components; the error, trend and seasonolity patterns. Each new observation updates these components using exponentially decaying weights, which assign greater importance to more recent data [8]. Bayesian Dynamic Linear Model (BDLM): This Bayesian state-space model combines an observation component, which relates the observed data to a latent state vector, and a state-evolution equation, which controls how the latent state changes over time. The Bayesian Dynamic Linear Model supports time-varying regression within the assumption on the data following a Gaussian distribution, which allows for the relationship between independent variables and the forecasting target variable to vary dynamically [9]. Bayesian Dynamic Generalized Linear Model: This model extends the DLM model to address cases in which the time series data follow distributions outside the Gaussian distribution. While it preserves the state-space structure of DLM models, it introduces a link function to fit non-Gaussian data into the model training [10]. C. Transformers-based forecasting models Recently, the use of Large Language Models (LLMs) as well as transformer-based deep learning models has grown notorily within the field of software engineering [11], [12]. Given the rapid research rhythm currently existing on the discovering of efficient and explainable transformer-based models, we will rely on the currently state-of-the-art transformer-based models [13]: TIMEGPT: We adopt the first foundational transformerbased time-series forecasting model [13], [14]. This model can forecast a wide range of time-series data and even perform anomaly detection. However, we do not discard the option of looking for an alternative model since this model is not opensource. LAG-LLAMA: This open-source model allows for only univariate forecasting. With a decoder-only architecture, it provides a range of possible future values through probabilities outcomes, instead of deterministic answers [15]. CHRONOS: Built on top of the T5 family of transformerbased models, it introduced the tokenization step through the one it transforms a real-valued time series to a set of tokens of a finite vocabulary, allowing T5 models to use their language model architecture and this start the training procedure [16], [17]. MOIRAI: Better called masked encoder-based universal time-series forecasting, it is a probabilistic forecasting model, meaning that it outputs a future distribution of values. For that, it uses patching to tokenize the input time-series data, and also uses a mixture of distributions of the final predicted distribution [18]. TimesFM: Developed by Google Research [19], this is the only deterministic model that we will consider for the study so far. As a deterministic model, TimesFM gives a single forecast value for a future time step. It is important to note that, while our choice stands on stateof-the-art models, we do not discard allowing our research to include newer models if we can demonstrated the outdated condition of the listed models [20], [21]. II. PERFORMANCE EVALUATION A. Time Series performance evaluation metrics We use three performance metrics to compare and evaluate the forecasting performance of the defined models. The first performance metric is the Mean Absolute Percentage Error (MAPE). MAPE is a statistical metric commonly used to measure prediction precision [22]. Quantifies the magnitude of errors between the predicted values and the actual observed values by calculating the mean of the absolute value of the prediction error. Its formula is the following. MAP E =100 n n X i=1 |(Yi−ˆ Yi)2| Yi (1) where nis the number of observations, Yiis the actual observed value, and ˆ Yiis the i-th predicted value. As shown by the formula, MAPE is represented as a percentage metric, where a smaller percentage of error means better predictive performance, and the opposite is true with a higher percentage. MAPE has limitations when the actual observations are small or close to zero, which introduces bias into the model training. Therefore, we selected alternative accuracy measurement statistics to cover this issue. One of them, and the second metric to present is the Mean Absolute Error (MAE), which is characterized by measuring the average magnitude from the absolute value of prediction errors [23]. Its equation is MAE =Pn i=1 |Yi−ˆ Yi| n(2) where the variables follow the same notation as the ones addressed for MAPE MAE expresses the prediction performance error in absolute value; therefore, a smaller result signifies better predictive performance, while higher results represent poorer performance. We adopt as the third performance metric root mean squared error (RMSE) which captures the error value in the same value unit as the variable being predicted [24]. Its formula is RM SE =v u u t 1 n n X i=1 (Yi−ˆ Yi)2(3) where the variables follow the same notation as the ones explained previously. Although the definitions have value unit differences, a low error value indicates a high predictive performance in RMSE, and similarly, a high error value depicts a poor predictive performance. B. Expected time horizons To evaluate the performance of the defined models, we will train and test the models based on the time series built during the data serialization (explained in the submitted registered report). As observed by Robredo et al. [1], version control data is stochastic in nature, meaning that software development processes do not follow clear time frequencies such as those seen in weather or financial forecasting [22]. Nevertheless, based on the serialization levels we find from the existing version control data, we adopt the Walk-Forward Train-Test validation technique [25] for the performance evaluation of all models considered in this study. We choose this technique because it respects the temporal nature of the data and does not rely on randomness in the order of the observations. The Walk-Forward Train-Test technique trains and tests a model greedily and iteratively. Each iteration trains a model, predicts the value of the next data point of the time series, and compares the prediction against the real value. The real value is consequently added to the train set, and the model is trained again with the newly extended train set data. This cycle continues until all test data points are predicted. Within this setting we plan to investigate the impact of the selected models on the shortand long-term periods. Previous research has demonstrated that time-series prediction models strongly depend on sufficiently enough time series data to be trained, as well as to provide reliable longterm predictions [1]. Similarly, we plan to operationalize our findings following a two-fold strategy. First, we will adopt professionals’ preferences on forecasting weekly,bi-weekly and monthly forecasting window lengths [1], which should be applicable to the outcomes from all the TSA, Bayesian and transformer-based models. Second, we plan to perform multistep forecasts in order to assess the capabilities of the adopted models into longer forecasting horizons. Since this procedure depends on time-series data availability, we will investigate the performance of the resulting best models when performing multi-step forecasting as far as our collected data allows us to do, to enable practitioners observe the long-term impact of their current project maintenance level within the context of defect prediction. C. Posterior statistical evaluation In order to assess the significance on the impact of using any of the models within the scope of defect forecasting, we will define the following hypotheses: H01 There is no statistically significant difference in forecasting effectiveness between the adopted time-series forecasting models. H11 There is a statistically significant difference in forecasting effectiveness between the adopted time-series forecasting models. Furthermore, in case of a statistically significant difference identified, we are interested in investigating which is the best model that performs better than the others, and based on which of the error metric results. Hence, we perform a post hoc analysis testing the following hypothesis: H02 There is no statistically significant pairwise difference in prediction effectiveness between any of the adopted time-series forecasting models. H12 There is a statistically significant pairwise difference in prediction effectiveness between any of the adopted time-series forecasting models. Similarly, we would like to assess the impact of the defined forecast-windows (original, weekly, biweekly, monthly) on the forecasting effectiveness of the adopted models. Hence, we formulate the following hypotheses: H03 There is no statistically significant difference in prediction effectiveness when the adopted time-series forecasting models are trained with serialized or original data. H13 There is a statistically significant difference in prediction effectiveness when the adopted time-series forecasting models are trained with serialized or original data. Similarly as we aim to test the pairwise difference among models, we are keen to investigate which tested set of observations made the models perform better rather than the order sets. Therefore, we perform a post hoc analysis testing the following hypothesis: H04 There is no statistically significant difference in prediction effectiveness when the adopted time-series forecasting models are trained with a serialized timewindow and the original data. H14 There is a statistically significant difference in prediction effectiveness when the adopted time-series forecasting models are trained with a serialized timewindow and the original data. Following recent research [26], we will first test the normality on the distribution of the obtained model results. For that, we leverage the Anderson–Darling (AD) test [27] to test whether the observations to be tested are normally distributed (null hypothesis), or not (alternative hypothesis). If the AD test results in accepting the null hypothesis, we assess the differences in model performance using parametric tests. Hence, we apply a one-way ANOVA [22]. Conversely, if the AD test results in not normally distributed data we employ the Wilcoxon signed-rank tests (WT) [28]. If we were to fail the null hypothesis in any of the presented hypotheses, and therefore we would have evidence on statistically significant differences over the obtained error results, we perform a post hoc comparison using Dunn’s test without a control group in each of the cases (as presented in the formulated post-hoc null hypotheses). The Dunn’s test is a non-parametric statistical method commonly used for identifying specific group differences after a prior test has assessed the hypothesis with a statistically significant difference between the inference groups [29]. It extends statistical inference to multiple pairwise comparisons by assessing all possible group combinations without requiring a control group, thereby enabling us to identify which groups yield statistically significant results under specific paired comparisons. REFERENCES [1] M. Robredo, N. Saarim¨ aki, M. Esposito, D. Taibi, R. Pe˜ naloza, and V. Lenarduzzi, “Evaluating time-dependent methods and seasonal effects in code technical debt prediction,” Journal of Systems and Software, p. 112545, 2025. [2] M. Mathioudaki, D. Tsoukalas, M. Siavvas, and D. Kehagias, “Comparing univariate and multivariate time series models for technical debt forecasting,” in International Conference on Computational Science and Its Applications. Springer, 2022, pp. 62–78. [3] G. E. Box, S. C. Hillmer, and G. C. Tiao, “Analysis and modeling of seasonal time series,” in Seasonal analysis of economic time series. NBER, 1978, pp. 309–344. [4] H. Hong, B. P. Carlin, T. A. Shamliyan, J. F. Wyman, R. Ramakrishnan, F. Sainfort, and R. L. Kane, “Comparing bayesian and frequentist approaches for multiple outcome mixed treatment comparisons,” Medical Decision Making, vol. 33, no. 5, pp. 702–714, 2013. [5] F. J. Samaniego, A comparison of the Bayesian and frequentist approaches to estimation. Springer, 2010, vol. 24. [6] L. V. Jospin, H. Laga, F. Boussaid, W. Buntine, and M. Bennamoun, “Hands-on bayesian neural networks—a tutorial for deep learning users,” IEEE Computational Intelligence Magazine, vol. 17, no. 2, pp. 29–48, 2022. [7] A. C. Harvey, Dynamic models for volatility and heavy tails: with applications to financial and economic time series. Cambridge University Press, 2013, vol. 52. [8] R. J. Hyndman, M. Akram, and B. C. Archibald, “The admissible parameter space for exponential smoothing models,” Annals of the Institute of Statistical Mathematics, vol. 60, no. 2, pp. 407–426, 2008. [9] J. Nakajima and M. West, “Bayesian analysis of latent threshold dynamic models,” Journal of Business & Economic Statistics, vol. 31, no. 2, pp. 151–164, 2013. [10] M. West, P. J. Harrison, and H. S. Migon, “Dynamic generalized linear models and bayesian forecasting,” Journal of the American Statistical Association, vol. 80, no. 389, pp. 73–83, 1985. [11] Q. Zhang, C. Fang, Y. Xie, Y. Zhang, Y. Yang, W. Sun, S. Yu, and Z. Chen, “A survey on large language models for software engineering,” CoRR, vol. abs/2312.15223, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2312.15223 [12] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in IEEE/ACM International Conference on Software Engineering: Future of Software Engineering, ICSE-FoSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 31–53. [Online]. Available: https://doi.org/10.1109/ICSE-FoSE59343. 2023.00008 [13] M. Peixeiro, Time series forecasting in python. Simon and Schuster, 2022. [14] A. Garza, C. Challu, and M. Mergenthaler-Canseco, “Timegpt-1,” arXiv preprint arXiv:2310.03589, 2023. [15] K. Rasul, A. Ashok, A. R. Williams, A. Khorasani, G. Adamopoulos, R. Bhagwatkar, M. Biloˇ s, H. Ghonia, N. Hassen, A. Schneider et al., “Lag-llama: Towards foundation models for time series forecasting,” in R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023. [16] A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapuram, S. P. Arango, S. Kapoor et al., “Chronos: Learning the language of time series,” arXiv preprint arXiv:2403.07815, 2024. [17] A. Roberts and C. Raffel, “Exploring transfer learning with t5: the textto-text transfer transformer,” Google AI blog, 2020. [18] G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified training of universal time series forecasting transformers,” 2024. [19] A. Das, W. Kong, R. Sen, and Y. Zhou, “A decoder-only foundation model for time-series forecasting,” in Forty-first International Conference on Machine Learning, 2024. [20] Y. Liu, H. Zhang, C. Li, X. Huang, J. Wang, and M. Long, “Timer: Generative pre-trained transformers are large time series models,” arXiv preprint arXiv:2402.02368, 2024. [21] A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are transformers effective for time series forecasting?” in Proceedings of the AAAI conference on artificial intelligence, vol. 37, no. 9, 2023, pp. 11 121–11 128. [22] P. J. Brockwell and R. A. Davis, Introduction to time series and forecasting. Springer, 2002. [23] J. D. Cryer and K.-S. Chan, Time series analysis: with applications in R. Springer, 2008. [24] D. C. Montgomery, C. L. Jennings, and M. Kulahci, Introduction to time series analysis and forecasting. John Wiley & Sons, 2015. [25] M. Stone, “Cross-validatory choice and assessment of statistical predictions,” Journal of the royal statistical society: Series B (Methodological), vol. 36, no. 2, pp. 111–133, 1974. [26] M. Esposito, V. Falaschi, and D. Falessi, “An extensive comparison of static application security testing tools,” in International Conference on Evaluation and Assessment in Software Engineering, 2024, pp. 69–78. [27] T. W. Anderson and D. A. Darling, “Asymptotic theory of certain” goodness of fit” criteria based on stochastic processes,” The annals of mathematical statistics, pp. 193–212, 1952. [28] F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics bulletin, vol. 1, no. 6, pp. 80–83, 1945. [29] O. J. Dunn, “Multiple comparisons using rank sums,” Technometrics, vol. 6, no. 3, pp. 241–252, 1964.