Time Complexity of Population-Based Metaheuristics

MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX ISSN: 1803-3814 (Printed), 2571-3701 (Online) https://doi.org/10.13164/mendel.2023.k.261 Optimizing Neural Networks for Academic Performance Classification Using Feature Selection and Resampling Approach Didi Supriyadi1,4,  , Purwanto Purwanto1,2, Budi Warsito3 1Doctorate Program of Information Systems, School of Postgraduate Studies Universitas Diponegoro, Semarang, Indonesia 2Department of Chemical Engineering, Faculty of Engineering, Universitas Diponegoro, Semarang, Indonesia 3Department of Statistics, Faculty of Science and Mathematics, Universitas Diponegoro, Semarang, Indonesia 4Department of Information System, Faculty of Informatics, Institut Teknologi Telkom Purwokerto, Banyumas, Indonesia didisup[email protected]  , purw[email protected], [email protected] Abstract The features present in large datasets significantly affect the performance of machine learning models. Redundant and irrelevant features will be rejected and cause a decrease in machine learning model performance. This paper proposes HyFeS-ROS-ANN: Hybrid Feature Selection and Resampling combination method for binary classification using artificial neural network multilayer perceptron (MLP). The first stage of this approach is to use a combination of two feature selection methods to select essential features that are highly correlated with model performance. The second stage of this approach is to use a combination of resampling methods to handle unbalanced data classes. Both approaches are applied to the academic performance classification model using the MLP neural network. This research dataset is obtained using three-dimensional (3D) frameworks such as the Big Five Personality to determine the Personality that affects academic performance from the student dimension, the Family Influence Scale (FIS), which measures factors that affect academic performance from the family dimension, and Higher Education Institutions Service Quality (HEISQUAL) to measure service quality and its influence on academic performance from the Education institution dimension. Previous research shows that the CoR-ANN algorithm has a model accuracy rate of 94%. The research results based on the dataset show that our proposed method can improve accuracy by selecting more relevant and essential features in improving model performance. The results show that the features are reduced from 135 to 108, while the HyFS-ROS-ANN model for binary classification accuracy increases to 100%. Keywords: Feature Selection, Imbalanced Dataset, Resampling Approach, Neural Network, Academic Performance, Personality, Family, Service Quality. Received: 28 November 2023 Accepted: 11 December 2023 Online: 12 December 2023 Published: 20 December 2023 1 Introduction The academic performance of students is subject to a variety of influences stemming from diverse sources. These factors range from the quality of education to a student’s unique circumstances, each contributing to the overall academic performance [9]. In terms of the student dimension, research conducted by Nye et al. [28] and Foong et al. [13] demonstrates a compelling correlation between student motivation and academic performance. This finding has significant implications for educational institutions and underscores the importance of cultivating student motivation as a key factor in achieving academic success. Furthermore, according to Mateus et al. [24], academic performance is also influenced by personality traits. In terms of family life, Mishra [25] conducted research on the impact of family on academic performance. Additionally, Vautero et al. [40] utilized the Family Influence Scale (FIS) framework to explore the factors that drive student academic performance. Looking from the perspective of higher education institutions, the academic performance of students is significantly impacted by their satisfaction levels [38]. The satisfaction of students is closely tied to the quality of services provided by the institutions, which includes the caliber of teaching staff, the curriculum, the facilities, and the development of student skills and competencies [1]. Previous studies have used machine learning to predict student academic performance. Machine learning enables computers to learn automatically from a set of data, making intelligent decisions based on pattern recognition from complex data [15]. Classification is a supervised machine learning technique that predicts class labels for test data by referring to already labeled classes from a set of available training data. It employs an iterative algorithm to acquire knowledge and discover hidden knowledge. Zeineddine et al. [44] achieved up to 83% accuracy with their ensemble model, while Lottering et al. [21] reported 94.14% ac261 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX curacy using another classification algorithm. Baashar et al. [6] analyzed student academic performance with Neural Network Back Propagation algorithm, obtaining an accuracy of 89%. Research also shows that Neural Networks (NN) algorithm outperformed Support Vector Machine (SVM) and Random Forest (RF) algorithms in personality classification, achieving an accuracy of 76% [36]. When working with machine learning models, it’s important to curate data carefully to avoid performance degradation [42]. This is especially important in Educational Data Mining (EDM), where selecting the most relevant features is crucial for accurate predictive models. A thorough investigation into feature selection techniques can greatly impact the outcome of student performance models and contribute to the overall advancement of the field [7]. In the domain of EDM, having an imbalanced dataset can be a significant challenge that leads to inaccurate outcomes and poor performance. Alongside selecting relevant features for the classification model, achieving a balanced dataset is equally crucial for success. The primary difficulty faced is in creating a model that can effectively classify minority classes, as these students stand to benefit the most from targeted pedagogical interventions. To address this issue, numerous resampling techniques have been developed, and researchers continue to investigate the most effective strategies for dealing with imbalanced data [27], [23] Addressing the issue of imbalanced class datasets can be achieved using two primary approaches, which are data-level methods and algorithm-level methods. Data-level methods involve pre-processing techniques that rebalance the class distribution, with options for under-sampling or over-sampling. Algorithm-level methods employ hybrid and ensemble techniques to tackle the class imbalance problem [23]. This study extends Supriyadi et al.’s research that used neural networks to classify student academic performance [36,35]. This research proposes a model called HyFeS-CoR-ANN, which stands for Hybrid Feature Selection and a combination of Resampling methods for binary classification using artificial neural network (ANN). The model integrates hybrid feature selection with resampling methods in neural network models for binary classification. The main objective of this integration is to solve the problem of feature selection and handling of unbalanced dataset classes. The goal is to create a neural network model that can effectively classify the academic performance of students based on whether or not they graduate on time. The dataset was collected by considering multiple dimensions and factors that affect students’ academic performance using the Big Five Personality traits [34] framework to determine which personality traits affect academic performance. The Family Influence Scale (FIS) [14] was used to measure factors from the family dimension that affect academic performance, while the Higher Education Institutions Service Quality (HEISQUAL) [1] was used to measure students’ level of satisfaction with university services and how it influences their academic performance from the Higher Education dimension. This study makes the following contributions: (1) A new dataset includes data on factors affecting academic performance, collected using three questionnaires: BFP for personality, FIS for family, and HEISQUAL for college. (2) Our proposed approach utilizes feature selection combine with resampling techniques to optimize academic performance modeling with neural networks for binary classification. (3) The feature selection process combines Pearson Correlation and Recursive Feature Elimination techniques to generate relevant features and enhance the ANN model’s performance. (4) The utilization of resampling techniques to address imbalanced class datasets, we can significantly enhance the accuracy and effectiveness of our ANN models. The research is organized into the following sections: Section 1 - Introduction: This section provides background information about the problem, objectives, and contributions of the research. Section 2 - Materials and Methods: This section describes the materials used in the research, the research methods, and the proposed model. Section 3 – Experimental and Results: This section interprets the experimental environment, experimental design, results and analysis of the research. Section 4 - Conclusion: This section concludes the research, outlines its limitations and suggests future work that can be carried out. 2 Materials and Methods 2.1 Questionnaires Personality Traits. The International Personality Item Pool (The IPIP-Big5) test is a personality test that measures five main personality traits. These traits were first explored by Costa and McCrae [11] in the revised NEO personality inventory (NEO-PI-R). The test consists of 50 questions, with 10 questions for each of the five traits: Intellect or Openness (O), Conscientiousness (C), Extraversion (E), Agreeableness (A), and Emotional Stability or Neuroticism (N). The test uses a 5-point Likert scale to measure each response. The scores for each trait should range from 0 to 40, based on the answers given to the 10 questions for that trait [34]. This test is useful in determining a student’s personality type and understanding their strengths and weaknesses. Below is a brief description of each personality trait [18]: (1) Extraversion (E) - Seek fulfilment from sources within the community - High scores: social and outgoing - Low scores: prefer working alone and reserved. 262 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Supriyadi-g2igHX,gO ptimizing Neural Networks for Academic Performance Classification Using Feature Selection... (2) Agreeableness (A) - Adjust behaviour to suit others - High scores: polite, empathetic, and cooperative - Low scores: blunt, sceptical, and critical. (3) Conscientiousness (C) - Honest, diligent, and dependable - High scores: organized, responsible, and rule-following - Low scores: impulsive, careless, or deceitful. (4) Neuroticism (N) - Tendency towards anxiety, insecurity, and negative emotions - High scores: easily stressed, moody, and reactive - Low scores: more relaxed, stable, and resilient. (5) Openness (O) - Curiosity, creativity, and appreciation for new experiences - High scores: imaginative, artistic, and adventurous - Low scores: traditional, conventional, and practical. Family Influence Scale (FIS). The FIS measures how one’s family influences career and occupational choices. It has four domains: informational and financial support, family expectations, and values/benefits as shown in Table 1. The Likert scale tool ranges from 1-5. Fouad et al. [14] introduced it, and Vautero et al. [40] and Silva et al. [33] used it to study family influence on students’ academic performance. Table 1: Domains and Description of FIS. Domains Features Count Description Informational support 8 One’s family can be a valuable resource in a job search by providing relevant information Family expectation 6 Ensuring a clear understanding of the career goals that meet the expectations of one’s family Financial support 5 The extent to which a family is willing to offer financial assistance to others Values/benefits 3 The family anticipates that its members will pursue careers that are in alignment with their values Higher Education Institutions Service Quality (HEISQUAL). The HEISQUAL instrument, developed by Abbas [1], is a tool used to evaluate the quality of higher education institutions. It consists of 63 statement items that were selected after conducting an Exploratory Factor Analysis (EFA). The statement items are grouped into seven main themes or dimensions, which include teachers’ profile, curriculum, infrastructure and facilities, management and support staff, employment quality, safety and security, and students’ skills development. These seven dimensions are further divided into sixteen sub-dimensions, which include Subject Knowledge, Communication Skills, Teaching Style, Behaviour with Students, Curriculum Quality, Learning Facilities, Supportive Facilities, Cleanliness and Maintenance, Behaviour with Students, Administrative Work, Links with Employers, Employability Training, Security Measures, Safety Equipment, Extra-Curricular Activities, and Personal Development as shown in Table 2. The HEISQUAL instrument uses a 5-point Likert-type rating scale. Our study employed three highly effective instruments - BFP, FIS, and HEISQUAL - to collect comprehensive data. The resulting 135 features were carefully assessed using a 5-point Likert scale and included 50 features from BFP, 22 from FIS, and 63 from HEISQUAL. With this robust dataset, we are confident in our ability to draw meaningful conclusions and insights that will inform future research and decision making. 2.2 Pre-Processing Pre-processing is a crucial step in data processing and manipulation for machine learning. It involves examining missing data, determining features and labels, assessing data distribution, and dividing data into training and testing datasets. It’s recommended to divide the dataset into a ratio of 80/20 or 70/30 for more accurate and reliable results. Pre-processing improves algorithm accuracy and efficiency by eliminating inconsistencies and errors in the data. It’s crucial to pay close attention to this stage to ensure that the data is accurate, clean, and suitable for analysis. Failing to do so may lead to inaccurate models and flawed analyses, which can have significant consequences in both academic and business settings. 2.3 Feature Selection Feature selection aims to simplify the model by including only the most significant and relevant features, it involves identifying and selecting a subset of input variables that are most useful in predicting the target variable. There are many approaches to selecting the most important features including filtering, wrapper, and embedded methods [17]. Filtering methods rank the variables based on some statistical measure, such as correlation or mutual information, and select the top-ranked variables. Wrapper methods use a model to evaluate the performance of a subset of features and iterate over all possible feature combinations to find the best subset. Embedded methods, on the other hand, perform feature selection as part of the model-building process, such as Lasso regression or decision tree algorithms. Choosing the most appropriate features can lead to a significant reduction in the dimensionality of the raw dataset. This reduction can have a positive impact on model performance by addressing the issues of model variance and overfitting. Additionally, reducing the number of features can lead to a reduction in the computational cost of the model, making it more efficient [31]. This study aimed to enhance the performance of classification models for student academic performance by testing various feature selection techniques. The study employed filtering methods including Pearson Correlation Coefficient (PCC) [22], which is known to improve performance [20]. Additionally, the study utilized wrapper methods such as Recursive feature Elimination (RFE) [37], and a combination of both methods, to produce relevant and appropriate features. The selection of appropriate features is crucial in achieving 263 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Table 2: Domains and Description of HEISQUAL. Domains Sub-domains Feature Count Description Teachers’ profile Subject knowledge 4 An instructor’s deep understanding of the content they are responsible for imparting to their students Communication skills 4 The instructor’s skills to effectively convey information, create a positive learning environment, and build meaningful relationships with students Teaching style 4 The approach and methods a instructor uses to facilitate learning and engage with students Behaviour with students 5 The quality and interaction of instructors with students that contributes to creating a positive and effective learning environment Curriculum Curriculum quality 4Quality education requires effective, relevant, and appropriate content and experiences for students. Infrastructure and Facilities Learning facilities 4 Support Learning facilities include physical and environmental elements that contribute to students’ educational experience. Supportive facilities 3 Support resources and spaces contribute to holistic and supportive student well-being, safety, and development. Cleanliness and maintenance 3 The level of cleanliness and maintenance of facilities to create a conducive and healthy learning environment. Managementand support staff Behaviour with students 3 The quality and interaction of management and support staff with students that contributes to creating a positive and effective learning environment Administrative work 6 The efficient functioning of any educational institution depends on quality administrative work. Employment quality Link with employers 3 Quality relationships between educational institutions and employers to provide meaningful learning experiences for students. Employability training 4 Support the development of skills, knowledge, and attributes necessary for individuals to succeed in the world of work. Safetyand security Security measures 4 Ensure the safety of students, staff, and the overall school community. Safety equipment 3 Support safety equipment for students, staff, and visitors to the educational environment. Students’ skills development Extra-curricular activities 3 Activities for students’ learning, growth, skills, interests, and socializing beyond academics. Personal development 6 Support personal growth by developing skills and knowledge for self-awareness, potential, and fulfilment. optimal classification model performance. By employing these techniques, the study aimed to identify features that are highly correlated with the target variable. The use of filtering methods allows the study to identify features with high correlation, while wrapper methods such as RFE select the most relevant features by recursively eliminating the least important ones. 2.4 Imbalanced Datasets Dealing When dealing with imbalanced datasets, it is important to have effective strategies in place. These strategies should help to address the issue of unequal distribution of data across the classes in the dataset. A few effective strategies include using oversampling or under-sampling techniques, using synthetic data generation, or changing the decision threshold. By using these strategies, it is possible to better train a model that can accurately predict outcomes for imbalanced datasets. A plethora of resampling techniques have been developed to address the issue of unbalanced data sets in machine learning. These techniques encompass under-sampling, oversampling, combining under-sampling and oversampling, and ensemble sampling. The primary objective of these techniques is to modify the ratio between majority and minority classes to construct a more balanced set of datasets for machine learning training. When the dataset for the training process exhibits a balanced ratio between classes, it can lead to a significant improvement in the performance of the classification model [8]. This study employs the Random Under-sampling (RUS) and Random Oversampling (ROS) techniques as well as a combination of both. Notably, both techniques are known for their high efficacy [26]. 2.5 Artificial Neural Networks for Classification A neural network is an information processing system that bears similarity to biological neural networks. It consists of dendrites, which receive information, a soma, which processes information, and axons, which transmit information to other cells. This type of network has become an important tool in various applications, including image and speech recognition, natural language processing, and predictive modelling. Researchers have shown a growing interest in the potential of neural networks, and studies in this area continue to expand our understanding of their capabilities and limitations [39,32]. A neural network is a type of algorithm used for supervised learning, which is commonly implemented for classification and prediction purposes [4]. 264 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Supriyadi-g2igHX,gO ptimizing Neural Networks for Academic Performance Classification Using Feature Selection Figure 1: Architecture Artificial Neural Networks for classifying academic performance. Neurons constitute the fundamental components of Artificial Neural Networks (ANNs) that enable the mapping from different input layers to output layers. Each neuron comprises a summation block where the inputs are weighted and undergo an activation function, following which the output is calculated. The weights represent the strength of the connections between the neurons and are crucial for the network to minimize the error concerning the output. In general, the topology of an ANN, i.e., the arrangement of connections between neurons, the learning algorithm, i.e., the approach to obtaining the strength of the connections, and the activation function, together characterize any ANN [29] as shown in Figure 1. Artificial Neural Networks have revolutionized the field of education by providing a powerful tool to predict students’ academic performance. The use of a simple Artificial Neural Network as a prediction model, expressed mathematically in Equation (1), has proven to be a game-changer in the quest for improving educational outcomes. This study represents a significant milestone in the pursuit of a better future for students worldwide, as it offers new insights and opportunities to enhance their academic progress and success. y=fw(X) (1) The neural network uses a predefined function fw to calculate the prediction output yfrom the input X. This function consists of multiple stages connected by a non-linear function, with each stage having W parameters used for multiplication and addition. An example is shown in Figure 3. Input vector Xundergoes a linear dot-product operation with parameter W1before being transmitted to neurons. The neurons apply a non-linear function such as ReLU, which sets the negative weighted sum to zero and returns the weighted sum as represented in Equation (2). Re = max(0, X) (2) The hidden layer output values Z undergo the same linear operation, but with different parameters W2. The output layer neurons take the weighted sum and pass it through the sigmoid function Equation (3), which compresses the result into a value between 0 and 1. S(X) = 1 1 + e−x(3) For binary classification, the sigmoid output falls within [0,1]. The decision boundary sets the output to 0 or 1. Here, <0.5 means ”Pass on time,” while ≥0.5 means ”Pass not on time.” Training a neural network involves choosing the best Wparameter value for accurate predictions. 2.6 Performance Evaluation Understanding the performance of the student academic performance classification model with machine learning algorithms can be done using a confusion matrix. This matrix provides an overview of how well the model can classify classes into specific categories. It divides the classification results into four main groups, namely: (1) True Positive (TP): the number of positive classes the model correctly identifies as positive. (2) True Negative (TN): the number of negative classes the model correctly identifies as negative. (3) False Positive (FP): the number of negative classes the model incorrectly identifies as positive. (4) False Negative (FN): the number of positive classes the model incorrectly identifies as negative. The confusion matrix describes the performance of the student academic performance classification model, including four performance indicators as below and mathematically modelled in Equations (4)– (7)[36,2]: (1) Accuracy: the proportion of total classes that are correctly classified (both students who ”PASSED ON TIME” and students who ”PASSED NOT ON TIME”). 265 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Accuracy =TP +TN TP +T N +F P +F N (4) (2) Precision: the proportion of positive classes (PASSED ON TIME) that are genuinely positive (also PASSED ON TIME) of the total classified as positive (PASSED ON TIME) by the model. Precision =TP TP +F P (5) (3) Recall or Sensitivity: the proportion of positive classes (PASSED ON TIME) that are genuinely positive (PASSED ON TIME) from the total classes that are positive (PASSED ON TIME). Recall =TP TP +F N (6) (4) F1 Score: a harmonious combination of precision and recall. F1=2×Precision ×Recall Precision +Recall (7) Figure 2: Proposed model HyFS-ROS-ANN. 2.7 Proposed Model for Solution The present study proposes a work method to address the issue of student academic performance based on a dataset obtained from the dimensions of student personality (BFP), family support (FIS), and college service quality (HEISQUAL). The proposed methodology adopts an approach that employs a machine learning algorithm and comprises four primary stages. These stages include data pre-processing, model training and validation, model testing, and model evaluation, as illustrated in Figure 2. To elucidate, the data pre-processing stage entails data cleaning and validation, define feature and label, label encoding, and normalization and integrating the data to ensure that the machine learning algorithm can process it efficiently. In this study, the data was collected using three different questionnaires, resulting in the identification of a total of 135 features. To improve the performance and computational process, it is crucial to select the most relevant features during the pre-processing stage. Two feature selection techniques were evaluated in this research, namely filtering and wrapper. The filtering technique used was Pearson Correlation Coefficient (PCC), which measures the linear relationship between features. The wrapper technique used was Recursive Feature Elimination (RFE), which recursively removes the least significant feature until the desired number of features is obtained. Both methods were evaluated separately, and a hybrid approach combining both techniques (PCC-RFE) was also tested. This selection process ensures that only relevant and significant features are retained, contributing to a more efficient and accurate analysis of the data. The PCC method, which is a statistical technique, evaluates the correlation between features and the target variable, and the results showed that it was effective in identifying relevant features. The RFE method, which is a machine learning algorithm, iteratively eliminates the least relevant features until the desired number of features is reached. The hybrid Feature Selection (HyFS) approach using PCC and RFE combines the strengths of both methods. This approach explores the possibility of the HyFS approach outperforming each technique in identifying the most relevant features. The HyFS using PCC-RFE algorithm can be mathematically represented as follows: Xis a feature matrix with dimensions m×n, where mis the number of samples and nis the number of features. ρxy is the Pearson correlation coefficient between feature x and target y. We select nselected features and assign an importance weight wito each feature i. We rank the features based on their importance using a machine learning model. Thus, the algorithm of HyFS using PCC-RFE as shown in Table 3. In order to develop a reliable and accurate predictive model for student academic performance, it is essential to address the issue of imbalanced datasets. In this regard, the present research incorporates a range of techniques for handling imbalanced datasets, such as Random Under-sampling (RUS) and Random Oversampling (ROS) methods. The RUS method involves randomly removing samples from the majority class, whereas the ROS method involves generating new samples for the minority class. This study investigates the effectiveness of Random Under-Sampling (RUS), Random Over-Sampling (ROS), and their combined approaches in addressing class imbalance issues in binary classification. Table 4 shows the mathematical descriptions of these algorithms. The Combined algorithm integrates both techniques to enhance dataset balance, thereby mitigating the impact of class imbalance. Once the best features have been chosen and a balanced dataset has been obtained, the ANN classification model is trained. To achieve this, the dataset is divided into three parts, namely, the training dataset, the validation dataset, and the testing dataset. In training and testing experiments, the primary objective is 266 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Supriyadi-g2igHX,gO ptimizing Neural Networks for Academic Performance Classification Using Feature Selection Table 3: Proposed HyFS Algorithm using PCC-RFE as part of HyFS-ROS-ANN. Step Algorithm HyFS using PCC-RFE 1. Calculate ρxy between each feature xiand target yusing Equation (8). ρxy =Cov(xi, y) Std(xi)×Std(y)(8) 2. Select features with |ρxy|values above a certain threshold as candidate features to be included in SelectedFeatures. 3. Set SelectedFeatures =∅(empty set). 4. As long as |SelectedFeatures|< nselected, perform the following iterations: - Train the Model on features Xremaining = X−SelectedFeatures. - Calculate the importance of feature wi using Model.feature importances or similar metric. -Identify the least important features based on the model criteria. - Remove the least important features from SelectedFeatures. 5. If |SelectedFeatures|=nselected, stop the process, otherwise repeat Step 4. 6. Output of the model is the set of SelectedFeatures that contains the selected features. Table 4: Handling Imbalanced Datasets using RUS and ROS as part of HyFS-ROS-ANN. No Operation RUS 1. Randomly remove instances from Cma, resulting in a new set Cma−rus Cma−rus =RandomSubset(Cma, TargetRatio) ROS 2. SRandomly replicate instances from Cmi, resulting in a new set Cmi−ros Cmi−rus =RandomSubset(Cmi, TargetRatio) Combine 3. Combine RUS and ROS: - Apply RUS to Cma to obtain Cma−rus - Apply ROS to Cmi to obtain Cmi−ros The balanced dataset is given by Equation (9) Cbalanced =Cma−rus ∪Cmi−ros (9) to recognize and comprehend complex patterns within the data. These patterns are subsequently utilized by the learning model to make accurate predictions. The efficiency of the learning model is determined based on its ability to effectively generalize to new data. Generalization refers to the model’s ability to correctly predict outcomes based on previously unseen data. Experimental results serve as a key metric for evaluating the learning model’s efficiency. In order to avoid the issue of poor generalization, candidate classifiers were experimented with using two separation criteria. This approach helps to identify which classifiers are most effective in handling new data. A reduced dataset was used, and then divided into two separate sets consisting of 70% for training and 30% for testing. By doing so, the learning model is exposed to a wide range of data, leading to improved efficiency when generating predictions [45]. During the model testing stage, the performance of the trained model is evaluated using data that hasn’t been seen before. On the other hand, the model evaluation stage involves analysing and refining the model’s performance to optimize it. To evaluate the performance of classification models in machine learning, there are different methods. One of the most common methods is using a confusion matrix that measures accuracy, precision, recall (sensitivity), and the F1-score. The assessment of student performance is conducted through an evaluation of the model’s ability to accurately classify students into binary categories. These categories represent ”Passed on time” and ”Passed not on time”, which are denoted by the respective values of 0 and 1. This method of evaluation is used to gauge the effectiveness of the model in predicting students’ academic outcomes. 3 Experiments and Results 3.1 Experimental Environment The present study’s experimental procedures were conducted on a desktop equipped with 16 gigabytes of RAM and an Intel(R) Core (TM) i7-8550U central processing unit, running the 64-bit Windows 11 Home Single Language Version 22H2 operating system. Our experimental environment was developed on the Python 3.10 programming language, with the Neural Networks Multi-Layer Perceptron model being implemented on TensorFlow 2.14.0. Our data processing, feature selection, and visualization functions were carried out using Scikit-Learn, Numpy, pandas, and matplotlib software packages. 3.2 Experimental Design The objective of our research was to develop an accurate and effective model for classifying student academic performance by exploring various techniques integrated with neural network algorithms. Our study involved a comprehensive exploration of the following techniques: (1) ANN classification models without feature selection techniques or resampling techniques (ANN) (2) ANN classification models integrated with PCC or RFE feature selection techniques (FS-ANN) (3) ANN classification models integrated with a hybrid of PCC and RFE feature selection techniques (HyFS-ANN) (4) ANN classification models integrated with RUS or ROS resampling techniques (R-ANN) 267 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Table 5: Experimental Results. Model Algorithm Feature Selection Resampling Accuracy Precision Recall F1-score ANN ANN NA NA 67 63 61 61 RUS-ANN ANN NA RUS 53 54 54 53 ROS-ANN ANN NA ROS 72 72 72 72 CoR-ANN ANN NA CoR 51 51 51 51 PCC-ANN ANN PCC NA 97 96 96 96 PCC-RUS-ANN ANN PCC RUS 96 96 96 96 PCC-ROS-ANN ANN PCC ROS 99 99 99 99 PCC-CoR-ANN ANN PCC CoR 98 98 98 98 RFE-ANN ANN RFE NA 62 58 58 58 RFE-RUS-ANN ANN RFE RUS 56 56 56 55 RFE-ROS-ANN ANN RFE ROS 73 74 73 73 RFE-CoR-ANN ANN RFE CoR 53 53 53 53 HyFS-ANN ANN HyFS NA 97 97 95 96 HyFS-RUS-ANN ANN HyFS RUS 93 94 93 93 HyFS-ROS-ANN ANN HyFS ROS 100 100 100 100 HyFS-CoR-ANN ANN HyFS CoR 98 98 98 98 (5) ANN classification models integrated with a combination of RUS and ROS resampling techniques (CoR-ANN) (6) ANN classification models integrated with both feature selection and resampling techniques (FSR-ANN) (7) ANN classification models integrated with hybrid feature selection techniques and a combination of resampling techniques (HyFS-CoR-ANN). The performance parameters, which encompassed accuracy, precision, recall, and f1-score, were analyzed during the observation. These techniques were employed to evaluate the effectiveness of the tested scheme. The results were meticulously recorded and analyzed to provide valuable insights into the scheme’s performance and identify any areas that require further optimization. Figure 3: Ratio Train-Val-Test Datasets. 3.3 Dataset In machine learning, datasets play a critical role in building a classification model for student academic performance in higher education. The effectiveness of performance testing and model generalization can be determined through these datasets. The Dimensions of Student Intelligence (DSI) questionnaire, which combines the BFP, FIS and HEISQUAL questionnaires, was used to collect data from July 2022 to June 2023. A total of 397 Indonesian undergraduate program graduates completed the DSI questionnaire which consists of 135 indicators. The dataset showed that 148 respondents graduated on time (within 3.5-4 years), while 249 respondents graduated later than four years. The dataset shows an imbalance between Pass on Time and Pass Not on Time classes. Therefore, the dataset needs to be balanced between on-time and late graduates. The imbalance ratio (IR) as shown in Equation 10 is used to measure the level of dataset imbalance, which increases proportionally [46] if it is greater than 1. Based on the number of datasets obtained, the dataset imbalance ratio generated in this study is 1.68. IR =Nmaj Nmin (10) 3.4 Results of Proposed Model HyFS-ROS-ANN The dataset has been divided into two distinct parts: training data and test data, with a ratio of 70:30. Subsequently, a 50:50 ratio has been employed on the test data to generate validation data and testing data. A visual representation of the dataset division can be found in Figure 3. The dataset undergoes training and testing using predefined models and techniques specified in the experimental design section. This ensures the data is analyzed accurately and the results obtained are unbiased. Careful attention was paid to each process step, from model selection to result analysis. The findings of the explorations conducted are presented in Table 5. The proposed model seeks to assess the effectiveness of the ANN algorithm as a classification model combined with feature selection and resampling techniques. The resulting performance can substantially improve the accuracy of predicting student performance. In analyzing the performance of the proposed model, some important things that need to be observed and analyzed are as follows: (1) Analyzing attributes through feature selection methods. (2) Examining the dataset using resampling methods. (3) Evaluating the performance model based on feature selection and resampling methods. 268 MENDEL — Soft Computing Journal, Volume 29, No.gk, .2+2K#2` 2023, Brno, Czech RepublicX Supriyadi-g2igHX,gO ptimizing Neural Networks for Academic Performance Classification Using Feature Selection Table 6: Comparison of Feature Selection Method with Previous Study. Study Classifier Feature Selection Method Performance Jebadurai et al., [30] DT Filtering (Pearson – Spearman) Accuracy: 91 - 92% Zaffar et al., [43] MLP Filtering (Chi Square) Precision: 76.9% Duong et al., [12] LGBM Filtering (Pearson) Accuracy: 92.62% Jeon and Oh, [16] KNN, SVM, RF, NB Hybrid-RFE Accuracy: 95.8% Channabasavaraju and Vinayakamurthy, [10] RF RFE Accuracy: 83.49% Proposed ANN HyFS (PCC-RFE) Accuracy: 100% Table 7: Comparison of Resampling Method with Previous Study. Study Algorithm Balancing Approach Performance Bagui and Li, [8] ANN Random Resampling (RURO) Recall: 96% Salih and Khalaf, [30] SVM Heuristic Resampling (SMOTE) Accuracy: 78.84% Duong et al., [12] LGBM Class weight Accuracy: 92.62% Proposed ANN ROS Accuracy: 100% Analyzing attributes through feature selection methods. This study aims to investigate the effectiveness of filtering and wrapper feature selection methods, specifically PCC and RFE. These methods have been chosen because they have been shown to enhance the performance of models in numerous previous studies. Table 6 presents a comparison of our approach with previous research. The DSI questionnaire was developed by integrating the BFP, FIS, and HEISQUAL, which resulted in the use of 135 indicators as attributes. These attributes represented various aspects of a personality, family influence, and service quality of institution. To refine the questionnaire further, PCC (Pearson Correlation Coefficient) and RFE (Recursive Feature Elimination) feature selection techniques were applied. PCC measures the linear correlation between two variables, while RFE recursively eliminates less important features until the most significant ones are retained. As a result, the number of attributes was reduced to 108, indicating that the most relevant and significant attributes were retained while filtering out the less important ones. This in turn ensured that the DSI questionnaire was more precise and effective in assessing the factors that affect students’ academic performance. Examining datasets using the resampling method. The research aims to overcome imbalanced class distribution in the dataset to create an accurate model for student academic performance classification. A resampling technique has been employed based on a thorough literature review detailed in Table 7 and compared against the proposed model. Table 4’s formula generates variations in the dataset resulting in different sets of data as shown in Table 8. Table 8: Datasets Generated using Resampling Methods. Class Original RUS ROS CoR (RUS-ROS) 0 148 148 249 148 1 249 148 249 148 Total 397 296 498 296 Meanwhile, the CoR dataset is generated by applying formula 11, which is calculated as follows: n(A∪B) = n(A) + n(B)−n(A∩B) (11) Suppose: n(A∪B) = total number of elements in both Aand B,n(A) = number of elements in set A,n(B) = number of elements in set B,n(A∩B) = number of elements common to both Aand B. Thus, the number of datasets of CoR as follows: CoR=148+249-(249148)=397-101=296. Evaluating performance models. The selection of artificial neural networks was based on their widespread usage in academic literature as shown in Table 10 and their ability to represent tree-based and deep-learning classifiers. The familiarity of these networks in the literature makes them a suitable choice for research and analysis. Additionally, their ability to represent complex relationships between variables and their capacity to learn from data make them a valuable tool for classification tasks [3]. The subsequent section presents an in-depth analysis of the performance evaluation results of the proposed HyFS-ROS-ANN model. The evaluation was conducted using the confusion matrix parameters, which include accuracy, precision, recall, and f1-score. The primary objective of the HyFSROS-ANN model is to classify student academic performance based on various dimensions and influencing factors such as student personality, family support, and college service quality. The performance evaluation was conducted based on a predetermined experimental design scenario. Table 5 and Figure 4 provides a detailed report of the experimental results as observed in the confusion matrix parameters. The analysis was conducted by comparing the performance of four models, including the HyFS-ROS-ANN model, PCC-ROSANN model, PCC-CoR-ANN model, and HyFS-CoRANN model. The results indicated that the HyFSROS-ANN model exhibited exceptional performance by achieving 100% accuracy, precision, recall, and f1score values. The PCC-ROS-ANN model achieved 99% accuracy, precision, recall, and f1-score values, while the PCC-CoR-ANN and HyFS-CoR-ANN models achieved 98% accuracy, precision, recall, and F1score values. After conducting a thorough analysis of the confusion matrix conditions for each model that was tested, as outlined in Table 9, it was observed that the HyFSROS-ANN model demonstrated exceptional perfor269