Navigating AI in Naval Architecture: A Comparative Effectiveness Study of Machine Learning Models for Ship Stability

89 Navigating AI in Naval Architecture: A Comparative Effectiveness Study of Machine Learning Models for Ship Stability Karolina Bierkowska, NTNU, Ålesund/Norway, [email protected] Henrique Murilo Gaspar, NTNU, Ålesund/Norway, henrique.[email protected] Tomasz Hinz, Gdańsk Tech/Deltamarin, Gdańsk/Poland, [email protected]du.pl Abstract This study comparatively analyses diverse AI/ML models on an established dataset of hull variants and Second-Generation Intact Stability Criteria metrics, a time-consuming task in the early stages of ship design. This selection encompasses diverse AI techniques, each recognised for its unique strengths, featuring Artificial Neural Networks, Decision Trees, Probabilistic Models, and Large Language Models. In this article, we focus primarily on one of the failure modes: excessive acceleration. This study benchmarks the models against each other in terms of predictive accuracy, computational efficiency, robustness. The objective is to evaluate the analyses and identify a "superior" AI/ML strategies for faster, more reliable early-stage design stability assessment. This research guides naval architects in selecting suitable emergent AI tools to enhance design space exploration, ultimately contributing to filter the current AI "hype" into useful NA practices in the industry. 1. Introduction 1.1. Motivation for Using SGISC in This Study Ship stability has a significant impact on operations, including safety, efficiency, and regulatory compliance. Between 1998 and 2014, nine documented accidents occurred due to failures in dynamic stability systems. In October 1998, the APL China lost approximately 400 containers overboard and damaged another 400 due to parametric rolling, France et al. (2003). In September 2008, aboard the Chicago Express, one person died and several were injured when the ship rolled to an angle of 45 degrees due to excessive acceleration, Kaufmann (2009). In January 2012, the Rabaul Queen sank, reportedly due to broaching-to, Australia and Mininga (n.d.). These incidents represent a subset of documented cases. Of the nine accidents, eight were compliant with the 2008 IS Code, IMO (2009), the current intact stability criteria—classified as safe in terms of stability. The underlying cause of each accident was related to stability, highlighting that the current criteria do not sufficiently address all relevant failure modes. Based on these findings, the development of new stability criteria was initiated. In 2020, the IMO’s Maritime Safety Committee approved the interim guidelines for the new stability criteria, with the first explanatory notes published in April 2023, IMO (2020,2023). A first revision of the document was published in 2024, IMO (2024). Although the criteria are not yet mandatory, designers are encouraged to consider them when designing new ships, Begovic et al. (2023). These new criteria are known as the Second Generation Intact Stability Criteria (SGISC). They are not intended to replace the 2008 IS Code, but rather to complement it. SGISC addresses five stability failure modes: parametric rolling, pure loss of stability, broaching-to, dead ship condition, and excessive acceleration. SGISC is based on a multi-level framework, Fig.1, where Level 1 is the most conservative but also the easiest to calculate, IMO (2024). The second level is based on solving the differential equation of a simplified model of ship motion. 90 Fig.1: Scheme of SGISC framework, Petacco and Gualeni (2020) 1.2. Motivation for Using AI To calculate SGISC excessive acceleration, you need information such as main dimensions and hull shape, bilge keel geometry, locations of passengers and crew, and loading conditions. These design parameters are often unavailable during the early stages of ship design, while at later stages, main dimensions and other characteristics are difficult or impossible to modify. Therefore, it is crucial to evaluate this criterion before deciding on the main dimensions, as they significantly affect the outcome, Sayli et al. (2007). Setting up the required parameters for this criterion takes approximately 15 minutes. Even when all required dimensions are available, evaluating multiple design options using AI is significantly faster, often requiring only seconds. AI has already demonstrated its applicability in ship design. For example, it can be used to predict GM and KG at the preliminary stage, Alkan et al. (2004), to optimize trim, Vasilev et al. (2024), or to develop hull form, Bagazinski and Ahmed (2023), Khan et al. (2023), Ichinose and Gaspar (2023). 2. Data Preparation for AI Models This study focuses on the failure mode of excessive acceleration, evaluated at Level 2 of the SGISC framework. The criterion was calculated using the NAPA software, and the hull geometry was generated through a parametric model in CAESES, Harries et al. (2019). Further details on the parametric model preparation and calculation process can be found in previous work, Bierkowska et al. (2025). The models were prepared according to the input configurations listed in Table I. The data were grouped based on dimensional parameters, and for each input set, the worst-case outcome was selected. Specifically, if any combination within a set (with identical dimensional values) failed the criterion (A01 = 0), the entire set was marked as failing. This data preparation method ensures that the models are trained using the most safety-critical cases, making their predictions more conservative. The dataset was then split into a training set (90% of the data) and a test set (10%). 91 Table I: Inputs Model name input Model.1 LOA, Beam, Height, CB Model.2 LOA, Beam, Height, CB, Length Keel, Breadth Keel, GM Model.3 LOA, Beam, Height, CB, Length Keel, Breadth Keel, GM, X Point, Z Point Model.4 LOA, Beam, Height, CB, LCB Model.5 LOA, Beam, Height, CB, Length Keel, Breadth Keel, GM, LCB Model.6 LOA, Beam, Height, CB, Length Keel, Breadth Keel, GM, X Point, Z Point, LCB Model.7 LOA, Beam, Height, CB, LCB, T Model.8 LOA, Beam, Height, CB, Length Keel, Breadth Keel, GM, LCB, T Model.9 LOA, Beam, Height, CB, Length Keel, Breadth Keel, GM, X Point, Z Point, LCB, T Model.10 LOA, Beam, CB, Length Keel, Breadth Keel, GM Model.11 LOA, Beam, CB, Length Keel, Breadth Keel, GM, X Point, Z Point Model.12 LOA, Beam, CB, LCB Model.13 LOA, Beam, CB, Length Keel, Breadth Keel, GM, LCB Model.14 LOA, Beam, CB, Length Keel, Breadth Keel, GM, X Point, Z Point, LCB Model.15 LOA, Beam, CB, LCB, T Model.16 LOA, Beam, CB, Length Keel, Breadth Keel, GM, LCB, T Model.17 LOA, Beam, CB, Length Keel, Breadth Keel, GM, X Point, Z Point, LCB, T 3. Using AI to Evaluate SGISC Stability 3.1. Deep Neural Network TensorFlow, Shanmugamani (2018), was used to train the Deep Neural Network (DNN) models. All models were trained using the same architecture. Table II presents the number of neurons in each layer for different iterations. A normalization layer was placed after the input layer. After each hidden layer, a dropout layer, Salehin and Kang (2023), with a rate of 0.2 was added. Table II: Number of neurons on layers for different iteration Iteration number Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 DNN-0 32 16 8 DNN-1 64 32 16 8 DNN-2 16 8 DNN-3 128 64 32 16 8 DNN-4 256 128 64 32 16 8 DNN-5 512 256 128 64 32 16 8 DNN-6 1024 512 256 128 64 32 16 8 The Rectified Linear Unit (ReLU), Panda (2023), was used as the activation function. The output layer uses a sigmoid activation function, which produces values between 0 and 1. Predictions greater than 0.5 were classified as 1 (pass), while those less than or equal to 0.5 were classified as 0 (fail). Adaptive Moment Estimation (Adam), Kingma and Ba (2014), was used as the optimizer. The binary Cross-Entropy function was employed as the loss function. Model performance was evaluated using the binary accuracy metric. 25% of the training data was used for validation. The model exhibiting the lowest validation loss was selected for evaluation. 3.2. Large Language Models For the Large Language Models (LLM) experiments, Ollama, https://github.com/ollama/ollama, was used, specifically the phi3:instruct model, https://ollama.com/library/phi3:instruct. The phi3:instruct was not trained; instead, it was used through prompt-based queries. In each prompt, n examples from the training set that were similar to the test case were included. These examples were selected based 92 on the smallest Euclidean distance to the test vector, after standardizing the features. Phi3:instruct was asked to predict whether the test case met the stability criterion. To determine the optimal configuration, the models were tested with n values of 2, 10, and 20. For each model, 100 test examples were randomly selected to determine which n value produced the best results. The full test set was then evaluated using that optimal n. The value of n used for each model is shown in Table III. Table III: Number of examples in prompt model 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 n 10 2 10 2 20 2 20 2 2 10 20 10 20 10 10 20 2 3.3. Decision Tree and Gradient Boosting Another machine learning model used in these studies is Decision Trees. Decision Trees, Watt et al. (2020), are among the most commonly used models in supervised learning, offering an intuitive and interpretable framework for classification tasks. Decision Tree induction is typically performed through recursive partitioning of the input space, guided by criteria such as information gain, gain ratio, or the Gini index, Yale et al. (2017), which determine the attribute that best separates the data at each step. In this part of the analysis, the library used to create the Decision Tree was the Scikit-learn library, Pedregosa et al. (2011). An example of a Decision Tree is shown in Fig.2. Fig.2: Example of Decision Tree (Model 1, max depth 3) Fig.3: Example of one tree from Gradient Boosting (Model 1, max depth 3) 93 While a single Decision Tree offers interpretability and simplicity, its instability and tendency to overfit have motivated the development of ensemble approaches Gradient Boosting is an ensemble learning technique that builds predictive models in a stage-wise, sequential manner, where each new decision tree is trained to reduce the errors (residuals) of the previous ensemble. Gradient Boosting, Friedman (2001), optimizes a chosen loss function using gradient descent, allowing it to achieve high predictive accuracy—though at the cost of greater risk of overfitting and higher computational demand. The scikit-learn library was used again. Example of Gradient Boosting is shown in Fig.3. 3.4 Probabilistic programming Probabilistic programming provides a framework for specifying complex statistical models in a highlevel, declarative manner, enabling automated inference over probabilistic models, Blei et al. (2017). Unlike traditional machine learning approaches, which often yield point estimates, probabilistic programming embraces uncertainty by treating model parameters as random variables with associated probability distributions, Ness (2025). Within this paradigm, Bayesian Neural Networks (BNNs) Pinheiro Cinelli et al. (2021) extend conventional neural networks by placing prior distributions over their weights and biases, resulting in posterior distributions. One of the most widely used libraries in probabilistic programming is PyMC, Abril-Pla et al. (2023). Example of probabilistic programming model is shown in Fig.4. Fig.4: Example of probabilistic programming 4. Matthews Correlation Coefficient metrics Some models are imbalanced—for example, some have only 2% of cases where the criterion is passed—accuracy is not a reliable metric. Therefore, the Matthews Correlation Coefficient was used instead. The Matthews Correlation Coefficient (MCC), Chicco and Jurman (2020), is a measure of how well a binary classification model performs. An MCC of +1 indicates perfect prediction, 0 indicates random guessing, and −1 indicates complete misclassification. It is particularly useful for imbalanced datasets. MCC is defined as: 94 𝑀𝐶𝐶 = 𝑇𝑃 ⋅𝑇𝑁 −𝐹𝑃 ⋅𝐹𝑁 √(𝑇𝑃 +𝐹𝑃)(𝑇𝑃 +𝐹𝑁)(𝑇𝑁 +𝐹𝑃)(𝑇𝑁 +𝐹𝑁) Where: - True Positive (TP) – both actual and predicted values are 1 - True Negative (TN) – both actual and predicted values are 0 - False Positive (FP) – actual value is 0, but predicted as 1 - False Negative (FN) – actual value is 1, but predicted as 0 In cases where the phi3:instruct failed to provide a prediction—likely due to hallucination—these were treated as False Positives if the actual value was 0, and as False Negatives if the actual value was 1. 5. Results MCC scores for phi3:instruct, Probabilistic programming, Decision Tree and Gradient Boosting are presented in Table IV. MCC scores for the DNNs are presented in Table V. Table IV: MCC score for phi3:instruct, Probabilistic programming, Decision Tree and Gradient Boosting model phi3:instruct Probabilistic programming Decision Tree Gradient Boosting Model.1 0.211 0 0.146 0.061 Model.2 0.485 0 0.815 0.742 Model.3 -0.989 0 0.872 0.774 Model.4 0.232 0 0.489 0.202 Model.5 -0.785 0 0.899 0.778 Model.6 -0.983 0 0.837 0.796 Model.7 0.002 0 0.733 0.321 Model.8 0.544 0 0.907 0.786 Model.9 -0.811 0 0.902 0.805 Model.10 0.503 -0.007 0.765 0.731 Model.11 -0.992 0 0.864 0.769 Model.12 0.264 0 0.582 0.376 Model.13 -0.917 0 0.872 0.767 Model.14 -0.947 0 0.895 0.794 Model.15 0.362 0 0.693 0.321 Model.16 -0.918 0.004 0.896 0.782 Model.17 -0.926 0 0.898 0.802 95 Table V: MCC score for Deep Neural Networks model DNN-0 DNN-1 DNN-2 DNN-3 DNN-4 DNN-5 DNN-6 Model.1 0 0 0 0 0.206 0 0 Model.2 0.64 0.667 0.63 0.698 0.723 0.782 0.831 Model.3 0.643 0.668 0.619 0.675 0.735 0.822 0.849 Model.4 0.206 0.367 0 0 0 0 0.206 Model.5 0.693 0.744 0.645 0.808 0.893 0.926 0.945 Model.6 0.658 0.714 0.635 0.801 0.781 0.921 0.944 Model.7 0.51 0.559 0.206 0.585 0.596 0.663 0.581 Model.8 0.722 0.82 0.662 0.897 0.943 0.962 0.971 Model.9 0.708 0.772 0.654 0.871 0.918 0.953 0.956 Model.10 0.637 0.653 0.626 0.67 0.67 0.692 0.709 Model.11 0.646 0.653 0.621 0.665 0.682 0.704 0.688 Model.12 0.358 0.367 0.292 0.361 0 0 0.161 Model.13 0.668 0.725 0.646 0.794 0.839 0.882 0.892 Model.14 0.659 0.691 0.618 0.785 0.853 0.897 0.914 Model.15 0.409 0.495 0.142 0.596 0.585 0 0.495 Model.16 0.699 0.75 0.663 0.846 0.901 0.938 0.946 Model.17 0.695 0.742 0.626 0.825 0.883 0.939 0.94 The plot of MCC scores for all models is presented in Fig.5. For clarity, only the DNN results corresponding to the neuron configuration that produced the best performance are presented. Fig.5: The plot of MCC score for all models 96 6. Discussions This study explores the potential of using various AI techniques in early-stage ship design to accelerate the calculation process. This approach is particularly useful when not all required dimensions are available, or when designers wish to test multiple sets of dimensional parameters. For example, predictions using phi3:instruct take approximately 5 seconds, while DNN models require only a few milliseconds. In contrast, calculations performed in NAPA software take around 15 minutes. The results demonstrate promising performance, particularly for DNNs and Decision Trees with MCC values exceeding 0.9. The best-performing DNN model achieved an MCC of 0.971. In contrast, Probabilistic programming assigned the same class to all inputs, demonstrating it ineffective for this application. Phi3:instruct, on the other hand, exhibited poor scores (close to -1) likely caused by hallucinations. Using alternative models from Ollama may help decrease the frequency of these hallucinations. If its performance improves, phi3:instruct could serve as a viable alternative for users without AI expertise—which was the primary motivation for including it in this study. It does not require a large dataset, is open-source, and operates offline, ensuring that no data is transmitted to external servers. Future work could investigate whether using examples with varying combinations of dimensional parameters in the phi3:instruct prompt affects prediction accuracy. For instance, one prompt might include examples with bilge keel dimensions, while another uses position-related parameters such as X Point and Z Point. This is a realistic approach, as in practice we may not always have access to a complete set of data from previous projects, making it necessary to work with whatever dimensional parameters are available. Most DNN models achieved their best results in iteration 6, suggesting that further iterations with more neurons could yield even better performance. Acknowledgements This paper is partly funded by the Smart European Shipbuilding Projekt (SEUS). References ABRIL-PLA, O.; ANDREANI, V.; CARROLL, C.; DONG, L.; FONNESBECK, C.J.; KOCHUROV, M.; KUMAR, R.; LAO, J.; LUHMANN, C.C.; MARTIN, O.A.; et al. (2023), PyMC: a modern, and comprehensive probabilistic programming framework in Python, PeerJ. Computer Science 9, p. e1516, https://doi.org/10.7717/peerj-cs.1516 ALKAN, A.D.; GULEZ, K.; YILMAZ, H. (2004), Design of a robust neural network structure for determining initial stability particulars of fishing vessels, Ocean Eng. 31(5–6), pp.761–777 AUSTRALIA, R.; WILSON MININGA, W. (n.d.), Commission of Inquiry into the sinking of Rabaul Queen COMMISSION OF INQUIRY INTO THE SINKING OF RABAUL QUEEN, https://www.academia.edu/29381006/Commission_of_Inquiry_into_the_sinking_of_Rabaul_Queen_ COMMISSION_OF_INQUIRY_INTO_THE_SINKING_OF_RABAUL_QUEEN BAGAZINSKI, N.J.; AHMED, F. (2023), Ship-D: Ship Hull Dataset for Design Optimization using Machine Learning, arXiv, https://doi.org/10.48550/ARXIV.2305.08279 97 BEGOVIC, E.; BERTORELLO, C.; RINAURO, B.; ROSANO, G. (2023), Simplified operational guidance for second generation intact stability criteria, Ocean Eng. 270, p. 113583 BIERKOWSKA, K.; HINZ, T.; MAZERSKI, G. (2025), Application of Neural Networks in EarlyStage Ship Design for Stability Evaluation Using IMO Second, Innovations in Sustainable Maritime Technology (IMAM 2025) BLEI, D.M.; KUCUKELBIR, A.; McAULIFFE, J.D. (2017), Variational Inference: A Review for Statisticians, J. American Statistical Association 112(518), pp.859–877 CHICCO, D.; JURMAN, G. (2020), The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation, BMC Genomics 21(1), p.6, https://doi.org/10.1186/s12864-019-6413-7 FRANCE, W.N.; LEVADOU, M.; TREAKLE, T.W.; PAULLING, J.R.; MICHEL, R.K.; MOORE, C. (2003), An Investigation of Head-Sea Parametric Rolling and Its Influence on Container Lashing Systems, Marine Technology and SNAME News, 40(01), pp.1–19 FRIEDMAN, J.H. (2001), Greedy function approximation: A gradient boosting machine, The Annals of Statistics 29(5), https://doi.org/10.1214/aos/1013203451 HARRIES, S.; ABT, C.; BRENNER, M. (2019), Upfront CAD—Parametric Modeling Techniques for Shape Optimization, Advances in Evolutionary and Deterministic Methods for Design, Optimization and Control in Engineering and Sciences, Springer pp.191–211 ICHINOSE, Y.; GASPAR, H.M. (2023), Interactive Ship Flow Simulation Enhanced by Neural Network Model in a Web Environment, ECMS, pp.155–161, https://doi.org/10.7148/2023-0155 IMO (2009), International Code on Intact Stability, 2008, Int. Mar. Org., London IMO (2020), Interim guidelines on the second generation intact stability criteria, Int. Mar. Org., London IMO (2023), Explanatory Notes to the Interim Guidelines on the Second Generation Intact Stability Criteria. MSC.1/Circ.1652, Int. Mar. Org. London IMO (2024), Explanatory Notes to the Interim Guidelines on the Second Generation Intact Stability Criteria. MSC.1/Circ.1652/Corr.1, Int. Mar. Org. London KAUFMANN, J. (2009), Fatal accident on board the CMV Chicago express during typhoon “Hugupit” on September 24 2008 off the Coast of Hong Kong, Investigation Report 510(08), Bundesstelle für Seeunfalluntersuchung, KHAN, S.; GOUCHER-LAMBERT, K.; KOSTAS, K.; KAKLIS, P. (2023), ShipHullGAN: A generic parametric modeller for ship hull design using deep convolutional generative model, Computer Methods in Applied Mechanics and Eng. 411, p.116051, https://doi.org/10.1016/j.cma. 2023.116051 KINGMA, D.P.; BA, J. (2014), Adam: A Method for Stochastic Optimization, arXiv, https://arxiv.org/ pdf/1412.6980 NESS, R.O. (2025), Causal AI, Manning