TRANSFORMER-BASED CODE ANALYSIS FOR AUTOMATED VULNERABILITY DISCOVERY

Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [444] TRANSFORMER-BASED CODE ANALYSIS FOR AUTOMATED VULNERABILITY DISCOVERY Jonas Meier* and Elena Rossi Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark * Correspondence: jona[email protected]m ABSTRACT The exponential growth in software complexity has resulted in an unprecedented increase in security vulnerabilities, posing significant threats to modern computing systems. Traditional vulnerability detection approaches rely heavily on manual feature engineering and pattern matching, which are labor-intensive, subjective, and often produce high false positive or false negative rates. This paper presents a comprehensive investigation of Transformer-based neural network architectures for automated vulnerability discovery in source code. Leveraging the self-attention mechanism pioneered by the Transformer architecture, our approach automatically learns semantic and syntactic features from large-scale code repositories without requiring human-defined vulnerability patterns. We propose a six-stage deep learning framework that transforms source code into vector representations, processes them through multi-layer bidirectional networks, and classifies vulnerabilities with high accuracy. Through extensive experiments on the SATE IV Juliet test suite and real-world vulnerability datasets, our Transformer-based approach demonstrates superior performance compared to traditional static analysis tools including Flawfinder, Clang, and Cppcheck, as well as conventional deep learning methods such as CNN and RNN. The results show that our model achieves 92.9% F1-score with a true positive rate exceeding 95% while maintaining false positive rates below 5%, significantly outperforming baseline methods. This research contributes to advancing automated vulnerability detection through the application of state-of-the-art natural language processing techniques to software security. Keywords: Transformer networks, vulnerability detection, code analysis, deep learning, BLSTM, software security, automated detection 1. INTRODUCTION Software vulnerabilities represent critical weaknesses in computer systems that can be exploited by malicious actors to compromise data integrity, confidentiality, and availability. The landscape of cybersecurity threats has evolved dramatically over the past decade, with the number of disclosed vulnerabilities increasing exponentially each year. According to recent systematic literature reviews, more than 40,000 Common Vulnerabilities and Exposures identifiers were published in 2024 alone, reflecting the growing complexity and scale of modern software systems [1]. The financial and reputational costs associated with security breaches have reached unprecedented levels, with major incidents resulting in losses exceeding billions of dollars globally. This alarming trend underscores the urgent need for more effective and automated approaches to vulnerability Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [445] detection that can scale with the rapid pace of software development while maintaining high accuracy and low false alarm rates. Traditional vulnerability detection methodologies have predominantly relied on static analysis tools, dynamic testing frameworks, and manual code review processes [2]. Static analysis tools such as Flawfinder, Cppcheck, and Clang employ rule-based pattern matching defined by security experts to identify potential vulnerabilities through lexical analysis and control flow examination. While these tools can detect known vulnerability patterns with reasonable speed, they suffer from fundamental limitations including extremely high false positive rates and inability to detect novel vulnerability types that do not match their predefined patterns [3]. As demonstrated in empirical evaluations on standard test suites, traditional static analyzers typically achieve true positive rates below 40% while generating false positive rates exceeding 50%, making them impractical for deployment in continuous integration environments where developers require reliable and actionable security feedback [4]. The requirement for human experts to continuously update and maintain detection rules makes the approach both expensive and difficult to scale as new vulnerability patterns emerge in modern programming paradigms and frameworks. Dynamic analysis approaches complement static analysis by examining program behavior during execution, allowing them to detect vulnerabilities that manifest only under specific runtime conditions [5]. Fuzzing techniques generate random or mutated inputs to trigger unexpected program behaviors that may indicate security vulnerabilities, while symbolic execution attempts to systematically explore program paths by treating inputs as symbolic variables. However, dynamic analysis methods face significant challenges in achieving comprehensive code coverage, especially in large and complex software systems where exploring all possible execution paths is computationally infeasible due to path explosion problems [6]. Furthermore, dynamic testing requires substantial infrastructure for test case generation, execution monitoring, and result analysis, making it resource-intensive and time-consuming for large-scale software projects. Hybrid approaches combining static and dynamic analysis have been proposed to leverage the strengths of both methodologies, but they still require substantial computational resources and expert knowledge to configure and interpret results effectively. The advent of deep learning has opened new avenues for automated vulnerability detection by enabling machines to automatically learn distinguishing features from large volumes of code without explicit human guidance [7]. Early deep learning approaches to vulnerability detection primarily utilized Convolutional Neural Networks and Long Short-Term Memory networks to process sequential representations of source code [8]. VulDeePecker, one of the pioneering systems in this domain, demonstrated the feasibility of using Bidirectional LSTM networks to detect vulnerabilities by learning patterns from code gadgets extracted through data flow analysis [9]. The system achieved significant improvements over traditional static analysis tools by automatically learning vulnerability patterns from labeled training data, reducing false negative rates from over 90% to approximately 7% on certain vulnerability types. While these methods demonstrated promising results compared to traditional approaches, they faced limitations in capturing long-range dependencies and complex semantic relationships within code structures that span multiple functions or modules. Recurrent neural networks, despite their sequential processing capabilities, struggle with vanishing gradient problems when analyzing lengthy code sequences that commonly occur in real-world software systems [10]. The unidirectional nature of standard RNNs prevents them from fully understanding bidirectional context that is crucial for accurate vulnerability detection, as security flaws often involve interactions between earlier and later Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [446] code statements. Bidirectional LSTM networks address some of these limitations by processing sequences in both forward and backward directions, enabling the model to capture context from both past and future tokens [11]. However, LSTM-based approaches still face challenges with very long sequences due to their sequential processing nature, which limits parallelization during training and inference. Additionally, the fixed-size hidden state in LSTMs may not adequately represent complex code semantics that involve multiple interacting program elements across distant code locations. The introduction of the Transformer architecture has revolutionized natural language processing by demonstrating unprecedented capabilities in understanding complex linguistic patterns through self-attention mechanisms [12]. The Transformer's ability to process sequential data in parallel while capturing long-range dependencies makes it particularly suitable for code analysis tasks where understanding relationships between distant code elements is essential for identifying vulnerabilities. Unlike recurrent architectures that process tokens sequentially with information flowing through hidden states, Transformers can attend to all positions in a sequence simultaneously through multi-head attention mechanisms, enabling them to capture dependencies between distant code elements that may be critical for vulnerability detection [13]. Recent studies have shown that Transformer-based models can achieve state-of-the-art performance on various code-related tasks including code completion, bug detection, clone detection, and vulnerability identification [14]. Pre-trained language models such as CodeBERT and GraphCodeBERT have demonstrated remarkable capabilities in understanding both natural language and programming language contexts by training on millions of source code functions from open-source repositories [15]. These models learn general-purpose representations that capture syntactic structures, semantic meanings, and common programming patterns, which can be fine-tuned for specific downstream tasks including vulnerability detection. CodeBERT achieves this through a bimodal pre-training objective that processes both code and natural language documentation, enabling it to understand the semantic relationships between code implementations and their textual descriptions. GraphCodeBERT extends this approach by incorporating structural information from data flow graphs during pre-training, providing enhanced understanding of code dependencies that are critical for security analysis [16]. This research investigates the application of Transformer-based architectures for automated vulnerability discovery, addressing the limitations of previous approaches while leveraging the strengths of attention mechanisms and pre-trained models. Our contributions include the development of a comprehensive six-stage framework that transforms source code into semantically meaningful representations suitable for Transformer models, the adaptation of pre-trained language models specifically for vulnerability detection tasks through targeted fine-tuning strategies, and extensive experimental evaluation on standard benchmarks demonstrating the superiority of our approach over both traditional static analysis tools and conventional deep learning methods. Through systematic experiments on the SATE IV Juliet test suite, we demonstrate that our Transformer-based approach achieves true positive rates exceeding 95% while maintaining false positive rates below 5%, representing a substantial improvement over existing methods. By visualizing learned attention patterns and analyzing model behavior, we provide insights into how Transformer models identify vulnerabilities and what code characteristics they rely upon for making predictions. 2. LITERATURE REVIEW Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [447] The field of automated vulnerability detection has witnessed significant evolution over the past decade, transitioning from rule-based approaches to sophisticated machine learning and deep learning techniques. Understanding this evolution is essential for contextualizing our Transformer-based approach and identifying the gaps that our research addresses. This section provides a comprehensive review of existing vulnerability detection methodologies, organized into three main categories: traditional static and dynamic analysis approaches, conventional machine learning techniques, and modern deep learning-based methods with emphasis on the transition from recurrent architectures to attention-based models. Traditional static analysis tools have long been the cornerstone of vulnerability detection in software development environments, offering the advantage of examining source code without requiring program execution [17]. These tools analyze source code through lexical analysis, syntax parsing, control flow graph construction, and pattern matching against predefined vulnerability signatures. Flawfinder represents an early example of static analysis tools that scan C and C++ source code for potentially dangerous function calls such as strcpy, gets, and sprintf that are commonly associated with buffer overflow vulnerabilities [18]. The tool operates by maintaining a database of security-sensitive functions and flagging their usage along with a risk assessment score. However, Flawfinder's simple lexical pattern matching approach results in extremely high false positive rates, as it cannot distinguish between safe and unsafe usage contexts, flagging all instances of potentially dangerous functions regardless of whether proper bounds checking or input validation is implemented. Cppcheck offers more sophisticated analysis capabilities by performing limited data flow analysis and detecting various types of programming errors including memory leaks, null pointer dereferences, and buffer overflows [19]. The tool constructs simplified control flow graphs and performs basic reaching definitions analysis to identify potential security issues. Despite these enhancements, Cppcheck still relies on manually defined detection rules that must be continuously updated as new vulnerability patterns emerge. Clang Static Analyzer, integrated into the LLVM compiler infrastructure, provides industrial-strength static analysis through symbolic execution and path-sensitive analysis [20]. The analyzer explores multiple execution paths through the program, maintaining symbolic values for variables and detecting inconsistencies or violations of programming invariants. While more accurate than simpler lexical scanners, Clang still produces significant numbers of false positives and struggles with complex inter-procedural analysis scenarios where vulnerabilities involve interactions across multiple functions. Commercial static analysis tools like Checkmarx and Fortify offer more comprehensive analysis capabilities, including sophisticated taint analysis for tracking untrusted data flow from sources to sinks, data flow analysis across function boundaries, and integration with development environments for real-time feedback [21]. These tools employ multiple analysis techniques including abstract interpretation, symbolic execution, and constraint solving to identify potential vulnerabilities. However, they fundamentally remain limited by their reliance on manually crafted detection rules defined by security experts, which must be constantly updated to address new vulnerability types and programming patterns. The effectiveness of these tools varies significantly depending on the quality and comprehensiveness of their rule databases, and they often require substantial tuning and configuration to reduce false positive rates to acceptable levels for specific projects. Dynamic analysis approaches examine program behavior during execution to detect vulnerabilities that may not be apparent through static code inspection alone [22]. Fuzzing techniques systematically generate test inputs to Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [448] explore program behavior and trigger unexpected conditions that may indicate security vulnerabilities. Coverage-guided fuzzing tools like AFL (American Fuzzy Lop) use evolutionary algorithms to generate test cases that maximize code coverage, increasing the likelihood of discovering deeply embedded bugs [23]. Dynamic taint analysis tracks the propagation of untrusted input through program execution, detecting when tainted data reaches security-sensitive operations without proper sanitization. While dynamic analysis can detect certain classes of vulnerabilities that static analysis misses, particularly those involving complex runtime conditions, it faces fundamental limitations in achieving comprehensive code coverage within reasonable time bounds. Path explosion problems make exhaustive testing impractical for non-trivial programs, and dynamic analysis typically achieves only partial coverage of program behavior. The application of machine learning to vulnerability detection emerged as researchers recognized that security flaws often exhibit patterns that can be learned from historical data rather than explicitly encoded in rules [24]. Early machine learning approaches used traditional classifiers such as Support Vector Machines, Random Forests, and Naive Bayes, trained on hand-crafted features extracted from source code. These features typically included code complexity metrics such as cyclomatic complexity and lines of code, code churn statistics measuring the rate of code changes, and developer activity patterns such as the number of developers modifying a file [25]. While these methods showed promise in predicting vulnerable components at coarse granularities such as files or functions, they struggled with precisely localizing vulnerabilities within large codebases. The effectiveness of these approaches was fundamentally limited by the quality and relevance of manually engineered features, which required deep domain expertise and often failed to capture subtle vulnerability patterns that transcend simple statistical properties. Deep learning techniques have fundamentally transformed vulnerability detection by enabling automatic feature learning from raw source code representations, eliminating the need for manual feature engineering [26]. Convolutional Neural Networks were among the first deep learning architectures applied to this domain, treating source code as sequences of tokens and learning hierarchical feature representations through multiple convolutional layers with pooling operations. Russell and colleagues demonstrated that CNNs could achieve competitive performance on vulnerability detection tasks by learning features directly from lexed function source code, using one-dimensional convolutions over token sequences to capture local patterns indicative of vulnerabilities [27]. Their system combined CNN-based feature extraction with Random Forest classifiers for final vulnerability prediction, achieving significant improvements over static analysis tools on carefully constructed test datasets. However, CNNs' reliance on local receptive fields limits their ability to capture long-range dependencies between distant code elements that may be critical for understanding complex vulnerability patterns. Long Short-Term Memory networks and their bidirectional variants emerged as promising alternatives for vulnerability detection due to their ability to capture sequential dependencies and maintain long-term memory of previously processed tokens [28]. The VulExplore system combined CNNs with LSTM networks to leverage both local feature extraction through convolution and sequential pattern learning through recurrent processing, achieving improved accuracy on code metric-based vulnerability detection tasks. The hybrid architecture first applies convolutional layers to extract local features from code metrics, then feeds these features into LSTM layers to capture temporal dependencies and learn representations of vulnerability patterns that unfold over sequences of statements. VulDeePecker pioneered the application of Bidirectional LSTM specifically designed Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [449] for vulnerability detection, introducing the concept of code gadgets as focused representations of code relevant to potential vulnerabilities [29]. The system extracts program slices based on data dependencies related to library function calls, assembles them into code gadgets, and processes these gadgets through BLSTM networks to classify them as vulnerable or non-vulnerable. Graph-based deep learning approaches have gained prominence due to their ability to model structural relationships in code through program dependency graphs, control flow graphs, and abstract syntax trees [30]. Devign pioneered the use of Graph Neural Networks for vulnerability detection, representing code as graphs where nodes correspond to program statements and edges represent control and data dependencies between them. The graph neural network propagates information between nodes through message passing, allowing it to reason about complex interactions between different parts of the code and capture vulnerability patterns that involve multiple related statements across different program locations. LineVD extended this work by performing statement-level vulnerability detection using graph neural networks with attention mechanisms, providing finer-grained localization of security flaws down to individual lines of code rather than entire functions. However, graph-based methods face challenges in handling large and complex graphs that arise from real-world software systems, and the construction of accurate program dependency graphs itself requires sophisticated static analysis tools that may introduce errors or omissions in dependency relationships. The emergence of pre-trained language models has opened new possibilities for code analysis tasks by leveraging transfer learning from large code corpora containing millions of functions [31]. CodeBERT, a bimodal pre-trained model for programming and natural language, demonstrated that models pre-trained on extensive code repositories could achieve superior performance on downstream tasks including code search, code documentation generation, and vulnerability detection. The model employs a masked language modeling objective similar to BERT, randomly masking tokens in code sequences and training the model to predict them based on surrounding context, thereby learning rich representations that capture both syntactic and semantic properties of code. GraphCodeBERT enhanced this approach by incorporating structural information from data flow graphs during pre-training, enabling the model to better understand code dependencies beyond mere token sequences. VulCoBERT applied these pre-trained models specifically to vulnerability detection, combining CodeBERT embeddings with BiLSTM networks to classify code functions as vulnerable or non-vulnerable, achieving substantial improvements over models trained from scratch. Transformer-based models have recently emerged as the state-of-the-art approach for various code understanding tasks, offering significant advantages over recurrent architectures through their self-attention mechanisms and parallel processing capabilities. Thapa and colleagues explored the application of Transformer-based language models to software vulnerability detection, demonstrating that attention mechanisms can effectively capture long-range dependencies critical for identifying security flaws that involve interactions between distant code elements [32]. VulD-Transformer proposed using Transformer models to process code slices containing data and control dependencies extracted from program dependency graphs, showing significant improvements over LSTM-based approaches particularly when dealing with long code sequences exceeding 256 tokens where recurrent networks struggle with gradient propagation [33]. The self-attention mechanism enables Transformers to selectively focus on relevant code tokens regardless of their position in the sequence, overcoming the limitations of recurrent architectures in handling long-range dependencies through their ability to directly model relationships between any pair of positions in the input. Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [450] More recent work has explored various Transformer architectures for code analysis including encoder-only models like RoBERTa adapted for code understanding, decoder-only models like CodeGPT for code generation, and encoder-decoder architectures for code translation and repair tasks [34]. Transfer learning studies have investigated the effectiveness of different pre-training strategies and fine-tuning approaches for vulnerability prediction, comparing encoder-only models like CodeBERT with decoder-only models like CodeGPT on various vulnerability detection benchmarks [35]. Results indicate that while both architectures can achieve strong performance through appropriate fine-tuning, encoder-only models generally perform better for classification tasks like vulnerability detection due to their bidirectional context encoding, whereas decoder-only models excel at generative tasks. Additionally, research has explored various techniques for improving Transformer-based vulnerability detection including structure-aware soft prompt tuning that incorporates code structure information into prompts, and knowledge distillation approaches that compress large Transformer models into smaller, more efficient variants suitable for deployment in resource-constrained environments. Despite these advances, several challenges remain in applying Transformer-based models to vulnerability detection that motivate our research. The computational cost of training large Transformer models from scratch is substantial, requiring significant GPU resources and large amounts of training data that may not be available for specialized vulnerability types. The black-box nature of deep learning models raises interpretability concerns, as developers need to understand why a particular code snippet is flagged as vulnerable to make informed decisions about remediation and to trust the system's recommendations. Data quality and labeling issues persist, as obtaining accurately labeled vulnerability datasets is challenging due to the complexity of identifying true vulnerabilities versus false positives in real-world code, and the evolving nature of vulnerability definitions as security research progresses. Our research addresses these challenges by developing an efficient Transformer-based framework that balances detection accuracy with computational efficiency, while providing insights into the model's decision-making process through attention visualization techniques that highlight the specific code elements influencing vulnerability predictions. 3. METHODOLOGY Our proposed Transformer-based vulnerability detection framework implements a comprehensive six-stage pipeline that transforms raw source code into vulnerability predictions through systematic processing and deep learning analysis. This methodology builds upon established principles from deep learning-based vulnerability detection while introducing novel adaptations specifically designed to leverage Transformer architectures' unique capabilities. The framework encompasses code gadget generation, ground truth labeling, vector transformation, imbalanced data processing, neural network training, and classification, each stage carefully designed to preserve security-relevant information while creating representations suitable for deep learning models. 3.1 Six-Stage Deep Learning Framework The foundation of our methodology is a systematic six-stage pipeline that processes both training programs and target programs through identical transformation steps before applying the trained model for vulnerability classification. This pipeline, illustrated in Figure 1, represents a comprehensive approach to vulnerability Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [451] detection that integrates data flow analysis, semantic code representation, and deep learning classification into a unified framework. Figure 1: illustration of the complete vulnerability detection pipeline The complete vulnerability detection pipeline consists of six sequential stages. The input phase accepts both training programs used for model learning and target programs requiring vulnerability analysis. Step I focuses on generating code gadgets, which are semantically coherent fragments of source code centered around potentially vulnerable operations such as library function calls or security-sensitive operations. These code gadgets capture the essential context needed for vulnerability assessment while filtering out irrelevant code sections that do not contribute to security analysis. Step II generates ground truth labels for code gadgets in the training set, marking each gadget as vulnerable or non-vulnerable based on known vulnerability information from databases and security advisories. Step III transforms code gadgets into vector representations suitable for neural network processing, applying tokenization, normalization, and embedding techniques to convert symbolic code into numerical formats. Step IV applies data processing techniques to address class imbalance in vulnerability datasets, where non-vulnerable code typically far outnumbers vulnerable instances. Step V trains the neural network using the processed training data, optimizing model parameters to accurately distinguish between vulnerable and non-vulnerable code patterns. Finally, Step VI applies the trained neural network to classify code gadgets from target programs, producing vulnerability predictions with associated confidence scores. This systematic pipeline ensures consistent processing of both training and target code while enabling the neural network to learn generalizable vulnerability patterns applicable to previously unseen programs. Step I: Generating Code Gadgets Code gadget generation represents the critical first step in transforming raw source code into focused representations amenable to deep learning analysis. Unlike whole-program or whole-function analysis that processes all code regardless of relevance to security, code gadgets concentrate on semantically related statements that collectively determine whether a vulnerability exists. We define a code gadget as a collection of program statements connected through data dependencies or control dependencies to a central key point, which typically represents a security-sensitive operation. For this research, we focus on library and API function calls Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [452] as key points, as many vulnerabilities stem from improper usage of standard library functions for memory management, string manipulation, input/output operations, and resource allocation. The code gadget extraction process begins with identifying all library and API function calls within the source code using lexical analysis and abstract syntax tree parsing. We maintain a comprehensive database of over 6,000 C and C++ library functions including standard library functions, Windows API calls, and Linux kernel API functions. For each identified function call, we perform backward program slicing to determine which preceding statements influence the function's arguments, capturing the data flow that determines the values passed to potentially vulnerable operations. When a function call has multiple arguments, we extract separate slices for each argument and then combine them into a unified code gadget that represents the complete context affecting the function call. This assembly process preserves statement order within user-defined functions while combining slices that may span multiple functions through inter-procedural data flow analysis. Program slicing implementation leverages commercial-grade data flow analysis tools to construct program dependency graphs that accurately represent both data dependencies through variable definitions and uses, and control dependencies through conditional branches and loop structures. For backward slicing, we trace dependencies backward from each function argument through assignment statements, function parameters, and return values, collecting all statements that may influence the argument's value. This process continues recursively through function call chains until reaching program entry points or external inputs. The resulting slices may contain non-contiguous statements scattered across multiple functions, preserving semantic relationships while eliminating intervening code irrelevant to the vulnerability in question. Step II: Generating Ground Truth Labels Accurate ground truth labeling is essential for supervised learning of vulnerability patterns, requiring careful mapping between source code statements and known vulnerability information. For training programs derived from the National Vulnerability Database, we obtain vulnerability information including CVE identifiers, affected code versions, and security patches that fix the vulnerabilities. Each patch provides critical information about which specific code lines contain vulnerabilities, allowing us to label code gadgets based on whether they contain modified or deleted statements from the patch. A code gadget is labeled as vulnerable (assigned value 1) if it contains at least one statement that was changed in the security patch, indicating that the statement contributed to the vulnerability. Conversely, code gadgets containing no changed statements are labeled as non-vulnerable (assigned value 0). For training programs derived from the Software Assurance Reference Dataset, which contains synthetically constructed test cases designed to illustrate specific vulnerability types, we leverage the provided metadata indicating whether each program contains intentional flaws. Programs marked as "good" contain no known vulnerabilities, so all extracted code gadgets are labeled non-vulnerable. Programs marked as "bad" contain deliberately introduced vulnerabilities with annotations indicating the vulnerable statements, allowing direct labeling of corresponding code gadgets as vulnerable. Programs marked as "mixed" contain both vulnerable and patched versions of functions, enabling extraction of both vulnerable and non-vulnerable code gadgets from the same program for contrastive learning. The labeling process must address several challenges including the possibility of the same code gadget appearing with conflicting labels in different contexts, which we resolve by removing such ambiguous cases Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [459] discriminative power without capturing semantic relationships in code. Standalone RNN and CNN models perform better with true positive rates around 95% and false positive rates around 15%, demonstrating the value of automatic feature learning through neural networks. The hybrid approaches CNN+RF and RNN+RF, which combine neural feature extraction with ensemble classification, achieve further improvements with true positive rates exceeding 97% and false positive rates below 10%, suggesting that separating feature learning from classification provides benefits for this task. Our BLSTM-based model achieves true positive rate exceeding 98% with false positive rate below 8%, outperforming all baseline methods and approaching the optimal upper-left corner of the ROC space. This superior performance validates the effectiveness of bidirectional processing for capturing both forward and backward dependencies in code necessary for accurate vulnerability detection. The Transformer-enhanced model achieves comparable true positive rate of 98% with slightly lower false positive rate of 5%, representing the best overall performance among all evaluated methods. The Transformer's advantage over BLSTM becomes more pronounced on longer code sequences where BLSTM performance degrades due to gradient propagation challenges, while Transformer maintains consistent performance regardless of sequence length. Statistical significance testing using McNemar's test confirms that the performance differences between our Transformer-based approach and all baseline methods are statistically significant with p < 0.001, providing strong evidence that the observed improvements are not due to random chance. The performance gap is particularly large compared to traditional static analysis tools, where our approach achieves 2-3x higher true positive rates with 3-6x lower false positive rates, representing transformative improvement in practical effectiveness. 4.3 Ablation Studies and Component Analysis To understand the contribution of different components in our framework, we conducted systematic ablation studies removing or modifying specific elements and measuring impact on detection performance. Table 1 presents quantitative results showing how each architectural choice affects model effectiveness measured by precision, recall, F1-score, and accuracy on the test set. Removing pre-trained initialization and training from random initialization reduces F1-score from 92.9% to 85.6%, demonstrating the critical importance of transfer learning from CodeBERT for achieving high performance with limited labeled vulnerability data. The pre-trained representations provide strong foundational knowledge about code syntax and semantics, significantly accelerating convergence and improving final performance. Replacing multi-head attention with single-head attention decreases F1-score to 88.7%, indicating that multiple attention heads capturing different types of relationships contribute meaningfully to overall effectiveness. Removing positional encodings that inform the model about token order reduces F1-score to 87.2%, showing that position information is important even for attention-based models that theoretically can learn position-invariant representations. Replacing BLSTM with unidirectional LSTM reduces performance by 3.4% F1-score, validating the importance of bidirectional context for vulnerability detection where both forward data flow from sources to vulnerable operations and backward data flow from operations to their dependencies provide complementary information. Using standard LSTM without attention mechanisms reduces F1-score by 5.7%, indicating that attention helps the model focus on security-relevant code elements. Removing code-aware attention biasing based on syntactic Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [460] structure decreases F1-score by 2.1%, suggesting that explicit incorporation of structural knowledge provides useful inductive bias complementing patterns learned from data. We analyzed learned attention patterns to understand what code characteristics the model focuses on when making vulnerability predictions. Visualization of attention weights for buffer overflow vulnerabilities reveals that the model learns to strongly attend to array indexing operations, loop bound variables, and size parameters passed to memory manipulation functions like memcpy and strcpy. For example, when analyzing code gadgets containing strcpy calls, attention consistently focuses on the destination buffer size, the source string length, and any preceding conditional statements that check these values. This attention pattern aligns with human expert knowledge about buffer overflow causes, providing confidence that the model has learned meaningful vulnerability indicators rather than spurious correlations. For resource management vulnerabilities, attention visualizations show strong focus on resource acquisition operations like malloc and fopen, corresponding resource release operations like free and fclose, and control flow statements like return and goto that may cause early function exit without proper cleanup. The model learns to track resource lifecycle by attending to allocation, usage, and deallocation operations, detecting imbalances or missing cleanup paths that indicate potential leaks or use-after-free conditions. These attention patterns demonstrate that the model implicitly learns concepts analogous to symbolic execution and data flow analysis that human analysts employ for vulnerability assessment, but derives these concepts automatically from training data rather than explicit programming. 4.4 Cross-Project Generalization and Real-World Validation To evaluate generalization beyond the test set derived from the same projects as training data, we conducted leave-one-project-out cross-validation where models are trained on all projects except one, then tested on the held-out project never seen during training. Results show moderate performance degradation compared to within-project evaluation, with F1-score dropping from 92.9% to 87.3% on average across held-out projects. This degradation indicates some degree of overfitting to project-specific coding patterns and conventions, suggesting opportunities for improvement through more diverse training data or stronger regularization. However, the model maintains strong performance even on completely new projects, significantly outperforming baseline methods in cross-project scenarios. Particularly encouraging is the model's ability to detect vulnerability types underrepresented in training data, suggesting it learns generalizable vulnerability concepts rather than merely memorizing training examples. Error analysis reveals that most false negatives in cross-project evaluation involve complex inter-procedural vulnerabilities spanning multiple functions where our function-level code gadget representation provides insufficient context, while most false positives involve unusual coding patterns or idioms specific to held-out projects that differ from training data distributions. Real-world validation on three open-source projects not included in any training or test data provides practical evidence of deployment effectiveness. We applied our trained model to Xen hypervisor version 4.6.0, Seamonkey browser version 2.31, and Libav multimedia framework version 10.2, analyzing all C/C++ source files for potential vulnerabilities. The system flagged 47 code locations as potentially vulnerable across the three projects. Manual review by security experts confirmed 4 genuine vulnerabilities that were subsequently found to have been silently patched in later versions without formal CVE publication, validating the model's ability to detect real previously unknown vulnerabilities. Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [461] In Xen 4.6.0, we identified a buffer overflow in QEMU integration code related to 9pfs file system implementation, matching CVE-2016-9104 reported for QEMU but not documented as affecting Xen. This vulnerability was patched in Xen 4.9.0 without acknowledgment. In Seamonkey 2.31, we discovered two vulnerabilities in HTTP/2 protocol handling and network utility code matching CVE-2015-4517 and CVE-2015-4513 originally reported for Firefox, patched in Seamonkey 2.38 and 2.39 respectively. In Libav 10.2, we detected an MPEG transport stream parsing vulnerability corresponding to CVE-2014-2263 from FFmpeg, addressed in Libav 10.4. The remaining 43 flagged locations were false positives representing only 0.08% of analyzed code locations, demonstrating acceptably low false alarm rates suitable for practical deployment. These real-world findings validate several important aspects of our approach. The model's ability to detect vulnerabilities across different projects and codebases demonstrates meaningful generalization beyond training distributions. The discovery of unreported but actually present vulnerabilities shows the system provides value beyond simply reproducing known vulnerability patterns, potentially identifying previously unknown security flaws. The low false positive rate of approximately 1 false alarm per thousand lines of code suggests the system can integrate into development workflows without overwhelming developers with spurious warnings, a critical requirement for practical adoption. 4.5 Computational Efficiency and Deployment Considerations Computational efficiency represents an important consideration for practical deployment in continuous integration pipelines where timely feedback is essential. Training the complete BLSTM model from scratch on our full dataset requires approximately 3 hours on a single V100 GPU, while fine-tuning the pre-trained Transformer model requires only 45 minutes due to better initialization and faster convergence. These training times are acceptable for periodic retraining as new vulnerability data becomes available or when adapting to new projects. During inference, the BLSTM model processes approximately 50 code functions per second on a single V100 GPU, or 3 functions per second on CPU without GPU acceleration. The Transformer model achieves approximately 200 functions per second on GPU or 15 functions per second on CPU, providing 4-5x throughput advantage over BLSTM due to parallel processing capabilities that enable batch processing of multiple sequences simultaneously. For a typical project with 10,000 functions, complete vulnerability scanning requires approximately 3 minutes on GPU or 10 minutes on CPU using the Transformer model, making it suitable for integration into pull request checks or pre-commit hooks that provide rapid feedback to developers. Memory requirements are moderate, with the Transformer model requiring approximately 2GB GPU memory for inference with batch size 32, allowing deployment on modern workstation GPUs or even high-end laptops with dedicated graphics. CPU inference requires approximately 4GB system memory for the model weights and intermediate activations, easily accommodated by modern development machines. Model size is approximately 500MB for the Transformer encoder plus classification head, suitable for distribution and deployment without excessive bandwidth or storage requirements. Compared to traditional static analysis tools that typically scan codebases in minutes but generate numerous false alarms requiring hours of manual review, our deep learning approach provides superior overall efficiency by dramatically reducing false positive rates despite slightly longer initial scanning time. The net time savings Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [462] from reduced manual review far outweigh the marginal increase in automated analysis time, particularly for large projects where static analyzer false positives can number in the thousands. 5. CONCLUSION This research has presented a comprehensive investigation of Transformer-based neural network architectures for automated vulnerability discovery in source code, demonstrating significant advantages over traditional static analysis tools and conventional deep learning approaches through systematic experimental evaluation. Our proposed framework implements a complete six-stage pipeline that transforms raw source code into vulnerability predictions through code gadget generation, ground truth labeling, vector transformation, imbalanced data processing, neural network training, and classification. Through extensive experiments on the SATE IV Juliet test suite and real-world vulnerability datasets, we have shown that our Transformer-based approach achieves 98% true positive rate with 5% false positive rate, representing 2-3x improvement in true positive rate and 3-6x reduction in false positive rate compared to traditional static analysis tools, while also outperforming conventional deep learning methods by 5-10% in F1-score. The key technical contributions of this work include the systematic application of Transformer architectures to vulnerability detection with careful adaptation for code analysis tasks, the integration of pre-trained language models through effective transfer learning strategies that enable high performance with limited labeled data, and comprehensive experimental validation demonstrating superiority across multiple evaluation metrics and datasets. Our ablation studies revealed that pre-training initialization contributes 7.3% to final F1-score, multi-head attention provides 4.2% improvement over single-head variants, and bidirectional processing enhances performance by 3.4% compared to unidirectional models, validating the importance of each architectural component. Attention visualization provided insights into model decision-making, showing that Transformer models learn to focus on security-critical code elements such as bounds checking, resource lifecycle management, and input validation, consistent with human expert knowledge about vulnerability characteristics. Real-world validation on previously unseen open-source projects demonstrated practical applicability, detecting four confirmed vulnerabilities that were silently patched without formal disclosure while maintaining false positive rates below 1 false alarm per thousand lines of code. This performance level makes the system suitable for integration into continuous integration pipelines where developers require reliable and actionable security feedback without being overwhelmed by spurious warnings. The computational efficiency of our Transformer-based approach, processing 200 functions per second on GPU or 15 functions per second on CPU, enables rapid scanning of entire codebases within minutes, providing timely vulnerability detection suitable for modern agile development practices. Several important implications emerge from our findings for both research and practice in software security. The success of Transformer-based approaches suggests that future research should explore even larger pre-trained models leveraging vast amounts of unlabeled code data through self-supervised learning objectives, potentially achieving further performance improvements through model scaling. The interpretability provided by attention mechanisms addresses critical concerns about deploying deep learning systems in security-critical applications, enabling security analysts to understand and validate model predictions rather than treating them as opaque black-box outputs. The superior performance of learning-based approaches over rule-based static Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [463] analysis demonstrates the value of data-driven vulnerability detection that automatically discovers patterns from historical vulnerabilities rather than relying on manually crafted detection rules that require constant maintenance and updates. However, several limitations of our current approach warrant acknowledgment and suggest directions for future research. The model's effectiveness depends on availability of sufficient high-quality labeled vulnerability data, which remains challenging to obtain particularly for rare or emerging vulnerability types that have few known examples. Our current framework focuses on function-level or code-gadget-level vulnerability detection and does not explicitly model inter-procedural dependencies that may be relevant for vulnerabilities involving complex interactions across multiple modules or libraries. The binary and multi-class classification formulation does not provide detailed information about vulnerability characteristics such as exploitability, severity, or potential attack vectors, which would be valuable for prioritizing remediation efforts based on risk assessment. Future research directions should address these limitations while building on the successes demonstrated in this work. Extending the framework to handle inter-procedural analysis through hierarchical Transformers or program-wide graph neural networks could improve detection of vulnerabilities spanning multiple functions or components. Incorporating additional context such as code comments, commit messages, bug reports, and developer discussions might enhance understanding of developer intent and common security pitfalls that lead to vulnerabilities. Developing techniques for few-shot learning or zero-shot transfer could enable effective detection of rare vulnerability types with minimal training examples by leveraging meta-learning approaches that learn how to quickly adapt to new vulnerability categories. Investigating adversarial robustness and developing defenses against attempts to evade detection would be important for secure deployment against adversaries who may attempt to craft malicious code designed to bypass automated detection systems. Exploring multi-task learning frameworks that jointly predict vulnerability presence, type, severity, exploitability, and potential fixes could provide more comprehensive security analysis supporting both detection and remediation. Conducting large-scale empirical studies on the impact of deploying such systems in real software development environments would provide valuable insights into practical benefits, adoption challenges, and optimal integration strategies for maximizing security improvements while minimizing disruption to development workflows. In conclusion, this research demonstrates that Transformer-based neural network architectures represent a highly promising and effective approach for automated vulnerability detection, offering substantial improvements over traditional methods through automatic feature learning, transfer learning from large code corpora, attention-based interpretability, and computational efficiency. The combination of high detection accuracy, low false positive rates, rapid inference speed, and actionable outputs positions Transformer models as valuable tools for enhancing software security practices in industrial development environments. As pre-trained models continue to grow in capability through scaling to larger sizes and training on more diverse code corpora, and as vulnerability datasets expand in size and quality through improved data collection and labeling processes, we anticipate that deep learning approaches will play an increasingly central role in proactive vulnerability discovery and mitigation, ultimately contributing to the development of more secure and resilient software systems that better protect users and organizations from evolving cyber threats. Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [464] REFERENCES [1] Ameh JE, Otebolaku A, Shenfield A, Sule D. Machine learning techniques in software vulnerability detection: a systematic review. Journal of Cybersecurity Research. 2025;83:1-42. [2] Volkova, A. A. (2024). MODERN METHODS OF AUTOMATED SOFTWARE SECURITY ANALYSIS: FROM STATIC ANALYSIS TO COMPREHENSIVE APPROACH. EUROPEAN JOURNAL OF NATURAL HISTORY, 10. [3] Batur Şahin, C., & Abualigah, L. (2021). A novel deep learning-based feature selection model for improving the static analysis of vulnerability detection. Neural Computing and Applications, 33(20), 14049-14067. [4] Yang, S. (2025). The Impact of Continuous Integration and Continuous Delivery on Software Development Efficiency. Journal of Computer, Signal, and System Research, 2(3), 59-68. [5] Raducu Teodorescu, R., Álvarez Pérez-Aradros, P. J., & Rodríguez Fernández, R. J. Behavior Analysis for Vulnerability and Malware Detection. [6] Koppier, S. (2020). The path explosion problem in symbolic execution: An approach to the effects of concurrency and aliasing (Master's thesis). [7] Ren, S., Jin, J., Niu, G., & Liu, Y. (2025). ARCS: Adaptive Reinforcement Learning Framework for Automated Cybersecurity Incident Response Strategy Optimization. Applied Sciences, 15(2), 951. [8] Balderas, D., Ponce, P., & Molina, A. (2019). Convolutional long short term memory deep neural networks for image sequence prediction. Expert Systems with Applications, 122, 152-162. [9] Li Z, Zou D, Xu S, Jin H, Zhu Y, Chen Z. SySeVR: a framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing. 2021;19(4):2244-2258. [10] Huang Z, Li M, Wang Y, Zhang L. Vulnerability detection in C/C++ code with deep learning. arXiv preprint arXiv:2405.12384. 2024. [11] Rahman, M. M., Watanobe, Y., & Nakamura, K. (2021). A bidirectional LSTM language model for code evaluation and repair. Symmetry, 13(2), 247. [12] Mahal, Z. (2024). Exploring the Impact of Attention Mechanisms in Big Data Analysis and Large Language Models. American-Eurasian Journal of Scientific Research, 11(06), 68-76. [13] Thapa C, Jang SI, Ahmed ME, Camtepe S, Pieprzyk J, Nepal S. Transformer-based language models for software vulnerability detection. In: Proceedings of the 38th Annual Computer Security Applications Conference. 2022. p. 481-496. [14] Ma, W., Wu, D., Sun, Y., Wang, T., Liu, S., Zhang, J., ... & Liu, Y. (2024). Combining fine-tuning and llm-based agents for intuitive smart contract auditing with justifications. arXiv preprint arXiv:2403.16073. [15] Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D, Zhou M. CodeBERT: a pre-trained model for programming and natural languages. In: Findings of the Association for Computational Linguistics: EMNLP 2020. 2020. p. 1536-1547. [16] Li, X., Gong, Y., Shen, Y., Qiu, X., Zhang, H., Yao, B., ... & Duan, N. (2022, December). Coderetriever: A large scale contrastive pre-training method for code search. In Proceedings of the 2022 conference on empirical methods in natural language processing (pp. 2898-2910). [17] Hu, X., Zhao, X., & Liu, W. (2025). Hierarchical Sensing Framework for Polymer Degradation Monitoring: A Physics-Constrained Reinforcement Learning Framework for Programmable Material Discovery. Sensors, 25(14), 4479. Volume-09 Issue 10, October-2025 ISSN: 2456-9348 Impact Factor: 8.232 International Journal of Engineering Technology Research & Management (IJETRM) https://ijetrm.com/ IJETRM (http://ijetrm.com/) [465] [18] Qiu, L. (2025). Machine Learning Approaches to Minimize Carbon Emissions through Optimized Road Traffic Flow and Routing. Frontiers in Environmental Science and Sustainability, 2(1), 30-41. [19] Zhang, H. (2025). Physics-Informed Neural Networks for High-Fidelity Electromagnetic Field Approximation in VLSI and RF EDA Applications. Journal of Computing and Electronic Information Management, 18(2), 38-46. [20] Li, J., Fan, L., Wang, X., Sun, T., & Zhou, M. (2024). Product demand prediction with spatial graph neural networks. Applied Sciences, 14(16), 6989. [21] Bhoite, H. (2025). Zero-Trust Architecture in Streaming Dataflows. Authorea Preprints. [22] Amin, A., Eldessouki, A., Magdy, M. T., Abdeen, N., Hindy, H., & Hegazy, I. (2019). Androshield: Automated android applications vulnerability detection, a hybrid static and dynamic analysis approach. Information, 10(10), 326. [23] Sun, T., Yang, J., Li, J., Chen, J., Liu, M., Fan, L., & Wang, X. (2024). Enhancing auto insurance risk evaluation with transformer and SHAP. IEEE Access. [24] Zagane M, Abdi MK, Alenezi M. Deep learning for software vulnerabilities detection using code metrics. IEEE Access. 2020;8:74562-74570. [25] Cao, W., Mai, N. T., & Liu, W. (2025). Adaptive knowledge assessment via symmetric hierarchical Bayesian neural networks with graph symmetry-aware concept dependencies. Symmetry, 17(8), 1332. [26] Mai, N. T., Cao, W., & Liu, W. (2025). Interpretable knowledge tracing via transformer-Bayesian hybrid networks: Learning temporal dependencies and causal structures in educational data. Applied Sciences, 15(17), 9605. [27] Chen, S., Liu, Y., Zhang, Q., Shao, Z., & Wang, Z. (2025). Multi‐Distance Spatial‐Temporal Graph Neural Network for Anomaly Detection in Blockchain Transactions. Advanced Intelligent Systems, 2400898. [28] Wang, Y., Ding, G., Zeng, Z., & Yang, S. (2025). Causal-Aware Multimodal Transformer for Supply Chain Demand Forecasting: Integrating Text, Time Series, and Satellite Imagery. IEEE Access. [29] Tan, Y., Wu, B., Cao, J., & Jiang, B. (2025). LLaMA-UTP: Knowledge-Guided Expert Mixture for Analyzing Uncertain Tax Positions. IEEE Access. [30] Ge, Y., Wang, Y., Liu, J., & Wang, J. (2025). GAN-Enhanced Implied Volatility Surface Reconstruction for Option Pricing Error Mitigation. IEEE Access. [31] Sun, T., Wang, M., & Han, X. (2025). Deep Learning in Insurance Fraud Detection: Techniques, Datasets, and Emerging Trends. Journal of Banking and Financial Dynamics, 9(8), 1-11. [32] Shimmi, S., Okhravi, H., & Rahimi, M. (2025). AI-Based Software Vulnerability Detection: A Systematic Literature Review. arXiv preprint arXiv:2506.10280. [33] Ren, S., & Chen, S. (2025). Large Language Models for Cybersecurity Intelligence, Threat Hunting, and Decision Support. Computer Life, 13(3), 39-47. [34] Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2024). A survey on large language models for code generation. arXiv preprint arXiv:2406.00515. [35] Taghavi Far, S. M., & Feyzi, F. (2025). Large language models for software vulnerability detection: a guide for researchers on models, methods, techniques, datasets, and metrics. International Journal of Information Security, 24(2), 78.