Fundamentals of buffer overflow attacks and detection techniques

 Corresponding author: Bogdan Barchuk Copyright © 2025 Author(s) retain the copyright of this article. This article is published under the terms of the Creative Commons Attribution Liscense 4.0. Fundamentals of buffer overflow attacks and detection techniques Bogdan Barchuk * Independent researcher. World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 Publication history: Received on 31 January 2025; revised on 19 May 2025; accepted on 22 May 2025 Article DOI: https://doi.org/10.30574/wjarr.2025.26.2.1967 Abstract Buffer overflows remain a major security risk to software. The article presents the fundamentals of finding buffer overflow weaknesses and methods of analyzing them. Manual and automated fuzzing approaches allow the discovery of possible instances of the stack overflow attack. Immunity Debugger aids in discovering crashes, scrutinizing register clusters, and precisely determining the point at which buffer overflows occur within the memory of a program. Other approaches for identifying and eliminating such “badbytes” are also addressed. The study addresses approaches for bypassing the security measures implemented in contemporary OSs such as ASLR and DEP. This paper provides scripts and case studies allowing security experts and researchers to effectively locate, characterize, and remove far more vulnerabilities related to buffer overflows in many software systems. Keywords: Buffer Overflow; Fuzzing Techniques; Exploit Development; Debugger Analysis; ASLR Bypass; Shellcode Injection 1. Introduction A buffer overflow vulnerability is exploited when data written to a buffer overruns the buffer’s allocated memory, filling storage space outside the buffer. In either case, the machine might stop working or the intruder might secure comprehensive control over it by executing unwanted commands. Acquiring insights into buffer overflow scenarios and how they can be identified will enable you to develop robust code that is able to withstand vulnerability attacks. Tutorials and simulators are now available to teach users about buffer overflow attacks and how they function. The authors have designed an interactive tutorial that enables programmers to develop the ability to recognize and respond to such incidents. In addition, comprehensive investigations into buffer overflows in large applications have shown that this type of vulnerability is quite common and often appears in specific forms. A team of researchers examined numerous C/C++ projects and determined the origin as well as the main features of buffer overflows. The examination reveals that protecting large software projects from buffer overflows is extremely challenging and emphasizes the need for advanced ways and devices. It has been observed that a mix of educational materials and empirically-based studies plays an essential part in enhancing skills and innovation in protecting against buffer overflow assaults. 1.1. Overview A buffer overflow weakness occurs when data is copied into a memory space that's less than the number of bytes of memory that can be allocated. These weaknesses may result from inadequate checks on data entered by users or flawed World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2896 management of computer memory, enticing numerous attackers. Examples of methods used by hackers to exploit these types of vulnerabilities include inserting data through the command line, network connections, files loaded by the program or web forms and text fields. Understanding the origins of weaknesses enables you to identify and deal with them within software. The identification of flaws now relies largely on scrutiny of errors and the use of automated testing. Immunity Debugger enables you to monitor functions within a program, analyze its memory state and explore the reasons behind memory overflows in unstructured data. Examining the execution in detail enables security practitioners to identify the origin of an overflow and determine how it affects the management of the program. Fuzzing can help locate flaws by uncovering the effects caused by exposing software to a range of varied inputs. It ensures that various types of invalid inputs are injected into the system to identify potential problems. More advanced fuzzing approaches are enhancing the ability of experts to find bugs and weaknesses in computer software systems. They propose using a combination of static and dynamic techniques to increase the effectiveness of uncovering vulnerabilities in software applications. For instance, HotFuzz has been implemented to locate and reproduce issues like algorithmic denial of service using targeted micro-fuzzing methods. Combining debugging and fuzzing provides an effective technique for identifying and understanding security flaws related to buffer overflows in different software products. 1.2. Problem Statement Identifying potential buffer overflow flaws in current software systems can be very challenging. More sophisticated software makes it increasingly difficult to examine all the various forms of input used in the program’s source code. Advanced operating systems introduce randomization techniques for protection against buffer overflow exploits. Nonetheless, this development significantly complicates when determining buffer overflow flaws in applications. 1.3. Objectives This article describes methods for discovering and taking advantage of buffer overflow flaws. • Identifying the conditions that can make an input string susceptible to exploitation in different types of software. • Introducing the fuzzing process and tools commonly used for finding checking programs for vulnerabilities related to buffer overflows. • Steps provided for locating and examining the underpinnings of buffer overflow errors with Immunity Debugger. • Discussing means to bypass ASLR and DEP to increase consistency in locating and exploiting buffer overflow errors in application code. 1.4. Scope and Significance The paper discusses methods that can be employed to identify and capitalize on overflowing buffer flaws. By realizing this objective it helps those responsible for testing software and diagnosing and resolving similar problems in realworld scenarios. Carrying out outdoor activities involving the use of debuggers and fuzzers permits developing strategies that can be applied in various settings. The results provided contribute to enhancing an organization’s security by allowing it to identify and respond effectively to various kinds of cyber threats. 2. Literature review 2.1. Detection of Attack Vectors Identifying buffer overflow vulnerabilities involves determining all avenues through which an attacker offers data to a program. Attackers often use CLI, GUI and network connections to insert data that could potentially cause a buffer overflow. Understanding the features and transfers of such interfaces enables the identification of areas within them prone to overflow manipulation. Some software that utilizes a CLI interface accept a range of commands and parameters from users, some of which could lead to memory overflows. Many GUI interfaces contain various locations that could allow an adversary to inject data World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2897 that could cause buffer overflows. They highlight the significance of creating secure GUIs instead of insecure ones. In order to prevent security issues in a GUI, input validation and control is critical every time it is used. Using network interfaces in SDNs exposes the system to potential attacks since the continual communication depends on protocols. In SDNs, Latif et al. (2020) found that problems with the interface protocols often occur either due to using the protocols in the wrong way or by mistakenly handling inputs, resulting in buffer overflows. Identifying how data is transmitted or processed within the network protocols necessitates utilizing expert instruments. Tools such as Immunity Debugger and Mona help in analyzing a system’s behavior as different inputs are inputted. A variety of trace-based methods and comprehensive examination are applied by security professionals to locate potential locations where an overflow can take place. 2.2. Fuzzing the Target Fuzzing is a fundamental technique used in identifying buffer overflow vulnerabilities by sending unexpected or malformed inputs to a program to trigger abnormal behavior, such as crashes or memory corruption. The core concept involves systematically injecting payloads of increasing size or complexity, often composed of repeating characters like “AAAA,” to observe how the target software responds. A successful fuzzing attempt is typically indicated by an access violation or crash in the target program, which can be analyzed using debugging tools such as Immunity Debugger. In the screenshot provided (Image 1), the program vulnserver.exe is shown crashing with an access violation error. The critical clue here is the presence of 41414141 in several CPU registers, most notably in the Instruction Pointer (EIP) register. The hexadecimal value 41414141 corresponds to the ASCII characters “AAAA,” confirming that the input sent during fuzzing has overwritten the EIP. This register controls the flow of execution in the program, and its overwrite signifies a successful buffer overflow, where the attacker gains control over the execution path. Figure 1 Access violation crash in Immunity Debugger showing 41414141 (ASCII "AAAA") overwritten in the Instruction Pointer (EIP), indicating successful buffer overflow Fuzzing can be performed manually by incrementally increasing input sizes and monitoring program behavior. However, automation using scripts can expedite the process by sending batches of payloads with systematically varied lengths or contents. The key is to carefully observe when the program crashes and correlate the input size with the point of failure. World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2898 The evidence from Immunity Debugger, such as the overwritten registers and the access violation, provides invaluable insights. It not only confirms the vulnerability but also assists in pinpointing the exact location within the buffer where overflow occurs. This data is essential for crafting precise exploit payloads and further vulnerability analysis. In summary, fuzzing combined with debugger analysis, as demonstrated in Image 1, forms a critical foundation in detecting and exploiting buffer overflow vulnerabilities, enabling researchers to identify weak points and develop effective mitigation strategies. 2.3. Automating Fuzzing with Scripts Automating fuzzing is a vital step in efficiently identifying buffer overflow vulnerabilities, especially when testing complex software or network services. Manual fuzzing, while useful for initial discovery, can be time-consuming and error-prone. The use of scripting languages like Python enables security researchers to systematically generate and send payloads of varying sizes to target applications, thereby accelerating the vulnerability discovery process. The provided Python script (Image 2) demonstrates an effective approach to automating fuzzing against a network service, specifically the vulnerable vulnserver. The script begins by initializing a buffer array with a single "A" character and sets a counter at 100. Within a loop, it appends increasingly larger strings of "A"s—starting at 100 bytes and incrementing by 200 bytes each iteration—to the buffer until the list contains 30 payloads of increasing length. This approach ensures broad coverage of input sizes, which is crucial for uncovering the exact point at which the buffer overflow occurs. Using Python's socket library, the script establishes a TCP connection to the target IP address (192.168.15.230) on port 9999, which corresponds to the vulnerable service. For each payload string in the buffer, the script sends a command formatted as 'TRUN /.:/' concatenated with the fuzz string. The use of this command is specific to vulnserver's command structure, which processes input following TRUN /.:/. After sending the payload, the script closes the socket and proceeds to the next iteration. This automation allows for rapid testing of a wide range of inputs while monitoring the target's behavior, such as crashes or anomalies, which are indicative of potential vulnerabilities. Integrating automated fuzzing with debugging tools like Immunity Debugger enhances the ability to detect, analyze, and exploit buffer overflow conditions effectively. In conclusion, scripting fuzzing routines as shown provides a scalable, repeatable, and precise method for discovering overflow vulnerabilities, making it an indispensable tool in modern penetration testing and vulnerability research. Figure 2 Python script automating fuzzing by sending increasing payload sizes of “A” characters to the vulnerable network service vulnserver World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2899 2.4. Detection of Exact Overflow Position Identifying the precise location within a buffer where an overflow occurs is crucial for developing effective exploits and reliable detection techniques. A common and powerful approach involves using unique, non-repeating pattern strings generated by tools such as Metasploit’s pattern_create.rb. These patterns ensure that each substring within the input is distinct, enabling security analysts to accurately pinpoint the offset where control of program execution is gained. The Python script shown in Image 3 exemplifies this method by sending a carefully crafted pattern string to the target application. Instead of using repetitive characters like “A” or “B,” this script transmits a sequence that uniquely identifies each byte’s position within the buffer. This is achieved by incorporating the pattern—generated externally—into the payload sent through the socket to the vulnerable service, here accessed at IP address 192.168.15.230 on port 9999. The script’s try-except block ensures graceful handling of connection errors while delivering the test string. Upon sending this pattern, the program is expected to crash if an overflow exists, and the instruction pointer (EIP) register in the debugger will contain a value extracted from the pattern. By using tools such as pattern_offset.rb, security researchers can input the overwritten EIP value to calculate the exact position within the input where the overflow occurred. This offset is pivotal for subsequent exploit development, allowing precise overwriting of critical control structures like return addresses. Debugger tools like Immunity Debugger provide a real-time environment to monitor this process, revealing valuable information such as register states, memory dumps, and crash logs. The combination of pattern-based fuzzing and detailed debugging forms the backbone of exact overflow position detection, enabling researchers to transition from vulnerability discovery to exploitation with accuracy and confidence. Figure 3 Python script sending a unique pattern string to identify the exact offset in the input buffer where the overflow overwrites the EIP 2.5. Register Overwrite Validation After determining the exact offset where the buffer overflow occurs, it is essential to verify that critical registers, especially the Instruction Pointer (EIP), can be successfully overwritten. This validation step confirms control over program execution and is fundamental for exploit development. The Python script shown in Image 4 exemplifies this technique by sending a payload composed of a series of 2003 “A” characters followed by 4 “B” characters. The choice of “B” is deliberate; it corresponds to the hexadecimal value 0x42424242, making it easily recognizable in the debugger. By crafting the payload in this manner, security analysts can observe if the EIP register is overwritten with 42424242, confirming that the exact overwrite location has been pinpointed. World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2900 Using socket programming, the script establishes a TCP connection to the target service (192.168.15.230 on port 9999), then sends the carefully constructed payload as part of the TRUN /.:/ command. The try-except structure ensures that any connection failures are gracefully handled, prompting the user to check the debugger status. When this payload is executed, the debugger—such as Immunity Debugger—should display the EIP register containing the “BBBB” pattern. This outcome proves that the overflow precisely controls the program’s execution flow. If the register contains different values or the program does not crash, the offset calculation requires re-evaluation. This validation step is crucial because controlling EIP enables the attacker to redirect execution to malicious shellcode or other payloads. Additionally, it helps security researchers understand the program’s memory layout and guides further stages of exploit crafting, such as finding suitable jump instructions and bypassing protections like ASLR and DEP. In conclusion, register overwrite validation using controlled patterns is a cornerstone of buffer overflow exploitation, bridging the gap between vulnerability detection and practical exploitation. Figure 4 Python script sending a payload with 2003 “A” characters followed by 4 “B” characters (0x42424242) to validate control over the EIP register 2.6. Detection of Bad Characters Once control of the Instruction Pointer is verified, the next critical step is to discover which byte values the target application will corrupt, truncate, or transform during transit from input to memory. These so-called bad characters are fatal to reliable shellcode execution: if a payload contains a byte that the application replaces with a NULL (0x00), converts, or strips entirely, the shellcode will mis-align and crash before it achieves code-execution. The Python script in Image 5 illustrates a systematic bad-character test. After reproducing the confirmed offset ("A"*2003) and EIP overwrite placeholder ("B"*4), the script appends a sequence named badchars that enumerates virtually every byte from 0x01 through 0xFF. (Notice that 0x00—universally regarded as a terminator in C-style strings—has been intentionally omitted.) Because each byte appears exactly once and in ascending order, any deviation observed inside the debugger directly identifies which values the program cannot handle. World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2901 Using Python’s socket library, the script connects to 192.168.15.230 on port 9999 and delivers the composite payload as part of the TRUN /.:/ command. On receipt, the vulnerable service processes the data and—assuming the overflow is still reachable—crashes. Inside Immunity Debugger, the analyst then inspects the memory region just beyond the overwritten EIP. If the byte pattern reads smoothly from 0x01 to 0xFF, no additional bad characters exist; however, any missing, duplicated, or altered value pinpoints a byte that must be excluded when generating final shellcode. The methodology is iterative: remove the offending byte(s) from the badchars string, rerun the script, and compare the new memory dump until all problematic values are catalogued. Only after the complete “good-byte” list is known should msfvenom or a similar encoder be invoked to craft shellcode with the -b flag, explicitly excluding each discovered bad character. Incorporating rigorous bad-character testing early in exploit development prevents late-stage payload failures, ensuring that subsequent steps—such as locating a JMP ESP address or chaining ROP gadgets to bypass DEP—operate on stable, predictable bytecode. Thus, bad-character detection acts as the quality-control gate between proof-of-concept overflow and a fully weaponized exploit. Figure 5 Python script appending a complete sequence of byte values (badchars) after the shellcode to detect problematic characters that may disrupt payload execution 2.7. ASLR and DEP Bypass Techniques Modern operating systems deploy Address Space Layout Randomization (ASLR) and Data Execution Prevention (DEP) to frustrate classic buffer-overflow exploitation. ASLR randomizes the base addresses of executable modules, thwarting World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2902 hard-coded jumps, while DEP marks stack and heap pages non-executable, preventing direct shellcode execution. To overcome these protections, attackers first search for modules that are loaded without ASLR and are not compiled with DEP—for example, legacy DLLs shipped with third-party software. In Immunity Debugger, the Mona plug-in simplifies this task with commands such as !mona modules, highlighting libraries where all security flags are False. Once an unprotected module is located, the attacker hunts for an instruction that redirects execution to controlled data—commonly JMP ESP or CALL ESP. Mona’s !mona jmp -r esp enumerates such opcodes and records their fixed virtual addresses; placing one of these addresses (in little-endian form) over the overwritten EIP bypasses ASLR because the module’s base never changes. DEP still blocks code on the stack, so the exploit pivots to a Return-Oriented Programming (ROP) chain that calls VirtualProtect or NtProtectVirtualMemory to mark the shellcode region executable. Building these gadget chains requires harvesting short instruction sequences that end in a RET, a technique documented as the principal countermeasure target in contemporary control-flow integrity research (Tymburibá, 2020). A practical demonstration can be seen in the exploitation of the classic SL-Mail 5.5 overflow: researchers identified a non-ASLR, non-DEP SLMailSMTP.dll, inserted its JMP ESP address, chained gadgets to disable DEP, and finally executed reverse-shell payloads, validating that robust bypasses remain feasible when insecure libraries are present (Shafana & Pawar, 2021). In summary, bypassing ASLR and DEP revolves around three pillars: isolating unprotected modules, redirecting execution with stable pointers such as JMP ESP, and leveraging ROP to re-enable executable permissions or jump into already-executable memory. Mastery of debugger automation, opcode searches, and gadget cataloging transforms these ostensibly formidable defenses into surmountable hurdles for seasoned penetration testers. 2.8. Shellcode Generation and Injection With the crash point, offset, bad-character list, and bypass strategy confirmed, the final step is crafting and implanting executable payloads. msfvenom, part of the Metasploit Framework, is the de-facto utility for on-demand shellcode generation. By specifying the target platform (-p windows/shell_reverse_tcp), architecture (-a x86 or x64), local host and port (LHOST/LPORT), and an output format (-f python), testers obtain byte arrays ready for direct insertion into proof-of-concept scripts. The -e flag selects encoders—such as shikata_ga_nai—that polymorphically transform payloads to avoid intrusion-prevention signatures, while the -b option excludes discovered bad characters to guarantee reliable transmission. To cushion uncertainties in jump accuracy, exploits often prepend a NOP sled: a sequence of 0x90 bytes that functions like a conveyor belt, sliding execution safely into the shellcode even if the landing address is imprecise. Although as few as 16 NOPs can suffice, larger sleds (32–64 bytes) provide greater tolerance against slight variations introduced by ROP alignment or network encoding. Injection strategy depends on the bypass method. If DEP has been disabled via a ROP call to VirtualProtect, the exploit may simply jump straight to the NOP sled on the stack. Alternatively, some attacks locate already-executable memory—for instance, the .text section of a non-ASLR DLL—and copy the shellcode there, then direct EIP to that address. In Unicode or wide-character vulnerabilities, alphanumeric encoders like x86/alpha_mixed reshape the shellcode into acceptable byte patterns, later reassembling at runtime. Thorough testing remains essential. After embedding the generated shellcode and NOP sled into the final payload, repeated execution under the debugger verifies that registers, memory protections, and control flow behave exactly as scripted. Only after consistent, crash-free execution delivering a reverse shell—or alternate post-exploitation action— should the exploit be considered production-ready. Robust shellcode generation and disciplined injection practices thus complete the buffer-overflow exploitation lifecycle, transforming theoretical control of EIP into practical, dependable code execution on the target system. 3. Methodology 3.1. Research Design The researchers employed a systematic method to locate and use buffer overflow flaws. The first step involves identifying the source of inputs that may cause a buffer overflow in the target system. Automatically generated fuzzing inputs are utilized to inject huge or corrupt values into the system in order to trigger an overflow. After an overflow is identified, Immunity Debugger is utilized to trace the location where the instruction pointer is overwritten. The World Journal of Advanced Research and Reviews, 2025, 26(02), 2895-2908 2903 subsequent step involves designing unique payloads by leveraging the insights gained from the offset and the system’s address heap. Shellcode is then injected into the target process by implementing strategies that can overcome the security measures offered by ASLR and DEP. This approach ensures that every stage of the research is tested systematically, resulting in accurate and reliable outcomes when exploiting buffer overflows. 3.2. Data Collection The research focuses on collecting results from different aspects of buffer overflow analysis and exploitation. Fuzzing tools are used to generate various types of input and collect reports detailing the target system’s response to each input. The use of Immunity Debugger enables researchers to generate memory dumps, track the state of registers, and capture stack traces that show how and what areas of the system are impacted by the overflow. Furthermore, data about the effectiveness of payloads and errors encountered while scripting scripts is recorded during the testing process. This information plays a pivotal role in pinpointing the flaw, assessing its vulnerability, and improving the payload to exploit it consistently. Using this information helps provide an accurate analysis of the vulnerability and devise effective exploit techniques. 3.3. Case Studies/Examples 3.3.1. Case Study 1: The Test Web Application’s Vulnerability Was Exploited. An important task performed during a penetration test is determining and exploiting weaknesses in a web application to identify security vulnerabilities. The authentication mechanism implemented in a web application designed for teaching and testing purposes was revealed to include a serious buffer overflow vulnerability. This case study examines the process of identifying, investigating, and capitalizing on the vulnerability using traditional techniques for penet Discovery of the Vulnerability The application was examined as part of a regular security review and the flaw was identified. Users had to provide their login credentials by filling out a form and clicking the submit button to send them to the server. The program didn't adequately check and clean up the length of input, which is a weakness often found in many vulnerable applications. The username field did not limit input length, which meant that unchecked elements larger than the allocated memory space could be entered by users. Fuzzing was used to inject inputs that would cause the application to malfunction or crash. INPUT OVERFLOWING CAUSED THE APPLICATION TO FAIL IN VARIOUS WAYS AND OFTEN CRASHED COMPLETELY. This demonstrated that the application had a high chance of being exploited using a buffer overflow attack. Exploiting the Overflow The security team then focused on identifying the specific spot inside the buffer where the overflow occurs and the ways in which it influenced the application’s memory. They used Immunity Debugger to monitor the application’s responses when it was fed an excessive number of values. Immunity Debugger showed them that the EIP register was modified by injecting more data than the buffer could handle. The modified EIP pointed to the application being diverted from its intended course of operation. This indicated that the input was corrupting the buffer as well as taking control of the program’s flow of instructions, characteristic of a buffer overflow bug. They created a distinctive string of characters using Metasploit’s pattern_create.rb that would reveal the precise address at which the buffer overflow was occurring. The team prepared the payload by including various patterns and used the location of the crashed EIP to determine the correct offset. This gave them the information they needed to control the EIP in the correct way. Crafting the Exploit Armed with the offset, the team was able to create the appropriate payload. The attacker settled on modifying the EIP to jump to the shellcode on the stack. As a result, injecting the malicious code into the EIP enabled the attacker to obtain control over the server.