Threat Landscape in AI Systems

978-93-7143-144-6 390 Chapter -24 Threat Landscape in AI Systems Mr.Vishnu Kiran Bollu Sr. Security Administrator (Department of IT) OrganisationOC Tanner Salt Lake City, Utah, USA Email-id- [email protected] Abstract:-At an unprecedented level security concerns have been brought about by the quick spread of artificial intelligence (AI) systems in critical infrastructure, healthcare, finance, and autonomous systems. This chapter offers a thorough analysis of the changing threat landscape that AI systems must contend with, including adversarial assaults, data poisoning, model extraction, privacy violations, and new vulnerabilities in generative artificial intelligence and huge language models. We examine a classification of risks unique to AI, their possible effects on society trust and system integrity, and the sophisticated attack vectors that take use of the special features of neural network algorithms. The chapter provides a framework for comprehending the convergence of conventional cybersecurity risks with Intelligent-specific vulnerabilities, summarises recent research on adversarial machine learning, and examines real-world case studies of AI system hacks.We also look at the difficulties caused by deep neural network models' opacity, the dangers of supply chain attacks on AI systems, and the new dangers offered by AI-powered offensive capabilities. In the quickly emerging field of AI safeguards, this thorough study Threat Landscape in AI Systems 391 is the basis for creating effective defence plans and guiding policy choices. Keywords: Adversarial Machine Learning, AI Security, Data Poisoning, Model Extraction, Neural Network Attacks, Privacy Preservation, Threat Modeling, Deep Learning Vulnerabilities, AI Safety, Cybersecurity Introduction:-Artificial intelligence has evolved from a topic of study to an essential part of contemporary technology. Critical judgements in financial trading, driverless cars, criminal justice, healthcare diagnostics, and national security are now made by AI systems. Because of this widespread deployment, enemies now have a larger attack surface to exploit, which could have disastrous results. AI systems encounter a distinct set of risks that take use of the basic mathematical characteristics of machine learning techniques, in contrast to traditional software applications where flaws are mainly caused by implementation problems. Attacks against training data, model designs, inference procedures, and deployment environments are all part of the multifaceted danger landscape for AI systems. Through careful queries, adversaries can retrieve proprietary models, modify input data to create misclassification, poison training datasets to introduce backdoors, or take advantage of privacy flaws to recover critical training data. These attacks are appealing to a variety of threat actors, from lone hackers to state-sponsored adversaries, because they frequently involve few assets and tend to be carried out remotely. There are significant societal and economic stakes. According to a 2024 study, security issues involving AI might cost the world economy more than $200 billion a year by 2030. High-profile Mr.Vishnu Kiran Bollu 392 incidents have already shown how vulnerable AI systems are: adversarial examples have tricked autonomous car perception systems, carefully constructed perturbations have compromised facial recognition systems, and large computational models of language have been compromised compromised to produce malicious content or reveal personal details. These occurrences highlight how urgently thorough threat analysis and strong defences are needed. This chapter offers a methodical analysis of the threats that AI systems must contend with. Criminal assaults on model integrity, integrity of data threats such as poisoning and contamination, privacy and confidentiality violations, supply chain or deployment vulnerabilities, and new risks from machine learning and large models of languages are the five main threat classes around which we organise our analysis. We examine the technical mechanisms, practical applications, and detecting difficulties for each category. The chapter seeks to give academics, politicians, security professionals, and artificial intelligence developers a thorough grasp of the dangers associated with AI systems as well as the fundamental information required to create effective mitigation strategies. AI Threat Taxonomy and Classification:-A methodical classification framework that takes into account the many attack vectors, goals, and capabilities of possible adversaries is necessary to comprehend the threat landscape. The step of the AI lifecycle that has been selected, the opposition's comprehending and capabilities, the attack's goals, and the extent of impact are all included in this portion's thorough catalogue of AI hazards. Threat Landscape in AI Systems 393 Attack Surface Classification-The lifespan of an AI system offers several attack surfaces, each with unique susceptibility traits. Adversaries may introduce harmful samples, tamper with labels, or interfere with feature engineering procedures during the data collecting and preparation stage. The training phase is susceptible to resource fatigue attacks that impede development, poisoning episodes that gradually modify model behaviour, and theft of intellectual property through unauthorised access to training infrastructure. Models are vulnerable to inference-time attacks during the deployment process, such as membership inference attacks, adversarial examples, and model inversion. Systems are vulnerable to model extraction, idea drift abuse, and feedback mechanism exploitation after implementation. Adversary Knowledge and Capability Model-Threat actors have a wide range of skills and expertise, which significantly affects the viability of attacks and defensive tactics. White-box adversaries can launch sophisticated assaults like gradient-based antagonistic example creation and accurate backdoor injection since they have full knowledge of the model architecture, parameters, and training data. Gray-box adversaries must rely on transfer attacks or surrogate models because their knowledge is incomplete, possibly restricted to the model design or training data distribution. Through query optimisation, model collection, and input modification approaches, white-box antagonists can still carry out successful attacks even when they are limited to querying the model and observing its outputs. Mr.Vishnu Kiran Bollu 394 Table 1: AI Threat Taxonomy Threat Category Attack Vector Impact Level Detection Difficulty Adversarial Examples Input Perturbation High Very High Data Poisoning Training Data Manipulation Critical High Model Extraction Query-Based Stealing Medium Medium Privacy Attacks Inference/Inversion High Very High Prompt Injection Malicious Instructions MediumHigh Medium Adversarial Attacks on Model Integrity:-One of the most researched and worrisome threat categories in AI security is adversarial attacks. Attacks like these create subtly altered inputs that result in significant misclassifications by taking advantage of the continuous, high-dimensional character of neural network input fields. A major barrier to using AI systems in security-critical industries is the prevalence of hostile cases, which expose basic flaws in deep learning frameworks' generalisation. Threat Landscape in AI Systems 395 Evasion Attacks and Adversarial Examples-Adversarial examples are inputs that are specifically designed to induce errors in machine learning models. An adversarial example in image classification could be an image that seems the same to human viewers but leads a classifier to incorrectly identify it with high confidence. For example, an automated vehicle's perceiving system may identify a stop sign with strategically positioned stickers as a speed restriction sign. These perturbations frequently take advantage of the linear character of neural network decision boundaries in high-dimensional environments and the theory's responsiveness to frequently occurring aspects. Adversarial example generation has progressed from basic gradientbased approaches to complex optimisation strategies. By appending the sign of the loss gradient to the input, the Fast Gradient Sign Method (FGSM) executes a one-step attack. Stronger attacks, such as the Projected Gradient Descent (PGD) technique, restrict perturbations to remain undetectable while iteratively refining them. Adversarial example generation is formulated by the CarliniWagner assault as an optimisation issue that ensures misclassification while minimising perturbation size. Transfer attacks show that adversarial instances designed for one model frequently deceive other models, allowing for successful black-box assaults, but these white-board attacks presume complete model visibility. Physical World Adversarial Attacks-Adversarial examples pose a threat not only for electronic disturbances but also to real-world attacks that can withstand camera capture, printing, and environmental changes. Adversarial patches the fact that can be printed and placed in real-world settings have been shown by Mr.Vishnu Kiran Bollu 396 researchers to consistently produce misclassifications. Adversarial patterns on eyeglass frames that led to facial recognition algorithms misidentifying people were one particularly troubling example. Similar to this, autonomous automobiles misclassified road signs due to hostile disturbances, which raised grave safety concerns. Transformation robustness, including changes in viewing angle, distance, lighting, and camera features, must be taken into consideration by mechanical adversarial attacks. In order to overcome this difficulty, Expectation over Transformation (EOT) approaches optimise perturbations so that they continue to be effective throughout a distribution of expected transformations. The practical ramifications are significant: attackers may use physical disturbances that seem harmless to human viewers but are very successful against AI systems to breach safety devices, control autonomous cars, or avoid monitoring devices. Backdoor and Trojan AttacksWhen models are trained to act normally on clean inputs but display attacker-specified behaviour when confronted with inputs containing an unseen trigger pattern, cyber assaults pose a particularly pernicious threat. Backdoors remain inside the model itself and are only triggered by the attacker's known trigger, in contrast to adversarial instances that call for persample perturbations. Because the model operates successfully on regular data from experiments, backdoors are very hard to find using standard assessments of models. Backdoor attacks can be implemented by introducing triggered instances labelled with the target class into a tiny portion of training data. On clean data, the model maintains high accuracy while learning to connect the trigger with the wrong class. Advanced Threat Landscape in AI Systems 397 backdoor assaults employ undetectable triggers, adaptive triggers that adjust to input parameters, or semantic triggers that take advantage of inherent picture qualities. Adversaries may insert backdoors covertly in situations involving independently training data, models that have already been or contracted training facilities, which increases the risk. Data Integrity Threats and Poisoning Attacks:- The behaviour of AI systems is largely determined by the integrity and quality of training data. By adding malicious samples or alterations to the training dataset, data poisoning attacks take advantage of this reliance in order to impair model performance or introduce particular vulnerabilities. In contrast to inference-driven attacks, toxin assaults take place during the training phase, which makes them especially dangerous because they can harm deployed models indefinitely and are difficult to identify using conventional testing techniques. Label Flipping and Corruption Attacks-In order to impair the efficiency of models or cause particular errors, tag flipping attacks entail changing the labels of training data. A straightforward yet powerful approach reduces overall accuracy by randomly flipping a portion of the labels, leading the model to learn false associations. In order to maximise impact and minimise detectability, more complex assaults carefully choose which samples to flip. For example, while only impacting a small number of data, flipping the labels of samples close to decision borders might greatly skew the learnt class boundary. The fatal poisoning rate, the degree of visibility of infected objects, and the model's learning process are some of the variables that Mr.Vishnu Kiran Bollu 398 determine the impact of label poisoning. Certain machine learning algorithms are more resilient to label noise than others. Even after attaining low training error on correctly labelled data, deep neural networks trained with cross-entropy loss may still fit incorrectly titled examples, making them especially vulnerable. Statistical outlier analysis, convincing learning techniques that identify definitely mislabeled products, and group techniques that uncover discrepancies perhaps brought on by contaminated data are examples of detection procedures. Feature Pollution and Input Manipulation-Instead of changing the identifying information of training samples, information pollution attacks alter the input features. These assaults are especially pertinent in situations like recommendation systems, spam filters, or collaborative learning settings when training data is gathered from unreliable sources or user inputs. In order to evade detection, an adversary may introduce samples with carefully constructed feature values that change the model's decisions margin in their favour while preserving label correctness. Poisoning attacks, in which attackers introduce emails with spamlike characteristics but ham labels, are a prominent example. This technique teaches the filter to mistakenly identify coming spam as valid emails. Attackers can add fictitious user profiles with carefully constructed rating patterns to recommendation systems in order to promote or demote particular things. Differentiating between adversarial manipulation and valid data flow shifts is a difficulty in feature pollute defence, especially when the attacker is aware of the learning method and may adjust their poisoning tactics suitably. Threat Landscape in AI Systems 405 attacks make it difficult to discern between trusted system prompts and unauthorised user input by taking advantage of the basic architecture of language models that process instructions and data within the same context window. Creating user queries that bypass system guidelines or safety rules is known as direct prompt injection. When the LLM processes external material, like internet pages or documents, that contains hidden commands, it uses indirect prompt injection. The term "jailbreaking" describes methods used to get beyond content restrictions and safety measures put in place in LLMs. Roleplaying scenarios, hypothetical framings, encoded instructions, and multi-turn attacks are just a few of the tactics used by adversaries to progressively move the conversation towards outcomes that violate policy. Because models trained to be helpful may prioritise user satisfaction above policy adherence when provided with carefully written prompts, jailbreaking's efficacy highlights the challenge of matching LLM behaviour with safety objectives. Threats from inaccurate data and Artificial Content-Highly realistic synthetic content, including text, images, audio, and video, may be produced at previously unheard-of scale and quality thanks to generative AI. Because it allows for sophisticated disinformation operations, identity impersonation, and public discourse manipulation, this capability poses serious risks to information integrity. Unreliable data produced by LLMs can be grammatically correct, stylistically logical, and contextually suitable, making it harder to tell it apart from literature written by humans. From readily identifiable artefacts, deepfakes and artificial medium have developed into incredibly lifelike fakes that can fool both Mr.Vishnu Kiran Bollu 406 automatic detection systems and human observers. AI-generated propaganda, automated sockpuppet accounts, and coordinated inauthentic behaviour are examples of systematic manipulation that goes beyond individual deceit. Defence procedures, including as digital provenance technologies, watermarking methods, media literacy education, and legislative frameworks for synthesis content disclosure, must address both technological detection issues and wider societal consequences. Evaluation of Impact and Quantification of Risk:- Technical and sociological factors must be taken into account when estimating the possible impact of AI security risks. Depending on the application domain of the AI system, the type of breach, and the adversary's goals, the outcomes of successful assaults can differ significantly. Healthcare apps, driverless cars, and critical infrastructure systems all have catastrophic failure scenarios where AI flaws could be fatal. The banking system may experience systemic turbulence or losses in dollars. The loss of trust in AI systems is a significant, hard-to-measure effect that could prevent the adoption of helpful AI in all fields. Analysis of Sector-Specific Threats-Depending on their AI applications and adversary objectives, different industries have different threat profiles. Healthcare AI systems that analyse patient data as well as medical imaging are vulnerable to availability attacks that disrupt vital care, adversarial instances that lead to incorrect diagnoses, and privacy assaults. Financial firms that use AI for trading, credit scoring, or recognising fraud are vulnerable to poisoning assaults that alter decision limits, adversarial evasion, and model extraction to reverse-engineer detection systems. While Threat Landscape in AI Systems 407 national security technologies face powerful state-sponsored threats seeking to breach intelligence systems or manipulate decision support tools, vehicular autonomy represent important threats where coordinated assaults involving perception algorithms could result in accidents. Systematic Vulnerability and Cascade Failures-Modern AI systems are interconnected, which increases the risk of cascading failures, in which a compromised system spreads to other dependent systems. A single point of vulnerability can impact multiple downstream applications due to shared pre-trained models, shared training datasets, and interconnected decision pipelines. Systemic risk is increased when AI capabilities are concentrated in a small number of foundation models since flaws in these systems affect all derivative applications. Analysis of failure modes interactions, redundancy mechanisms, and dependence chains throughout the ecosystem of artificial intelligence is necessary to comprehend to reduce systemic vulnerability. Strategies for Screening and Identification:-Regular surveillance throughout the whole AI cycle, from data collection to deployment, is necessary for the effective detection of AI-specific hazards. AI-specific detection techniques that can identify adversarial inputs, detect data poisoning, identify anomalous model behaviour, and flag potential privacy breaches must be added to traditional security monitoring tools. Differentiating between malicious actions and genuine anomalies or distribution adjustments that naturally happen across manufacturing facilities is a challenge. Identification of Runtime Anomalies-By analysing input properties, predictions patterns, and query patterns, runtime Mr.Vishnu Kiran Bollu 408 monitoring systems can identify adversarial inputs, model extracting attempts, as well as additional during inference attacks. Inputs that substantially depart from the predicted distribution are identified via statistical anomaly detection, which may reveal adversarial cases. Suspiciously high-confidence predictions on odd inputs are flagged via prediction confidence analysis. Analysing query patterns reveals methodical investigation that may indicate model collection or investigations. To prevent interfering with normal use while detecting real attacks, these systems of detection must strike a balance between sensitivity and false positive rates. Verification of Data Integrity-Verification procedures are needed at every stage of the data pipeline to protect the integrity of training data. In order to facilitate auditing and contaminant traceability, lineage tracking keeps track of data sources, changes, and lineage. By comparing incoming data to expected distributions, statistical consistency checks identify abnormalities that may point to poisoning. In collaborative learning contexts, unwanted modifications are detected by consensus procedures. Recovering from poisoning assaults can be aided by periodic retraining from clean data sources, but doing so necessitates keeping verified clean information and systems to recognise when retraining is required. Future Risk Development and New Difficulties:-As both offensive and defensive capabilities develop, the AI threat scenario keeps changing quickly. Researchers and practitioners will need to pay attention to a number of indicators that point to new danger vectors. As AI systems become more autonomous and make important decisions with fewer people's oversight, new attack surfaces are created. The availability of AI-powered tools makes it Threat Landscape in AI Systems 409 easier to carry out complex attacks. General intelligence, or AI, capabilities have the potential to create whole new security concerns that are not sufficiently addressed by existing frameworks. Expanded areas of attack and new danger vectors are presented by multimedia AI systems which analyse and produce text, pictures, audio, and video input. Cross-modal attacks may employ one modality to compromise another or take advantage of discrepancies between modalities. Cyber-physical security concerns arise when AI systems are integrated with physical infrastructure through robots and IoT devices, where digital threats have immediate physical repercussions. Current cryptographic safeguards for AI models and data may eventually be threatened by improvements in quantum computer science, necessitating the development of entanglementresistant security protocols. Recursive security concerns arise from the development of defensive AI systems that are capable of automatically identifying and responding to attacks, where adversaries target the defensive systems themselves. As a result, there is an increasing arms race between defence and offensive capabilities. Sustained research, information exchange among stakeholders, and adaptable security frameworks that can address new difficulties as AI capabilities continue to progress are necessary to comprehend these changing dynamics and foresee future dangers. Conclusion:-One of the most intricate and significant security issues of the contemporary technological era is the threat landscape that AI systems must contend with. The many facets of AI security risks have been methodically explored in this chapter, ranging from supply chain assaults weakening confidence to upcoming threats Mr.Vishnu Kiran Bollu 410 from generative AI capabilities, from adversarial attacks exploiting model weaknesses to privacy breaches damaging sensitive data. Risk profiles of AI systems differ from those of standard software systems due to their special features, such as their dependence on data-driven learning, dimensional judgement spaces, and frequently opaque hidden representations. This approach yields several important observations. First, existing cybersecurity procedures are insufficient to address AI security or to approach it as an afterthought. Machine learning algorithms' mathematical characteristics produce intrinsic weaknesses that need for specific defensive strategies. Second, single sites of failure might spread across numerous applications and domains due to the interconnectedness of contemporary AI systems. Third, as AI develops quickly, new capabilities and danger vectors are constantly introduced, necessitating the use of adaptive security frameworks rather than static ones. AI security requires immediate attention because to the societal and economic stakes. The potential repercussions of security lapses increase in direct proportion to the growing integration of AI systems in vital infrastructure, healthcare, banking, and autonomous systems. Adversarial attacks that are successful could result in financial fraud, medical misdiagnoses, accidents involving autonomous vehicles, or breach of national security systems. Sensitive personal information could be exposed on an unprecedented scale due to privacy breaches. Perhaps the most subtle effect is the decline in public confidence in AI systems, which could hinder the widespread adoption of positive AI. Threat Landscape in AI Systems 411 Coordinated actions across several dimensions are needed to address these dangers. Defensive capabilities, such as strong training protocols, adversarial detection mechanisms, privacy-preserving strategies, and extensive monitoring systems, must be advanced by technical solutions. But technical solutions are not enough on their own. For AI systems, policy frameworks must set security requirements, liability frameworks, and incident response procedures. Sharing threat intelligence, standardising security procedures, and creating security-focused artificial intelligent techniques all depend on industry cooperation. Programs for education and awareness must equip security experts and AI developers to recognise and handle these particular difficulties. There are several unresolved issues facing the research community. There are still unanswered fundamental problems regarding the viability of provably safe AI systems, the conflict between model capacity and security, and the durability of learning algorithms. Sustained research is needed to develop thorough threat modelling frameworks, automated security testing tools, and security-bydesign principles for AI systems. Developments in the rapidly developing field of artificial intelligence ("AI") security will determine whether or not AI systems can be used in highly volatile scenarios in a secure and dependable manner. References 1. Biggio, B., & Roli, F. (2018). Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84, 317-331. Mr.Vishnu Kiran Bollu 412 2. Carlini, N., & Wagner, D. (2017). Towards evaluating the robustness of neural networks. IEEE Symposium on Security and Privacy (SP), 39-57. 3. Chen, X., Liu, C., Li, B., Lu, K., & Song, D. (2017). Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526. 4. Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4), 211-407. 5. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., ... & Song, D. (2018). Robust physical-world attacks on deep learning visual classification. IEEE Conference on Computer Vision and Pattern Recognition, 1625-1634. 6. Fredrikson, M., Jha, S., & Ristenpart, T. (2015). Model inversion attacks that exploit confidence information and basic countermeasures. ACM SIGSAC Conference on Computer and Communications Security, 1322-1333. 7. Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. International Conference on Learning Representations. 8. Gu, T., Dolan-Gavitt, B., & Garg, S. (2017). BadNets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733. 9. Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018). Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. IEEE Symposium on Security and Privacy (SP), 19-35. 10. Kurakin, A., Goodfellow, I., & Bengio, S. (2017). Adversarial examples in the physical world. International Conference on Learning Representations Workshop. Threat Landscape in AI Systems 413 11. Liu, Y., Ma, S., Aafer, Y., Lee, W. C., Zhai, J., Wang, W., & Zhang, X. (2018). Trojaning attack on neural networks. Network and Distributed System Security Symposium. 12. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations. 13. Nasr, M., Shokri, R., & Houmansadr, A. (2019). Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. IEEE Symposium on Security and Privacy (SP), 739-753. 14. Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017). Practical black-box attacks against machine learning. ACM Asia Conference on Computer and Communications Security, 506-519. 15. Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017). Membership inference attacks against machine learning models. IEEE Symposium on Security and Privacy (SP), 318. 16. Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016). Stealing machine learning models via prediction APIs. USENIX Security Symposium, 601-618. 17. Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., & Zhao, B. Y. (2019). Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. IEEE Symposium on Security and Privacy (SP), 707-723. 18. Xu, W., Evans, D., & Qi, Y. (2018). Feature squeezing: Detecting adversarial examples in deep neural networks. Network and Distributed System Security Symposium. 19. Zhang, J., Gu, Z., Jang, J., Wu, H., Stoecklin, M. P., Huang, H., & Molloy, I. (2018). Protecting intellectual property of Mr.Vishnu Kiran Bollu 414 deep neural networks with watermarking. ACM Asia Conference on Computer and Communications Security, 159-172. 20. Zou, A., Wang, Z., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.