Effects of the Current AI Ecosystem on Future GÉANT Security

© GÉANT Association 28-10-2025 Effects of the Current AI Ecosystem on Future GÉANT Security Nature of Document: White Paper Dissemination Level: Public Authors: Scott Campbell, (GÉANT Association) Abstract This paper looks at the current state of the art around AI/LLM tooling, both in terms of how it works, as well as the risks and benefits inherent with its use. A survey of publicly available information around how these technologies have been applied to attack and defence is provided, as well as how the collected material can inform current and future GÉANT Security Policy. Current trends show we must continue to work on phishing-resistant MFA, bolster user education around highly realistic hostile communications, such as phishing, and have a clear and direct policy around decision making that takes into consideration the falsifiable nature of human communication. Given the current state of how AI is perceived by the technical community, it is critical to remember that AI is just another technology, and its evaluation should be done in terms of real cost, real risk, and real benefit. Effects of the Current AI Ecosystem on Future GEANT Security 28 October 2025 ii Contents 1 Introduction 3 2 Technology Overview 3 2.1 Turning a Sack of Words into a Text Prognosticator 4 2.2 Tuning the Text Guesser: Creating the Assistant 5 2.3 Adding More Moving Parts: Agents 5 3 Structural Issues with LLM and Agents 6 3.1 Agents and Stability 6 3.2 Agents and Attack Surface 7 4 Realistic Threats and Benefits Involving Security and LLM 8 4.1 Phishing, Vishing, and Other Social Engineering Attacks 8 4.2 Augmenting Attacker Malware 9 4.3 Application Vulnerability Discovery 9 4.4 AI Personal Assistants 10 4.5 What About Defence? 10 4.6 Looking Ahead 10 5 Mapping Threats and Benefits to GÉANT Security Policy 10 6 Conclusion 11 Glossary 12 References 12 Figures Figure 2.1: Representation of how an LLM might learn about and use an Agent tool. In this case a tool for doing math operations. The MCP protocol can be thought of as a generic web interaction in terms of its mechanics 6 Figure 3.1: Diagram of a simple LLM-Agent interaction. At each in/out circle, the possibility of hallucination is introduced to the workflow 7 Figure 3.2: Example of a more complex LLM-Agent interaction. Note: The possibility of hallucination is introduced with each in/out cycle. This introduces a multiplicative effect for uncertainty in the results 7 Effects of the Current AI Ecosystem on Future GEANT Security Document ID: 28 October 2025 3 1 Introduction This document addresses the question of how changes introduced by artificial intelligence (AI) will influence choices around security policies within the GÉANT organization. The term 'policy' here encompasses the whole security ecosystem rather than just the normal description of rules and procedures. The short-term influences that AI has had on Security coordination are not unique but are shared across most of the GÉANT Association’s organisational components. For example, Purchasing/Procurement, HR, or Software Engineering could all become the likely focus of an attacker if the attacker’s end goals are aligned with money, personal information, or technology. The Security ecosystem is representative of the sort of distributed organisation common within GÉANT (and its partners), having a blend of both technological and policy concerns. Because of these commonalties, we hope that more general conclusions can be taken from this document. AI is less a technical expression than a generalised marketing term [ 1 ]. In this paper, the term will be used interchangeably with Large Language Model (LLM), which is described in detail in Section 2. There are other related technologies based on the Generative Pre-trained Transformer (the GPT in ChatGPT), which will not be covered in detail here, but are mentioned for completeness and to dispel the notion that LLMs represent all of AI. Avoiding the transient, ever-changing promise of what could be, and separating out strategic actions and organisational needs is critical to understanding the influence of this technology. Articles and information referenced in this document avoid polarised, pro and con opinions, and focus on the content. Neither blindly accepting ideas, nor rejecting them outright will be critical in understanding the impact of LLM on security. That said, the burden of proof lies with the LLM community to show that there are sound, factual reasons to trust their exceptional claims. Ultimately any new technology needs to be understood before evaluation and analysis. For all choices there is always a cost, and usually a benefit. In the first part of this evaluation an overview of the basic elements of LLMs as well as inherent strengths and weaknesses are presented. The second part of the paper reviews the technology as it relates to security - specifically how the global community is using AI, and what influences this will have to GÉANT. The remainder of this document is broken out into the following sections: • Section 2 Technology Summary provides an overview of how large language models are created, including agents. • Section 3 Structural Issues with LLM/agents: Explore immutable characteristics of LLMs/agents that have significant relevance from a security perspective. • Section 4 Realistic Threats and Benefits Involving Security and LLM includes observed attacks and tooling grouped by attack type. • Section 5 Mapping Threats and Benefits to GÉANT Security Policy details how the previously described threats influence security policy. • Section 6 Concludes with a consolidated review of threats and responses. 2 Technology Overview While a detailed understanding of the technologies surrounding LLMs is not required to understand the repercussions of their use, a small amount of familiarity can go a long way in understanding their benefits, issues, and limitations. This section will provide some background to LLMs, as well as how agents have been introduced to address some fundamental limitations of LLMs. Effects of the Current AI Ecosystem on Future GEANT Security 28 October 2025 4 2.1 Turning a Sack of Words into a Text Prognosticator While there are some exceptionally good explanations about the inner working of LLMs, they tend to be a bit long and complex, which distracts from explainability and ease of understanding. Since explainability is the primary objective here, we will look at a vastly simpler model - namely the mechanics of data reduction - for insights and intuition into LLM behaviours. This idea has been borrowed from an article by Ted Chiang in the New Yorker magazine [ 2 ]. Here the initial building of a large language model can be explored by thinking about lossy compression. Additional helpful details from a Medium article by A. G. Elrod [ 3 ], which uses the same metaphor to explore the results of feeding the output from one LLM into another. In the Chiang article, we are provided with the somewhat arbitrary task of reducing the content of the Internet to a small searchable dataset. Imagine that you’re about to lose your access to the Internet forever. In preparation, you plan to create a compressed copy of all the text on the Web. Unfortunately, your private server has only one percent of the space needed; you can’t use a lossless compression algorithm if you want everything to fit. Instead, you write a lossy algorithm that identifies statistical regularities in the text and stores them in a specialized file format. Because you have virtually unlimited computational power, your algorithm can identify extraordinarily nuanced statistical regularities, which allows you to achieve the desired compression ratio. The only catch associated with this process is that, because the text has been so highly compressed, you can’t look for information by searching for an exact quote; you’ll never get an exact match, because the words aren’t what’s being stored. To solve this problem, you create an interface that accepts queries in the form of questions and responds with answers that convey the gist of what you have on your server. From this we can imagine why it requires so much time and energy to create an LLM: data reduction is costly! The replacement of the original data with an approximation also provides some insight into why hallucinations are an integral part of the way LLMs operate, rather than something that can be worked around or corrected. This analogy to lossy compression is not just a way to understand ChatGPT’s facility at repackaging information found on the Web by using different words. It’s also a way to understand the “hallucinations,” or nonsensical answers to factual questions. These hallucinations are compression artifacts. If a compression algorithm is designed to reconstruct text after ninetynine per cent of the original has been discarded, significant portions of the answer will have to be entirely fabricated. This analogy makes even more sense when we remember that a common technique used by lossy compression algorithms is interpolation—that is, estimating what’s missing by looking at what’s on either side of the gap. Again, the takeaway is not that LLMs use lossy compression, but the data reduction inherent in building language models results in content holes that will be filled in with the most likely collection of words. Hallucinations are a fundamental component of large language models, not something that can be optimised away. This analogy also works to explore an additional behaviour of LLMs. In the article by Elrod, they discuss what happens when you repeatedly feed a JPEG image into the lossy compression intrinsic to the image creation. Each iteration reduces the JPEG image quality till it is almost unrecognizable from the original. Similarly, taking the output of one LLM as the input to another reduces accuracy and increases hallucinations of the model, a situation called "Model Collapse" [ 4 ]. Specifically, Martin Briesch et al found that diversity degenerates and the learned distribution inevitably collapses to a single point with a speed proportional to the amount of synthetic data. Effects of the Current AI Ecosystem on Future GEANT Security 28 October 2025 5 At this point the output from the computational process is an un-tuned LLM called a base-model which is a raw mapping between syntactical elements. While this is really an amazing accomplishment, our model is still lacking in manners. 2.2 Tuning the Text Guesser: Creating the Assistant Once a user has generated the base-model by compressing a corpus of human writing, art, and leisure activity, the most difficult and energy consuming aspects of the process are complete. Unfortunately, there are two major limitations with base models [ 5 ]: • They cannot follow user instructions, as the pre-training data contains few instruction-response examples. • They tend to generate biased and adversarial content. To make the model’s responses seem more in line with human feedback and preferences, additional instruction tuning via (prompts, response) demonstration pairs and Reinforcement Learning from Human Feedback (RLHF) is done. This final modified model is called the Assistant Model. While the creation of the base model was done by analysing vast quantities of "internet documents", the Assistant Model is driven by a smaller number of highvalue, manually created question:answer pairs and other specific relationships. This secondary tuning is computationally cheaper to do than what is required for the base-model, and is the mechanism that is used when companies rebrand raw LLM engines for their specific use cases. When this step is complete, the result is a model whose idea relationships have been codified in the first step, and the interface to those relationships has been defined in the second step, resulting in something resembling a conversational AI, such as those provided by older OpenAI or Claude type models. 2.3 Adding More Moving Parts: Agents While the LLM described in Section 2.2 is interesting and sometimes useful to provide additional, on-demand information to users such as a summary block of text, it has some fundamental limitations. For example, it is awful at arithmetic: specific examples ("5+2=7") exist as lexical facts in symbolic relationships, but the general theory of numeric operations is not part of the set in terms of the basic ideas and logic - abstract ideas are not applied when identifying solutions. To combat this problem, more specialised tools called agents were developed. Agents address the inherent weaknesses of the base-model when it comes to abstract reasoning, long-term state, and web browsing, by providing specialised functions to the LLM that act on and correct these specific limitations. Agents, in a variety of forms have been in use for several years. Originally this was in the form of bespoke, integration-like, retrieval-augmented generation. [ 6 ] With the introduction of the Model Context Protocol (MCP), this process has been vastly simplified. Pre-built MCP servers based on commercial and open source services are available for almost drop-in use. The effort has been successful enough that the protocol has become an industry standard. [ 7 ] Mechanically, things can be described two parts: server and client. On the server (green in Figure 2.1), public metadata describes the tool name, detailed description, input/output schemas, and a function handler. This is shared using the MCP protocol. When the client checks in with the name/function, it responds with the additional metadata to ensure a consistent understanding of the agent. [ 8 ] Effects of the Current AI Ecosystem on Future GEANT Security 28 October 2025 6 Figure 2.1: Representation of how an LLM might learn about and use an Agent tool. In this case, a tool for doing math operations. The MCP protocol can be thought of as a generic web interaction in terms of its mechanics When the LLM (yellow) starts up, it looks over its list of agents, which are provided as a set of general descriptions (for example, “weather tools”). The LLM then queries each tool's associated server to see details such as a function name (for example, “forecast_by_city”), and what information is required (such as “city name”). When the LLM determines that a specific agent tool is needed (based on user input or task context), it creates a tool request in MCP, which includes function name and other required information. It then sends the MCP formatted request to the server over HTTP. Back on the tool server, the translation layer takes the LLM request (via MCP), maps this request to the corresponding function on the server, executes the function, and formats the result back to the agent via MCP. The ability for a LLM to outsource functionality to an external provider without excessive programming overhead has created tremendous growth with this technology. And like all things related to explosive growth, there are consequences, both good and bad. 3 Structural Issues with LLM and Agents In AI Agents in Action, author Michael Lanham noted several challenging behaviours while working with agents. Specifically, he provides this warning: While writing this book and working with and building agents over many hours, I have encountered several instances of agents going rogue with actions, from downloading files to writing and executing code when not intended, continually iterating from tool to tool, and even deleting files they shouldn’t have. Watching an agent emerge new behaviours using actions can be fun, but things can quickly go astray. [ 9 ] The addition of agents to the LLM ecosystem provides the ability to address novel tasks outside the scope of the original LLM training set. The LLM can use its ability to textually analyse a request ("Should I bring an umbrella to the market today?"), determine to appropriate tool (weather_tool), and request real time weather data for your area. This functionality greatly extends the usefulness of an otherwise static LLM model. Unfortunately, there are some unavoidable issues with agents. The first is behavioural: model responses become decreasingly predictable as the LLM and related agents become more complex. The second is structural: prompt driven actions represent a vastly larger attack surface as compared to a traditional API. 3.1 Agents and Stability In the simplest model, a perfectly deterministic agent is queried by an LLM, as illustrated in Figure 3.1, below. Effects of the Current AI Ecosystem on Future GEANT Security 28 October 2025 7 Figure 3.1: Diagram of a simple LLM-Agent interaction. At each in/out circle, the possibility of hallucination is introduced to the workflow Even with this simplest of models, there are potential hallucinations in tool selection stage, the creation of the structured data provided to the tool for processing, and the interpretation of the results reported by the tool. This multiplies the number of possible hallucinations. With the inclusion of more complex models (that might contain their own LLM), the number of places where hallucinations could happen continues to grow. Figure 3.2: Example of a more complex LLM-Agent interaction. Note: The possibility of hallucination is introduced with each in/out cycle. This introduces a multiplicative effect for uncertainty in the results At each potential hallucination point, the sensitivity to differences in initial input ensure that the related request/responses are non-deterministic. This natural instability in the larger system is proportional to the complexity of the total agent ecosystem, and is highly resistant to containment. Unpredictable actions represent an institutional risk for an organisation when “decisions” made by the LLM/agent can create reputational or financial loss. Such uncertainty is not entirely academic. The set of expected behaviours is much larger for AI/agent systems as compared to traditional deterministic systems which creates problems in distributed Authorisation that exceed most local protections. 3.2 Agents and Attack Surface The above section discussed questions of behavioural stability in LLM driven agents. This section looks at the relationship between prompts, prompt responses, and attack surface. Here, “attack surface” refers to how much of an application is open to direct interaction from an attacker, (with less being the most optimal). Effects of the Current AI Ecosystem on Future GEANT Security 28 October 2025 8 The most obvious contributor to attack surface is the use of unstructured text in the prompt request and response. This is compounded by any number of factors, including: • LLMs are using a statistical interpretation of the common (English) language for the (request, agent invocation, response analysis and communication of results. Deviations from common usage increase uncertainty. • Most of the functionality of MPC lies within the context window where the MCP server communicates in plain language with agents. That means that there’s potential for deceit and manipulation. For instance, how does one prevent anyone saying “I am the CEO.” [ 10 ] • In natural language, it is possible to approach the meaning of a prompt from many directions. Eliminating prompt interpretations does not remove the ability to confuse the intended meaning. Essentially, "you cannot reduce that attack surface in a way that matters." [ 11 ] • Traditional identity and access management (IAM) [ 12 ] systems are ill-equipped to handle the dynamic and expansive nature of machine identities introduced by MCP. Without robust controls, these agents can become vectors for unauthorised access and data breaches. [ 13 ] • It is not unusual for an agent interaction to "lie" [ 14 ] about what it did. This is not part of the traditional concept of attack surface but adds a dynamic element. • If a user-controlled prompt contains multiple languages, how that prompt is interpreted may not be well defined. This would be particularly true with the addition of slang into the request. Such uncertainty in attack surface is not entirely academic. In July 2025, [ 15 ] a prompt attack was published where interaction between the iMessage agent and the Stripe credit card processor agent of the Claude LLM allowed an attacker to mint unlimited Stripe "coupons" (i.e. account credits from your payment system) without alerting the LLM operator. In this case the attacker used a simple prompt injection via iMessage to convince the LLM that the transaction was allowed, regardless of fences built around the prompt infrastructure. 4 Realistic Threats and Benefits Involving Security and LLM To better understand how LLMs and agents are used by both attackers and defenders, an initial examination of verifiable reports about what representatives in the AI and security community are discussing will be carried out. Looking at the threat intelligence reports from OpenAI during February [ 16 ] and June 2025 [ 17 ], Google/Gemini [ 18 ] from January 2025, and Anthropic/Claude [ 19 ] from April 2025, there are handful of clear trends. First, there is no indication that this technology is being used in a novel or disruptive development. This is explicitly called out in each of the documents. In each of the reports there were several examples of the LLM tools being used for research and feedback in surrounding questions about technical processes. Given the propensity of LLMs to mix truth and hallucinations, this results in unreliable output for both the attackers and defenders. By far the most detailed information in the reports in on the use of LLM tooling for content generation for social engineering. Of the different security-related postings, the OpenAI documents provide the most detail, with around 80% of use cases documenting large scale social engineering via social media apps and deliberate resume falsification. 4.1 Phishing, Vishing, and Other Social Engineering Attacks One of the most predictably effective methods for an attacker to gain a foothold in a target organisation is phishing and social engineering attacks. Phishing is cheap and effective, and has proven incredibly resistant to technical controls. Effects of the Current AI Ecosystem on Future GEANT Security 28 October 2025 9 Unfortunately, LLMs are particularly good at creating persuasive sounding messages because they have millions of examples to work from. By taking a collection of emails or communications from a specific user along with some sort of prompt, such as “create an email requesting a password reset similar to the following emails”, it is trivial to use a LLM to create highly targeted emails that have the same language and communication characteristics as the original author. In the same way that it is possible to write persuasive emails, LLMs can also be used for voice phishing (or "vishing"). It is possible to train the voice agent to sound like a specific individual by providing sample audio files. The agent is not limited to a simple script, as it can also provide impromptu communications (albeit with some latency). An arXiv study on automated vishing attacks presented at the ACM Asia CCS security conference in 2025 presented an example, where a vishing tool called ViKing [19] was built and tested by pool of 240 participants. From that user pool "even when participants were most strongly cautioned”, 33% still disclosed sensitive information to ViKing’s bots. Participant feedback indicated that 46.25% regarded ViKing as mostly/highly credible and trustworthy, and 68.33% perceived their interactions with ViKing as realistic." The cost of a successful attack was between USD0.50-USD1.16. It is worth noting that this is with a tool built by a handful of graduate students and used against a set of people who were aware a test was happening. It is important to consider social engineering attacks against password resets, MFA workarounds, and system requests via help desk. These attacks are often ignored because the topic is not "sufficiently technical" to be considered interesting. Unfortunately, there is a good match between a security tactic that has proven to be exceptionally successful for attackers and a strong point for LLMs where the technology works exceptionally well. 4.2 Augmenting Attacker Malware A popular discussion point around attacker tooling is how LLMs might be used to boost an attacker’s skill set to make them more skilled and adaptable in any given situation. Benefits and issues around the more general class of code assistance also fall into this category. As suggested at the start of this section, the benefits for this type of enhancement are somewhat limited and tend to focus on the more junior attackers or those operating outside their areas of familiarity. There is little or no evidence that there are advanced tools that can anonymously engage in malicious activity. To a large degree, the new tooling used by attackers seems to be basic tools augmented by some sort of prompt generation for additional help. An example of this would be the unfortunately named “Lame Hug” [ 20 ] malware campaign, which claims to be the first publicly documented malware campaign to use an LLM to help carry out an attacker’s tasks and adapt tactics during a compromise without needing to execute new payloads. At first glance, this seems like a great deal of work for a modest functional advance. Unfortunately, there is a great deal of systematic activity around exploitation that would succumb to this sort of augmented automation. While the current generation of attack assistant tooling seems naive, it is an area that could change rapidly without anything too unexpected happening, and as such, deserves careful monitoring. 4.3 Application Vulnerability Discovery A variation on LLM attacker assistance is automated vulnerability discovery. The main distinction is vulnerability discovery looks at a specific codebase, while attacker activity looks at an application running within a larger ecosystem. Vulnerability discovery is a category covering the identification and exploitation of specific known vulnerabilities in software systems like web servers, as well as identifying issues in source code.