Towards a Systematic Comparison Framework for Cloud Services Customer Agreements

Towards a Systematic Comparison Framework for Cloud Services Customer Agreements Elena Molino-Peña1(B)and José María García1,2 1Smart Computer Systems Research and Engineering Laboratory (SCORE), Universidad de Sevilla, Seville, Spain {mmolino,josemgarcia}@us.es 2Research Institute of Informatics Engineering (I3US), Universidad de Sevilla, Seville, Spain Abstract. The growing need to understand and compare elements in service agreements has generated strong interest in the industry. Although there are projects and tools for the automatic detection of information in contracts, automatic analysis is still a developing area of research. This becomes even more relevant with the rise of cloud service organizations, which highlights the need for tools for comparing contractual agreements. In this paper, we present a framework designed to automate contract analysis and comparison. In order to demonstrate the effectiveness of this approach, we created a prototype that uses language models to automatically detect obligations, rights, and parties involved in contracts. In addition, we applied an initial metric to determine the extent to which the customer benefits compared to the provider. The results of the evaluation support the effectiveness of the system by facilitating the understanding and reasoning of both parties regarding the terms of the agreement. Keywords: Services Agreements ·Automatically analyse · Comparison Framework 1 Introduction In today’s digital age, most companies often outsource some of their infrastructure management and storage to cloud services, which exposes them to a whole range of risks in terms of security, loss of data property, business continuity disruptions and other associated challenges. To mitigate these risks, it is crucial to understand the terms and conditions offered by providers and to ensure This work has been partially supported by the following grants: PID2021-126227NB- C21, PID2021-126227NB-C22, TED2021-131023B-C21, and PDC2022-133521-I00 which are funded by MCIN/AEI/10.13039/501100011033 and “ERDF a way of making Europe”; and grant PYC20 RE 084 US, which is funded by Junta de Andalucia/ERDF, UE. https://doi.org/10.1007/978-981-97-0989-2_19 242 E. Molino-Peña and J. M. García that they fulfill the clauses stipulated in the service agreement (also known as Customer Agreement or CA) that governs cloud services. CAs are composed of a set of legal documents that contain the terms and conditions concerning the acquisition and use of the provided cloud service [15]. They are the basis for protecting the interests of the parties and are an essential resource for bringing legal actions, for obtaining compensation or for termination of the contract, among other possible scenarios. In spite of the importance of the agreement in the cloud computing ecosystem, there is currently no effective mechanism to clearly comprehend the terms and conditions and compare the services offered by different providers. This entails additional costs for companies, as they need a higher number of professionals to analyse and ensure both the successful evolution of their contracts and new subscriptions. Moreover, since this is a manual task and considering the heterogeneity of the content within each supplier’s contract, the analysis is frequently tedious, time-consuming, and prone to error. Therefore, the lack of a tool that allows for the extraction and analysis of the information contained in the agreement, as well as the need for a set of general metrics that facilitate the evaluation and comparison across services, are two of the primary challenges limiting the automated analysis of CAs. To address this issue, a solution is needed that is able to reason about terms and conditions defined in the agreements, providing answers to a set of analysis operations that allows customers to compare between CAs. In other words, we refer to a system capable of identifying the relevant information in the agreement, giving answer to questions that may arise for both parties throughout the entire lifecycle of the agreement [5], spanning from before, during, and after the subscription process. From these answers, we can derive the value of various indicators applicable to all agreements, providing an effective way to compare the characteristics of the agreement. In this article, we present the following contributions: – A framework to automatically analyse and compare CAs. We define the functionality of the system and the components necessary for its development. – A preliminary solution that applies the previous approach to partially address the analysis of Service Level Agreements (SLAs). It allows automatic extraction of obligations and rights of the parties and execution of an analysis operation. The remainder of the article is organized as follows. Section 2describes our motivating scenario. Then, Sect. 3introduces the proposed framework, while Sect. 4describes our initial solution for detecting obligations, rights and actor in SLAs. Section 5discusses some related work and finally in Sect. 6we discuss our conclusions and future work. 2 Motivating Scenario The increasing number of available cloud service providers presents a challenge for IT organizations when selecting the most suitable one for their business goals. Towards a Systematic Comparison Framework for Cloud Services CAs 243 Most cloud providers offer services with similar features but differ in the terms and conditions outlined in their agreements and the associated costs. At this point, the analysis and comparison of individual providers’ agreements can be a complex task due to the lack of uniformity and the absence of a set of measurable general characteristics. Therefore, having a system that allows customers to ask questions about any doubts they may have about the terms and conditions of services becomes even more crucial in this context. To better motivate the need for a CA comparative framework, let us examine the following example represented in Fig. 1, which provides a general overview of the situation a customer faces when they need to subscribe to a new service. Consider the case where the company Acme would like to host some of their systems on a cloud infrastructure, so that its personnel could search for the different Infrastructure-as-a-Service (IaaS) alternatives available. Moreover, they have to assess which of them best suits their needs. Of all the possibilities, imagine that they only select Amazon Web Services, Microsoft Azure and Google Cloud for simplicity, as exemplified in the figure. Each of these selected organisations structures their agreements differently, possibly including some vendor-specific policies. In addition, they have a multitude of documents that need to be analysed because they may contain information that can directly affect the services to be contracted. In this situation, comparing the terms and conditions established by each provider and selecting the one that most closely aligns with the customer’s requirements is often costly, particularly when properly managing potential operational risks that may arise. Fig. 1. Essential topics covered by each provider’s CA 244 E. Molino-Peña and J. M. García This scenario motivates the definition of a CAs comparative framework to effectively and efficiently support customers in these decisions, by automatically extracting and reasoning about the information contained in each of these agreements. 3 Systematic Comparison Framework This section aims to outline the minimum requirements necessary for building a framework that automates the CA analysis and facilitates comparisons between different providers. The main objective of this system is to accurately process agreements, enabling automated analysis operations. This analysis culminates in the calculation of various quantitative and qualitative metrics, allowing for an evaluation of the agreement from the perspective of operational risks and its impact on the client’s business. This provides an efficient means of comparing agreements from different providers. Fig. 2. Scenario of automatic analysis and comparison Operational risks arising from the terms and conditions established in the CAs can result from a wide variety of issues. Some of the situations that can generate weaknesses in the business can be highlighted, such as non-compliance with current regulations in terms of security and data protection, lack of responsibilities defined for certain participants in the contract, non-existent or ineffective clauses and inconsistencies between various conditions, among other possible aspects. Trying to mitigate and be aware in time of the scope of these potential losses and the rights granted can help the client to realise the actual coverage Towards a Systematic Comparison Framework for Cloud Services CAs 245 offered by each provider, to understand the responsibilities of the parties, to take legal action if deemed necessary, and in general to get the most out of the contracted service. Figure 2shows the general workflow of our proposed systematic CA comparison framework. The system can take as input the CAs from different providers and, in addition, a set of analysis operations that need to be solved. The automatic reasoner must obtain responses to each of the analysis operations using the information contained in the agreements to be evaluated, represented as Qi and Ai, respectively. This response must be able to be affirmative, negative, without sufficient information, or with detailed explanations. The ultimate goal is to facilitate the understanding of the differences between providers with respect to these analysed characteristics, as represented in the bottom-right part of Fig. 2. For this process to be effective, the automatic reasoner will need at least two critical and necessary components to automate CA analysis. The first of this is the information extractor, which in general should be responsible for processing the text in natural language and identifying those clauses that contain relevant information based on the query to be performed. There are several techniques that can be applied to solve this task, such as tokenization, Part-Of- Speech (POS) tagging and other preprocessing possibilities to analyse text. In addition, patterns, traditional Natural Language Processing (NLP) models and Large Language Models (LLMs) can be used to obtain the information. The second component is the engine responsible for enabling reasoning about the content of the agreement. This engine applies analysis operations to the structured information previously extracted. This allows obtaining simple answers, such as “yes” or “no”, as well as more complex answers, including detailed explanations of the answer provided. One of the ways to reason about the identified information is to interpret it and translate it into a formal language or structure that allows the application of a resolution or optimization algorithm to obtain a solution based on the defined constraint functions [4]. Another alternative is to use LLMs to process the detected clauses and answer the questions provided. Finally, another approach could involve establishing quantitative and qualitative metrics to derive conclusions from the information extracted. 4 Use Case Validation Cloud services are used to deliver value to customers by simplifying the management and maintenance of IT resources. It is important to keep in mind that the provider will make every effort to minimize its own losses. At this point, it is particularly interesting to understand the obligations and rights of each party involved to ensure that all parties are fulfilling their responsibilities. Furthermore, a comprehension of these clauses can help both parties to understand critical aspects such as who benefits the most, potential impacts on the business resulting from subscribing to the services, and whether both parties have fair and sufficient rights and responsibilities. Hence, it is essential to have a set of analysis operations that, through various metrics, provide a comprehensive representation of the agreement to be evaluated. 246 E. Molino-Peña and J. M. García In order to translate this idea into practice using our systematic comparison framework approach, we developed a prototype solution that can automatically extract from the SLA part of CAs their obligations and rights, together with the actors involved. On the other hand, we propose a first simple metric that allows to assess the risk or benefits of the parties, taking into account the relationship between the obligations and the rights of the parties involved in the agreement. Figure 3shows the workflow that the developed system follows. Its architecture is divided into two proposed components for the automatic reasoner, which are the information extractor and the analysis engine. These components will be explained in detail below. In general terms, the figure aims to show the different activities and elements necessary to achieve the expected results. First, we process the agreement by dividing the SLA text into sentences that are then evaluated by an LLM which is able to decide whether any of the clauses is an obligation, a right or a simple condition of the contract. Once we obtain these propositions, we use the Open AI GPT-3.5 turbo model to identify who is the actor involved in each of these clauses. Finally, using all of this information we can calculate the ratio of clauses that benefit each party. Fig. 3. Prototype solution workflow 4.1 Information Extractor The important advances in NLP and deep learning techniques lead us to consider that an excellent alternative to extract information from SLAs is to apply LLMs. Most of these models are prepared to address a variety of tasks, such as text classification, translation, question-answering, or text summarising. These models have been trained with a large corpus, which allows them to be flexible solutions for different domains. However, their main limitation is that they are not specifically designed to solve a particular problem. Therefore, it is usual to perform a fine-tuning process of a pre-trained model with a specific data set to address the required task [2]. Towards a Systematic Comparison Framework for Cloud Services CAs 247 Given the diversity of options, we decided to fine-tune a model specifically for classifying statements into the categories of obligation,right,orneither.In this context, due to the limited amount of labeled data in the agreement domain, we had to generate a corpus1of SLA sentences from various providers, classified into the categories mentioned above. Despite performing a manual tagging of 52 SLAs from 18 leading cloud providers, it is important to note that the available data set is still quite limited. Therefore, we decided to use an innovative approach known as SetFit for training a variety of models, getting the best results with the paraphrase-mpnet- base-v2 model 2, a modification of the Bert model [14]. SetFit is a framework that has proven to be highly effective for fitting pre-trained models, even when a limited number of examples are available, by applying contractive learning techniques [16]. The result is the paraphrase-mpnet-base-v2-sla-obligations-rights model3specially trained to detect obligations and rights in the SLA domain. The values obtained for the model evaluation metrics are as follows: Accuracy: 0.83, F1-macro: 0.829, Precision-macro: 0.8309, and Recall-macro: 0.83. After having a model that can extract a catalog of sentences indicating an obligation or right from any SLA, we query the OpenAI API to detect the actor using the gpt-3.5-turbo model4. The process consists of generating a prompt with the extracted obligations or rights, and as a result the model generates a list with the actors involved in the corresponding clause. We have developed an initial prototype5following this approach. The system has an API that allows interacting with all the previously defined functions. The proposed solution for information extraction automatically assigns the phrase type based on the semantics of the input data, avoiding the dependency on specific patterns. However, it is important to note that this assignment may be affected by the knowledge and possible biases of the classifier. In addition, one of the most robust models for actor detection is used, reducing possible problems produced by previous techniques used in other approaches. Although we need to further validate this solution, we have obtained promising early results, as outlined in Sect. 4.3. 4.2 Analysis Engine The information extracted allows us to reason about the terms of the agreement. Based on a set of metrics, we can, for instance, analyse whether the agreement is beneficial to the customer or the provider, as well as the risks involved in subscribing to the cloud service, among other analysis operations. Currently, 1Dataset available at https://huggingface.co/datasets/marmolpen3/slas-obligations- rights-sentences. 2https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2. 3https://huggingface.co/marmolpen3/paraphrase-mpnet-base-v2-sla-obligations- rights. 4https://platform.openai.com/docs/models/gpt-3-5. 5Prototype available at https://github.com/isa-group/iContracts. 248 E. Molino-Peña and J. M. García there are limitations regarding the impact that the detected clauses can have on the reasoning process. This is because the system is unable to identify the meaning of the clause. For the identification of a meaningful metric, we use the Goal Question Metric (GQM) method. Our metric seek to understand the balance of obligations and rights between parties involved in the agreement, through a quantitative analysis. In other words, it is about assessing the extent to which the contract could provide protection against potential risks. –Goal: The degree to which the terms and conditions are favourable to the parties. –Question: What is the distribution of favourable clauses for each party? –Client benefit:benefitclient = Obp Obp+Obc+Rc Rp+Rc 2 –Explanation: The customer’s benefits are represented by the sum of the percentage of the provider’s obligations and the percentage of the customer’s rights, where a higher number of obligations on the provider’s side leads to a higher coverage and consequently to a more significant benefit for the customer. –Provider benefit:benefitprovider = Obc Obp+Obc+Rp Rp+Rc 2 –Explanation: The provider’s benefit is represented by the sum of the percentage of the customer’s obligations and the percentage of the provider’s rights, where a higher number of obligations on the part of the customer leads to a higher benefit for the provider. In the formulas, Obprepresents the number of obligations on the provider’s side, and Rprepresents the number of rights. Conversely, Obcdenotes the number of obligations on the customer’s side, while Rcsignifies the number of rights. To calculate the ratio between the customer’s and provider’s percentage of benefit, we divide the customer’s percentage by the provider’s percentage. This determines whether the provider’s responsibilities are greater than those of the customer. 4.3 Initial Evaluation In order to perform an initial validation of the developed system, we propose to answer the following evaluation objectives: –Goal 1:How does the system behave in a real use case? We intend to evaluate the performance of the system against a real scenario, such as the one described in the motivation section (Fig. 1). –Goal 2:What are the results if an agreement is analysed with different tools? Our objective is to evaluate the performance of the system compared to other tools available for the automatic analysis of obligations and rights in SLAs. We will compare the results obtained when evaluating an SLA using the Chat GPT language model, the framework developed by Natalona et al. [12], and our solution approach. Towards a Systematic Comparison Framework for Cloud Services CAs 249 Fig. 4. Excerpt of Google Compute Engine obligations, rights and actors detected by our proposal Table 1. Results obtained for each SLA SLA EC2 Compute Engine App Service Obp3 2 6 Obc2 4 5 Rp3 0 0 Rc4 3 4 Metric 1 0.586 0.667 0.773 Metric 2 0.414 0.333 0.227 Ratio 1.415 2 3.4 Experiment 1: Analysis of Cloud Computing SLAs. In this experiment, we aim to answer the first goal by evaluating the obligations, rights and actors detected by our approach. We analyse the SLAs of the companies mentioned in the motivation scenario, which are AWS’ EC2 SLA, Microsoft’s App Service SLA and Google’s Compute Engine SLA. Figure 4showcases an excerpt of the information that our system can extract from the agreement. The global results are presented in Table 1, which shows a summary of the information extracted by the system. According to these results, the customer benefits more than the provider in all SLAs analysed, with the biggest difference in the Microsoft App Service SLA. Specifically, in this case, the customer’s benefit is 3.4 times higher than the provider or, in other words, the provider’s responsibilities are greater than those of the customer. The experimental study confirms the system’s ability to extract information from the SLA, helping the customer to better understand the responsibilities and rights of the parties, as well as the most supportive agreement.