COSI: Container Object Storage Interface

COSI: Container Object Storage Interface August 2024 AUTHOR: Karanjot Singh SUPERVISOR(S): Jack Henschel Ankur Kothiwal CERN openlab Report 2024 ABSTRACT The Container Object Storage Interface (COSI) is a new standard designed to provision and manage object storage within Kubernetes environments, similar to how the Container Storage Interface (CSI) handles block and file storage. As organizations increasingly rely on Kubernetes for orchestrating containerized workloads, the need for a robust, vendor-neutral interface for object storage has become paramount. The COSI project aims to fill this gap by providing a dynamic provisioning API that integrates seamlessly with Kubernetes, enabling users to manage object storage resources in a consistent and Kubernetes-native manner. At CERN, object storage plays a pivotal role in supporting various services such as storing attachments for Mattermost, GitLab, and registry container images from Harbor, as well as offering storage solutions to Platform-as-a-Service (PaaS) users. By integrating COSI into CERN’s Kubernetes (K8s) and PaaS services, we aim to streamline the provisioning and management of object storage. The report outlines the architecture, benefits, and processes associated with COSI, as well as insights gained from testing the Ceph COSI driver. Additionally, the project addresses critical aspects of data resilience through the implementation of S3 backups and Quotas, ensuring robust data protection and disaster recovery. 2 CERN openlab Report 2024 TABLE OF CONTENTS INTRODUCTION 01 Background 02 Container Object Storage Interface (COSI) Key Concepts Object Bucket Claim (OBC) Object Bucket (OB) Storage Class (SC) Bucket Access Benefits of COSI COSI Architecture COSI API and Object Lifecycle 03 POC using Ceph COSI Driver 04 S3 Backups and Quotas 05 Conclusion 06 3 CERN openlab Report 2024 1. INTRODUCTION Kubernetes has become a leading platform for orchestrating containerized applications, offering a flexible, scalable, and robust environment for deploying microservices. One of the key features of Kubernetes is the Container Storage Interface (CSI), which standardizes the way external storage systems are integrated, allowing users to manage file and block storage efficiently. However, until recently, Kubernetes lacked a standardized interface for managing object storage, which has become increasingly important due to the growing use of cloud-native applications that rely on object stores like Amazon S3, Google Cloud Storage, and Ceph Rados Gateway (RGW). The Container Object Storage Interface (COSI) aims to fill this gap by providing a vendor-agnostic, Kubernetes-native way to provision, manage, and consume object storage. This report delves into the COSI architecture, its components, and how it can be integrated into existing Kubernetes and OpenShift environments at CERN, particularly with Ceph as the object storage backend. It also includes the implementation of S3 backup and Quotas strategies, which are critical for ensuring data durability and availability. 2. Background Before the introduction of the Container Object Storage Interface (COSI), Kubernetes mainly handled block and file storage using the Container Storage Interface (CSI). Launched in 2017, CSI was created to overcome the limitations of in-tree storage plugins by offering a standardized approach for integrating various block and file storage systems with containerized workloads. This standardization enabled third-party storage providers to build and deploy their own volume plugins independently of the Kubernetes codebase, greatly enhancing the flexibility and scalability of the Kubernetes storage layer. Despite the improvements brought by CSI in block and file storage, Kubernetes still lacked a built-in solution for managing object storage—a critical component in today’s applications. Object storage, which features a flat and scalable structure with discrete objects stored in containers or buckets, provides key benefits such as cost-effectiveness, scalability, and easy access via network APIs like S3. These attributes make it ideal for handling the large amounts of unstructured data generated by modern applications. To address this gap, the Kubernetes community introduced COSI, extending Kubernetes’ support to object storage. Modeled after the CSI framework, COSI offers a unified, vendor-neutral layer for provisioning and managing object storage. It brings in new abstractions similar to those in CSI, including BucketClass, Bucket, and BucketClaim, allowing Kubernetes users to apply familiar concepts to object storage management. This represents a significant step forward for Kubernetes, meeting the increasing demand for a standardized and efficient way to manage object storage. 3. Container Object Storage Interface (COSI) The Container Object Storage Interface (COSI) introduces a standardized way for Kubernetes clusters to manage object storage, similar to how CSI has standardized block and file storage. COSI provides a vendor-neutral API that abstracts the complexities of different object storage providers, allowing Kubernetes users to seamlessly provision and manage object storage resources within their clusters. COSI aims to simplify the lifecycle management of object storage by introducing Kubernetes-native constructs that represent object storage resources, such as buckets and access credentials. This 4 CERN openlab Report 2024 abstraction enables Kubernetes users to interact with object storage in a way that is consistent with other Kubernetes resources, reducing the learning curve and operational overhead. a. Key Concepts Object storage differs from traditional block and file storage in several ways. In object storage, data is stored as objects in a flat namespace, rather than in a hierarchical file system. Each object typically consists of the data itself, metadata, and a unique identifier. This approach allows for virtually unlimited scalability and is particularly well-suited for large, unstructured datasets. COSI introduces several new Kubernetes Custom Resource Definitions (CRDs) to represent object storage resources: i. Object Bucket Claim (OBC): Similar to a Persistent Volume Claim (PVC) in CSI, an OBC is a namespaced resource that represents a user's request for an object bucket. The OBC specifies the desired storage class and other parameters, such as bucket name and region. ii. Object Bucket (OB): An OB is a cluster-scoped resource that represents the actual object bucket provisioned by the storage backend. It is analogous to a Persistent Volume (PV) in CSI. The OB contains information about the bucket's location, credentials, and other relevant details. iii. Storage Class (SC): In COSI, the Storage Class defines the parameters for provisioning object storage, such as the storage backend, region, bucket owner, and access policies. It is referenced by OBCs to determine how and where the bucket should be provisioned. iv. Bucket Access: This resource represents the credentials or access mechanism required to use the object bucket, such as access keys or IAM roles. It is similar to access controls in traditional storage systems, ensuring that only authorized users can access the bucket. b. Benefits of COSI COSI offers several benefits for Kubernetes users and administrators: i. Kubernetes-Native Integration: By integrating object storage directly into Kubernetes, COSI allows users to manage object storage resources using familiar Kubernetes tools, such as kubectl. This reduces the need for specialized knowledge and simplifies the operational workflow. ii. Self-Service Provisioning: With COSI, developers can provision and manage their own object storage resources without needing to rely on cluster administrators. This improves agility and reduces the time required to deploy new applications. Self service is easily possible with the current design as both the BucketRequest and BucketClaim resources are namespace scoped, and users need not have admin privileges to create, modify and delete them. iii. Vendor Neutrality: COSI abstracts the underlying storage provider, allowing users to switch between different object storage solutions without changing 5 CERN openlab Report 2024 their application code. This reduces vendor lock-in and provides greater flexibility in choosing the best storage solution for a given workload. c. COSI Architecture The COSI architecture is composed of three main components that work together to manage the lifecycle of object storage resources in Kubernetes(see Figure 1): i. COSI Controller Manager: The COSI Controller Manager is the central component responsible for processing changes to COSI API objects, such as the creation, update, or deletion of object buckets and access credentials. It ensures that the requested operations are validated, authorized, and executed according to the defined storage policies. The Controller Manager also handles the binding of Object Buckets (OBs) to Object Bucket Claims (OBCs), ensuring that the correct resources are allocated to users. ii. COSI SideCar: The COSI SideCar acts as an intermediary between the COSI Controller Manager and the storage provider-specific COSI Drivers. It translates COSI API requests into gRPC calls that are understood by the COSI Drivers. The SideCar is responsible for triggering all operations that require communication with the object storage backend, such as bucket creation, deletion, and access management. iii. COSI Driver: The COSI Driver is a vendor-specific component that interacts with the underlying object storage system. It implements the gRPC protocol defined by COSI and performs the actual operations on the storage backend, such as creating or deleting buckets, generating access credentials, and configuring access policies. Different storage providers can implement their own COSI Drivers to integrate with Kubernetes, enabling support for a wide range of object storage solutions. 6 CERN openlab Report 2024 Figure 1: Architecture of COSI d. COSI API and Object Lifecycle COSI introduces several new API objects in Kubernetes, each representing a different aspect of object storage management: i. Bucket: The Bucket resource represents a specific bucket in the object storage provider. It is a cluster-scoped resource that contains information about the bucket's location, storage class, and access policies. ii. BucketClaim: A BucketClaim is a namespaced resource that allows users to request the creation of a new bucket or access to an existing bucket. The BucketClaim specifies the desired storage class, bucket name, and other parameters, and is bound to a corresponding Bucket resource by the COSI Controller Manager. iii. BucketClass: The BucketClass resource defines common properties for multiple buckets, such as storage policies, replication settings, and access controls. It is referenced by BucketClaims to determine how the requested bucket should be provisioned and managed. iv. BucketAccess: The BucketAccess resource represents the credentials or access mechanism required to use a bucket. It is similar to traditional access controls in storage systems and ensures that only authorized users can access the bucket. v. BucketAccessClass: The BucketAccessClass resource configures access policies and mechanisms for buckets, such as key-based authentication or 7 CERN openlab Report 2024 IAM-based roles. It is referenced by BucketAccess resources to determine the appropriate access controls for a bucket. The object lifecycle in COSI involves several key stages: vi. Bucket Creation: When a user submits a BucketClaim, the COSI Controller Manager validates the request and provisions a new bucket in the object storage provider using the specified BucketClass. A corresponding Bucket resource is created in Kubernetes to represent the bucket. vii. Access Credential Generation: When a user requests access credentials via a BucketAccessClaim, the COSI Controller Manager generates these credentials, stores them in a BucketAccess resource, and makes them available to the user through a Kubernetes Secret. viii. Bucket Usage: Once the bucket is created and access credentials are generated, the user can store and retrieve objects in the bucket using standard S3 APIs or other compatible interfaces, with the credentials and bucket name exposed to the user through a Kubernetes Secret. ix. Bucket Deletion: When the user no longer needs the bucket, they can delete the BucketClaim, triggering the COSI Controller Manager to delete the corresponding Bucket and BucketAccess resources. The bucket is then deleted from the object storage provider, and all associated data is permanently removed. 4. Proof of Concept using Ceph COSI Driver To evaluate the functionality and performance of COSI, the Ceph COSI Driver was implemented as a reference driver for Ceph's RADOS Gateway (RGW). The implementation involved the following key steps: a. Installation of COSI CRDs: The initial step was the installation of the necessary Custom Resource Definitions (CRDs) for COSI within the Kubernetes cluster. These CRDs define the new API objects introduced by COSI, such as BucketClaims, Buckets, and BucketAccesses, enabling Kubernetes to manage object storage resources consistently. b. Deployment of COSI Controller and SideCar: The next phase involved deploying the COSI Controller Manager and the SideCar components in the Kubernetes cluster. The Controller Manager is responsible for managing the lifecycle of COSI resources and processing API requests, while the SideCar facilitates communication between the COSI Controller and the Ceph COSI Driver, translating Kubernetes API requests into actions on the Ceph RGW. c. Building and Installing the Ceph COSI Driver: The Ceph COSI Driver was built from the source with our custom logic for managing secrets and subsequently installed in the Kubernetes cluster. This driver is the critical component that interfaces directly with the Ceph RGW API, enabling the provisioning and management of object storage resources as Kubernetes-native objects. 8 CERN openlab Report 2024 d. Configuration of Storage Classes and Access Policies: The configuration process involved setting up StorageClasses and BucketAccessClasses to define the storage policies and access controls necessary for the Ceph RGW buckets. These configurations determine how storage resources are provisioned, accessed, and managed within the Kubernetes environment, with the COSI Controller Manager using them to process BucketClaims and BucketAccess requests. e. Testing and Validation: Comprehensive testing was conducted to validate the implementation. This included creating BucketClaims, provisioning buckets, generating access credentials, and performing CRUD (Create, Read, Update, Delete) operations on the buckets. The tests confirmed that the COSI framework can be successfully integrated with Kubernetes Magnum and PaaS Services. f. Key Challenges Encountered During Implementation i. Ceph COSI Driver Modifications: I adjusted the Ceph COSI Driver to pass Secrets directly instead of generating user-key pairs for buckets within the Kubernetes cluster. This adjustment serves as a workaround and is detailed in the modified codebase, which can be accessed here https://gitlab.cern.ch/karanjot/ceph-cosi/-/commit/d31be044faf09a290562ee4 872b1e1accdb57e02 . ii. Bucket Naming Challenge: During testing, I observed that buckets were being created with random names rather than the specified names in the bucket claims. Although this behavior was unexpected, it turns out to be a deliberate feature of COSI, designed to prevent potential bucket hijacking. To solve this issue for our test application, I developed an entrypoint script that exposes the Secret as a volume. This script ensures secure access to required resources by exporting environment variables like bucketName, SecretKey, and AccessID, using the Secret mounted within the Kubernetes pod. iii. Generic COSI Driver Development: We also developed a generic COSI driver that is vendor-neutral and uses the S3 API to perform create and delete operations. However, we encountered challenges with access management, as credential management varies depending on the vendor. 5. S3 Backups and Quotas Beyond the implementation of the Container Object Storage Interface (COSI), this project also focused on strengthening the reliability and resilience of our object storage by integrating S3-compatible backup and replication strategies. a. S3 Backups Backups are a critical component for safeguarding data against accidental deletion, corruption, or other unforeseen failures. S3 backups involve creating duplicate copies of data stored in S3 buckets, ensuring that data can be recovered in the event of loss. One of the key features of S3 is its native support for versioning, which allows us to retrieve specific versions of objects stored within a bucket at any time. 9