ChainKode: a CICD methodology for chaincode development in Hyperledger Fabric based on Kubernetes

ChainKode: A CICD methodology for chaincode development in Hyperledger Fabric based on Kubernetes YASIRU RATHSARA WITHARANAGE, CEIT-Basque Research and Technology Alliance (BRTA), Spain SANTIAGO FIGUEROA-LORENZO, CEIT-Basque Research and Technology Alliance (BRTA), Spain NASIBEH MOHAMMADZADEH, CEIT-Basque Research and Technology Alliance (BRTA), Spain SAIOA ARRIZABALAGA JUARISTI, CEIT-Basque Research and Technology Alliance (BRTA), Spain Hyperledger Fabric is a widely used blockchain technology that allows organizations to store and exchange data through permissioned networks. A chaincode extends its functionality by enabling participants to interact with the stored data via multiple programming languages. Consequently, a chaincode can be regarded as an external service to the Hyperledger Fabric network which can be deployed by container orchestration tools such as Kubernetes. This induces a distinctive Software Development Life Cycle (SDLC) for chaincodes along with the necessity for a sustainable framework to support it. To this end, we propose a methodology based on CICD practices that not only facilitates the development of chaincodes in Golang but also ensures that their functional and non-functional requirements are satisfied. In particular, we deliver an automated process that establishes adequate code quality, verifies the integrity of implementation, validates security vulnerabilities, evaluates performance and finally deploys the chaincode to a Hyperledger Fabric blockchain. We provide a performance analysis of our approach based on the conducted experiments and a discussion on its qualitative attributes. Additionally, we present a comprehensive ecosystem for Hyperledger Fabric with Kubernetes which includes monitoring and benchmarking tools. In essence, we strengthen the SDLC of chaincodes and facilitate the adoption of Hyperledger Fabric in cross-organizational use cases. Additional Key Words and Phrases: Hyperledger Fabric, CICD, Chaincode, Kubernetes 1 Introduction Blockchain has emerged as a profound technology that enables data sharing among multiple participants using an immutable distributed ledger. It has further evolved into a variety of blockchain platforms that can adapt to different industrial use cases befittingly. Hyperledger Fabric is one such technology where distinctive organizations can store and share data with high performance and privacy in a permissioned network [ 1 ]. It further allows participants to interact with these data by implementing arbitrary services via chaincodes in multiple programming languages (e.g. Java,Golang). Developing secure and performant chaincodes can be regarded as non-trivial due to their involvement in cross-organizational use cases with immutable data. Continuous Integration and Continuous Deployment/Delivery (CICD) provides an automated process to accelerate the Software Development Life Cycle (SDLC) while ensuring the efficacy of an implementation. Despite its benefits, Hyperledger Fabric still lacks a proper investigation of adopting CICD practices, thereby resulting in chaincodes with inadequate code quality, security vulnerabilities, performance limitations and inefficient development life cycles [2]. To the best of our knowledge, this can be regarded as the first academic research on proposing a CICD methodology for the development of chaincodes. In particular, our main contributions of this study can be outlined as follows. Authors’ Contact Information: Yasiru Rathsara Witharanage, CEIT-Basque Research and Technology Alliance (BRTA), Donostia/San Sebastian, Basque Country, Spain, r[email protected]; Santiago Figueroa-Lorenzo, CEIT-Basque Research and Technology Alliance (BRTA), Donostia/San Sebastian, Basque Country, Spain, [email protected]; Nasibeh Mohammadzadeh, CEIT-Basque Research and Technology Alliance (BRTA), Donostia/San Sebastian, Basque Country, Spain, [email protected]; Saioa Arrizabalaga Juaristi, CEIT-Basque Research and Technology Alliance (BRTA), Donostia/San Sebastian, Basque Country, Spain, [email protected]. Yasiru Rathsara Witharanage, Santiago Figueroa-Lorenzo, Nasibeh Mohammadzadeh, and Saioa Arrizabalaga Juaristi • An ecosystem of Hyperledger Fabric with benchmarking, monitoring and CICD tools for industrial use cases based on Docker and Kubernetes. •A comprehensive CICD methodology to support the Software Development Life Cycle (SDLC) of chaincodes. • Accommodate Golang in chaincode development to ensure the delivery of desired integrity, code quality, security and performance levels. The rest of this paper is structured as follows: Section 2 describes the background whereas section 3 outlines the related prior work. Section 4 analyses the requirements of a CICD methodology for Hyperledger Fabric. Section 5 proposes our approach followed by its implementation details in section 6. Sections 7 and 8 present an evaluation and a discussion of the implementation, respectively. Finally, section 9 concludes the paper. 2 Background In its simplest form, blockchain is a decentralized network where participants can store data as transactions in an immutable distributed ledger. In Hyperledger Fabric, admission of participants to a ledger can be governed, thus deriving an implicitly trusted network where all the participants are known. Consequently, Hyperledger Fabric is widely adopted in industrial use cases, specifically where distinctive organizations require exchanging data with each other while enforcing decentralization and data sovereignty. Hyperledger Fabric supports the existence of multiple ledgers within the same overlay network through a concept called channels. A channel not only segregates contextually different data but also imposes fine-grained privacy by restraining access to its data only to the channel members. It further allows organizations to interact with these data via a chaincode which contains a set of specific implementations known as smart contracts. Participants can invoke these smart contracts via client applications to perform various business use cases with data stored in channels. An organization in Hyperledger Fabric generally deploys its own sub-network with several components such as peers,orderers and Certificate Authorities (CAs). In essence, a peer joins a channel and allows client applications to interact with its channel data through a chaincode. A chaincode can be installed on a peer via buildpacks which contain the instructions to detect, build and run a chaincode instance. It is also possible to configure anchor peers which are used by other organizations and external applications to connect and retrieve information about the network [ 3 ]. An orderer participates in a collective process to order any submitted data, bundles them into a block and propagates this block to be committed by peers. In addition, there exists two types of CAs in Hyperledger Fabric: A TLS CA to enable Transport Layer Security (TLS) in communication and an organization CA to generate the identities of users. Correspondingly, it necessitates skilled personnel to deploy all these components, establish their connections and maintain software and hardware infrastructure while ensuring the reliability of components. In addition to the Fabric framework, Hyperledger provides a number of tools to supplement its ecosystem. Particularly, Blockchain Explorer allows the organizations to monitor a Hyperledger Fabric network in terms of its network infrastructure, deployed chaincodes and committed blocks with transactional data. Hyperledger Caliper is a benchmarking tool that evaluates the performance of a chaincode based on custom workloads and configurations. DevOps serves as a role that combines the tasks of developing, deploying and maintaining software services. It further introduces a set of practices commonly known as Continuous Integration and Continuous Deployment/Delivery (CICD) that accelerates and simplifies the Software Development Life Cycle (SDLC). This is accomplished through an automated process called a pipeline which consists of multiple jobs to build, test, deploy and monitor software services. In addition to the convenience, such pipelines enable compliance with standards by integrating validation tasks (e.g. unit tests, ChainKode: A CICD methodology for chaincode development in Hyperledger Fabric based on Kubernetes code quality checks, benchmarks) and verifying their corresponding results. The CICD pipelines can be monitored with tools such as Prometheus and Grafana, which retrieve the metrics and visualize them in graphs, respectively. Docker provides a container runtime that enables building and publishing software applications as portable images. These images can be deployed and managed with a container orchestration tool such as Kubernetes by using declarative manifest files. Gitlab is a cloud-native platform with features to support DevOps in software development. In general, developers use Gitlab as a Version Control System (VCS) to maintain source code files of the implementations. Gitlab further provides a private and secure container registry that can be used to publish, pull and deploy containerized images (e.g. Docker) of software services whenever necessary. Additionally, a CICD pipeline can be realized with Gitlab CI by implementing the flow in a declarative configuration file and executing its jobs by means of a Gitlab runner. These runners can also be hosted locally (self-managed) within an organization to provide additional security and privacy during the execution of a job. 3 Related Work Several research works have been done on the application of CICD methodologies for smart contracts in Ethereum blockchain. In particular, Maximilian Wöhrer et al. propose a CICD framework to support the development of smart contracts [ 4 ] whereas Hauke Precht et al. extend this work to deliver a containerized approach which is agnostic of a CI platform [ 5 ]. Security concerns related to Continuous Integration of smart contracts have been researched by Alvaro Reyes et al. [ 6 ]. In addition, there are several non-scientific publications in which the community has outlined general concerns related to CICD practices for Ethereum [7, 8]. A number of research attempts have been made on deploying Hyperledger Fabric on Kubernetes while being constrained to a specific use case. Particularly, Muhammad Rehan et al. deploy Hyperledger Fabric on Kubernetes for Supply Chain Management [ 9 ] while Xiubo Liang et al. exploit Kubernetes for both the infrastructure and development of chaincodes related to education [ 10 ]. Vladimir Yussupov et al. present an overview of the blockchain technology and serverless architecture [ 11 ], including service platforms which provide Hyperledger Fabric based on Kubernetes. However, these papers neither entail external components for a complete ecosystem of Hyperledger Fabric nor support the SDLC of chaincodes. Maciej Kopa et al. propose a CI methodology for Hyperledger Fabric which is only applicable for chaincodes based on Java and it further lacks the mechanisms for performance evaluation, continuous deployment and monitoring of the CICD processes [12]. Similarly, Nitin Gaur et al. discuss a basic CI flow for chaincodes but it does not validate the security of third-party dependencies as well as the performance of a chaincode [ 13 ]. A custom Hyperledger Fabric buildpack has been implemented by the community known as k8s builder 1 which enables installing and launching a chaincode as a Kubernetes service. Nevertheless, it does not segregate the SDLC of a chaincode since it requires a peer to launch the chaincode with each deployment. In addition, we noted several studies that addressed the performance of Hyperledger Fabric [14, 15] and used Hyperledger Caliper to perform benchmark tests [16, 17]. 4 Requirement Analysis We outline several requirements associated with the deployment of chaincodes as listed below. (1) Efficient deployments: The conventional method of deploying a chaincode in Hyperledger Fabric v2.x involves a number of sequential steps: package the chaincode in compliance with the required folder structure, install the chaincode on peers, approve the chaincode definition and subsequently commit it to a channel [ 18 ]. This 1https://github.com/hyperledger-labs/fabric-builder-k8s Yasiru Rathsara Witharanage, Santiago Figueroa-Lorenzo, Nasibeh Mohammadzadeh, and Saioa Arrizabalaga Juaristi Fig. 1. CICD flow with the delivery of chaincode properties entire process needs to be iterated whenever the implementation is modified which can potentially decelerate the pace of a chaincode’s development life cycle. (2) Support for development: The deployment process of a chaincode requires developers to possess operational knowledge [19] which can hinder the growth of developer community in Hyperledger Fabric. (3) Functional integrity: In contrast to conventional software, a chaincode interacts with data in a crossorganizational network and may offer services for multiple distinctive consortiums [ 20 ]. Therefore, it is mandatory to ensure that a deployed chaincode (i) satisfies its functional requirements and (ii) does not include any malevolent behaviour. (4) Service integrity: A peer should only interact with a valid chaincode instance and hence the integrity of a chaincode service should be validated upon the connection establishment with a peer. (5) Secure chaincodes: A chaincode demands to be highly secure with a thorough investigation of the core implementation and its external dependencies such as libraries and runtime environments [21]. (6) Chaincode lineage: A manifestation of chaincode life cycle events, including deployments, should be logged with non-repudiation to support traceability, auditability and transparency whenever necessary [22]. (7) Chaincode versioning: A deployed chaincode is referred by a version in the network which should not allow any modifications during its execution lifetime [18]. (8) Portable chaincodes: Organizations in a Hyperledger Fabric network may share and deploy the same chaincode package in order to maintain consistency [18]. (9) Performance of a chaincode: Blockchains consume more transaction latency and deliver less throughput compared to conventional databases [ 23 ]. Therefore, evaluating the performance of a chaincode prior to its deployment can result in awareness of the overall system and further restrain the deployment of any underperforming chaincodes. Concerns (1)-(2) imply that the adoption of a CICD methodology supports the SDLC of a chaincode in Hyperledger Fabric, whereas (3)-(9) demand that such a CICD process should provide the means to accomplish non-functional requirements of a chaincode (e.g. code quality, integrity, security, performance). 5 Methodology This section presents an overview of the proposed methodology where we introduce chaincode as a Kubernetes service concept and then describe the design of our CICD flow based on the requirements specified in section 4. 5.1 Chaincode as a Kubernetes Service By default, a chaincode is launched as a Docker container by the peer when its definition is committed to a channel. Since v2.0, Hyperledger Fabric eliminates this tight coupling with the peer and enables a chaincode to be launched as an ChainKode: A CICD methodology for chaincode development in Hyperledger Fabric based on Kubernetes external service. However, this can potentially result in an administrative overhead, if an organization has multiple such chaincode services deployed in Docker. A container orchestration tool such as Kubernetes can be used to overcome this complexity and manage chaincode containers conveniently within an organization. We leverage this possibility in our methodology to deploy the chaincode as a Kubernetes service (CCaaKS). In particular, it reduces the content of a chaincode package (requires only metadata and connection details) and the involvement of peers in deploying chaincodes (only during the initial deployment and whenever the package is updated). Consequently, it decouples the SDLC of a chaincode from the peer life cycle and leads to the potential of integrating CICD pipelines for the development of chaincodes. However, this approach mandates a custom buildpack to be mounted on peers, which handles an external chaincode by skipping its local installation and allowing the peer to connect to an external endpoint. 5.2 Design of the CICD pipeline We designed our CICD methodology based on the requirements in section 4, such that it provides an automated pipeline to deploy a chaincode (as a Kubernetes service) with adequate levels of code-quality, integrity, security and performance. Static code analysis tools and linters can be used to validate the chaincode at code level whereas unit and functional tests should be implemented to verify the integrity of algorithms [ 24 ]. Once these validation tests have been passed, the pipeline should compile the source code, build a container image and push it to a container registry such that it provides versioning and portability to deploy in multiple instances if necessary. Subsequently, Software Composition Analysis (SCA) tools can be integrated to scan the security vulnerabilities of any external dependency used for the chaincode service (e.g. third-party libraries in the implementation, base image for hosting the container). We defined two environments as staging and production in order to adapt our methodology to industrial use cases. A chaincode image derived from the previous stages can be initially deployed to the staging environment and experimented with benchmarking tests, since performance is regarded as a crucial element in blockchain technology [ 25 ]. The results of these evaluations should be validated against a predefined set of threshold values (configured in a YAML file) to ensure that the performance of the chaincode complies with the expected metrics (e.g. throughput, latency). If all the steps succeed without any failure, we deploy the chaincode image to the production environment with a manual approval to incorporate human intervention in the overall CICD pipeline. This also transforms our approach to aContinuous Integration and Continuous Delivery methodology due to the involvement of a semi-automated process. In case of any failure (e.g. a threshold is not satisfied, network issue), a clean-up process should be executed to remove any persisted changes as applicable (e.g. delete image from the container registry). In addition, Git best practices should be adopted throughout the entire process to comply with further requirements (e.g. associate each commit with a change request for chaincode lineage, sign commits for non-repudiation, protect release branches for authorization). Figure 1 shows the overall flow of our designed methodology with essential tasks that ensure the non-functional requirements of a chaincode as described in this section. 6 Implementation This section provides the implementation-specific details of our methodology. First, we discuss the infrastructure setup of the proposed ecosystem which is followed by a comprehensive description of the implemented CICD pipeline. Finally, we describe the monitoring tools related to our context. Yasiru Rathsara Witharanage, Santiago Figueroa-Lorenzo, Nasibeh Mohammadzadeh, and Saioa Arrizabalaga Juaristi 6.1 Infrastructure 6.1.1 Hyperledger Fabric: From the perspective of a network, Hyperledger Fabric can be regarded as a distributed system with a set of distinctive service components (e.g. peers, orderers, CAs). Accordingly, we utilized Kubernetes to maintain our network infrastructure by deploying Hyperledger Fabric components as Kubernetes resources and thus enforcing their accessibility constraints at the Kubernetes layer (e.g. anchor peers as LoadBalancer Service, internal peers and organization CA as ClusterIP Service and Fabric CLI as Deployment). As described in section 5.1, we also deploy chaincodes as Kubernetes services to support their SDLCs. 6.1.2 Supplementary tools: In addition to the core components, we deployed Blockchain Explorer and its database dependency (PostgreSQL) in Kubernetes for monitoring the blockchain network. We also defined several tools to facilitate and enhance the CICD process of chaincodes in our ecosystem. This includes a Gitlab runner to execute jobs, a set of services (Metrics-Exporter,Prometheus and Grafana) to monitor pipelines, SonarQube as a static code analysis tool and Caliper to evaluate the performance of deployed chaincodes. In particular, we used a self-managed Gitlab runner to provide additional security and privacy during the execution of individual jobs. Since these tools serve as utility software in our methodology, we deployed them as Docker containers despite that Kubernetes can also be used for such deployments. Figure 2 illustrates a basic network architecture of these components which serves as a blueprint to extend with additional components (e.g. peers and orderers) if necessary. Fig. 2. Network architecture 6.1.3 Environments: We simulated a staging environment in our study by deploying a Kubernetes cluster with microk8s 2 in a Linux Container (LXC) which was hosted on a bare-metal server. Subsequently, we used this Kubernetes cluster to establish a network with Hyperledger Fabric components and Blockchain Explorer as shown in Figure 2. We further used the same server instance to deploy a Gitlab runner and monitoring tools as Docker containers. In contrast, we only deployed a Kubernetes cluster with a Hyperledger Fabric network on a separate bare-metal server as the production environment. Figure 3 shows these environments with the components stacked on each server. 6.2 CICD process As outlined in section 5, we designed a CICD pipeline with distinctive stages and jobs (Figure 4) to ensure compliance of design goals in our methodology. In the following subsections, we dive into the implementation details of these stages. 6.2.1 Validate: We initiated the pipeline with code validation tasks related to Golang, since one of our primary objectives is to support chaincode development in Golang. Therefore, we used its native linter, go-lint, in which a set of linter checks can be configured via .golangci.yml as required. In addition, SonarQube is used as a static code analysis tool to ensure the quality of the implementation by inspecting clean-code attributes and security vulnerabilities. 2see more details at https://microk8s.io ChainKode: A CICD methodology for chaincode development in Hyperledger Fabric based on Kubernetes Fig. 3. Environments with component stacks Fig. 4. Directed Acyclic Graph of jobs with stages We included unit testing in the subsequent job (go-test) as it is considered a key mechanism for validating chaincodes [ 24 ]. Since these tests highly depend on the specific chaincode implementation, developers are responsible for implementing the corresponding test files such that a predefined test coverage threshold is satisfied. In case of critical scenarios, this threshold value should be set higher such that all possible behaviours of the chaincode are investigated and implemented as unit tests. 6.2.2 Build: During this stage, the pipeline compiles, builds and pushes a chaincode (Docker) image to the Gitlab container registry. We utilized Directed Acyclic Graphs in our pipeline by defining validate jobs as pre-requisites to the build stage, such that it does not proceed with any insufficient source code. 6.2.3 Test: As discussed in section 5, SCA tools play a key role in delivering secure chaincode services. To this end, we used Gitlab’s inbuilt container scanning job to inspect for security vulnerabilities in container environments where a chaincode will be executed. This is further complemented with a dependency scanning job which analyses the security of third-party libraries (modules in Golang) used in a chaincode implementation. Both of these jobs include build stage as a prerequisite and generate the corresponding test results as artifacts in the CICD pipeline. 6.2.4 Pre-deploy: This stage deploys a chaincode to the staging environment by executing several tasks automated with bash scripts. In particular, the pipeline defines a set of variables (e.g. chaincode name, version, package ID, service endpoint, image tag) in .gitlab-ci.yml and constructs a Kubernetes manifest file for the chaincode. Subsequently, it mounts this file on the staging server and bootstraps the chaincode as a Kubernetes service with its package ID and service endpoint as environment variables. To enhance the security in our approach, we store the credentials of Gitlab container registry as Kubernetes secrets [ 26 ], which provides authentication when downloading chaincode images. Since dependency and container scanning jobs are defined as prerequisites of this stage, a deployed chaincode in staging environment can be assumed to be secure if the prior jobs are sufficiently capable of detecting vulnerabilities. 6.2.5 Evaluate: In our proposed methodology, we integrated Hyperledger Caliper to evaluate the performance of a chaincode under different transaction loads as specified by a set of workload modules (with benchmark tests). As similar Yasiru Rathsara Witharanage, Santiago Figueroa-Lorenzo, Nasibeh Mohammadzadeh, and Saioa Arrizabalaga Juaristi Fig. 5. Comparison between methodologies to unit tests, the responsibility of implementing workload modules should be delegated to developers such that the benchmark tests adequately cover the critical functions of a chaincode. As of v0.6.0, Caliper only supports workload implementations in Node.js which can still be used to invoke Golang chaincodes. In addition to these tests, benchmark and network information should be configured with at least one anchor-peer, which will be used by Caliper to connect to the Hyperledger Fabric network. The report generated by benchmarks is included as an artifact of the job so that it can provide further insights in reference to any supplementary documentation [16, 27]. We also included an automated job to verify the generated performance results against a set of constraints. This requires an additional YAML configuration file which contains a list of threshold values as shown in Listing 1 for each defined test and metric in Caliper (e.g. max. number of failures, average latency upper-bound, min. throughput). Once the performance tests are completed, their results will be evaluated against these thresholds to determine if the chaincode satisfies the expected levels of performance. thresholds : get −a s s e t : f a i l : 2 max−latency −sec : 1 . 5 avg−latency −s ec : 0 . 5 throughput : 800 Listing 1. Threshold configuration for benchmarking 6.2.6 Deploy: In the final stage of our proposal, we use the manifest file constructed in pre-deploy stage to deploy the chaincode to the production environment. At this point, the chaincode implementation along with its Docker image can reasonably be considered as a service which satisfies the defined levels of code quality, security, integrity and performance. However, we configured the deploy stage with a manual approval step to detect and handle any unforeseen complications prior to the final deployment of a chaincode. Figure 5 summarizes the flow of our methodology as compared to the default process of deploying a chaincode in Hyperledger Fabric. Despite we present our approach confining to Golang, it can also be used as a general framework to accommodate chaincode development in other languages. ChainKode: A CICD methodology for chaincode development in Hyperledger Fabric based on Kubernetes 6.3 Monitoring Continuous monitoring is regarded as a crucial component of SDLC and in our context, we outline this in terms of blockchain network, chaincodes and CICD pipelines. Figure 2 shows how we integrated these monitoring tools into our ecosystem. In particular, we used a third-party tool, gitlab-ci-pipelines-exporter 3 , to collect metrics from the CICD pipelines and subsequently export to a Prometheus instance. When configured with a Gitlab Access Token and project repository, the exporter service pulls metrics exposed by Gitlab APIs which primarily relate to CICD jobs, pipelines and deployments (e.g. job completion, pipeline status, deployment duration). We extended this observability by exposing metrics from our self-managed Gitlab runner to provide further insights related to the actual servers running CICD pipelines such as their memory utilization, CPU time and Golang specific metrics (e.g. garbage collection, heap and stack allocations). Prometheus scrapes metrics from both of these jobs (i.e. gitlab-ci-pipelines-exporter and runner) and delivers them to a Grafana instance for visualization with dashboards (Figure 6). Fig. 6. Grafana dashboard for CICD pipelines Blockchain Explorer serves as a monitoring tool for the Hyperledger Fabric network. Specifically, it provides a dashboard as shown in Figure 7 with network information (e.g. status of nodes), committed blocks, deployed channels and transaction details (e.g. type of a transaction, read-write set, endorsers). Further, it allows to inspect the installed chaincodes and information about their smart contracts (e.g. title and version, parameters and their required formats, return data types, schemas with their properties). Fig. 7. Blockchain Explorer 7 Evaluation We conducted several experiments to further investigate the performance of our implemented methodology in different conditions as described in the following subsections. 3https://github.com/mvisonneau/gitlab-ci-pipelines-exporter