Towards DevOps: Practices and Patterns from the Portuguese Startup Scene

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Towards DevOps: Practices and Patterns from the Portuguese Startup Scene Carlos Manuel da Costa Martins Teixeira DISSERTATION Mestrado Integrado em Engenharia Informática e Computação Supervisor: Prof. Hugo Sereno Ferreira Co-Supervisor: Tiago Boldt Sousa September 5, 2016 Towards DevOps: Practices and Patterns from the Portuguese Startup Scene Carlos Manuel da Costa Martins Teixeira Mestrado Integrado em Engenharia Informática e Computação September 5, 2016 Abstract The DevOps movement aims to facilitate the work of both Developers and Operators by creating a new culture of collaboration between the two as well as introducing new practices and methodologies. With initial discussions going back as far as 2008/2009, ideas regarding the DevOps movement and what it means are still evolving and, eight years later, are still topics of discussion even within the community. Combining that with the fact that some studies are showing a significant number of companies adopting DevOps and an equal amount looking into doing the same, it becomes worrying that DevOps literature is still largely based on personal opinions rather than scientific or academic literature. This thesis represents a step toward a better and broader understanding, within the scientific community, of DevOps and its practices. With the objective of enabling more businesses to adopt DevOps and do so with less uncertainty we identify, in this thesis, 13 DevOps related patterns extracted from 25 Portuguese startup companies. Then, we conclude that the progress made with this thesis should be extended and we point out the direction to do so. i Contents 1 Introduction 1 1.1 Context ....................................... 1 1.2 Problem....................................... 1 1.3 Motivation...................................... 2 1.4 Goals ........................................ 2 1.5 Outline ....................................... 2 2 State of The Art 3 2.1 CloudComputing.................................. 3 2.1.1 Definition.................................. 3 2.1.2 Deliverymethods.............................. 4 2.1.3 ServiceLevels ............................... 4 2.1.4 Benefits................................... 5 2.2 DevOps ....................................... 5 2.2.1 Theculturallayer.............................. 6 2.2.2 The methodology layer . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.3 Thepracticeslayer ............................. 6 2.2.4 Thetoollayer................................ 7 2.2.5 Thefullonion................................ 7 2.2.6 ADevOpsdefinition ............................ 7 2.2.7 DevOpsBenefits .............................. 8 2.2.8 Patterns................................... 8 2.3 The Portuguese startup scene . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Apatternlanguage ................................. 10 3 Towards DevOps 11 3.1 Methodology .................................... 11 3.1.1 Adjusting Granularity . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.2 Collecting the information . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.3 Definingasample ............................. 12 3.1.4 Sample Characterization . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.1.5 Data Processing & Pattern Extraction . . . . . . . . . . . . . . . . . . . 14 3.1.6 Patterns................................... 15 4 Patterns from the Portuguese Startup Scene 16 4.1 Teamorchestration ................................. 16 4.2 Communication................................... 17 4.3 Version Control Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 ii CONTENTS 4.4 Cloud ........................................ 20 4.5 Reproducible Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.6 Deployingnewinstances .............................. 23 4.7 Scalling ....................................... 24 4.8 ContinuousIntegration ............................... 25 4.9 JobScheduling ................................... 26 4.10Auditability..................................... 27 4.11Alerting....................................... 28 4.12ErrorHandling ................................... 28 4.13CodeReview .................................... 30 5 Validation 31 5.1 Methodology .................................... 31 5.2 Ventureoak ..................................... 31 5.3 Objectives&Metrics................................ 32 5.4 InitialState ..................................... 32 5.5 AdoptionProcess.................................. 33 5.6 FinalState...................................... 33 5.7 Discussion...................................... 34 6 Conclusions 35 6.1 Contributions .................................... 35 6.2 FutureWork..................................... 35 6.2.1 Specializing the identified patterns . . . . . . . . . . . . . . . . . . . . . 35 6.2.2 DevOpsmonitoring............................. 35 6.2.3 Furthervalidation.............................. 35 6.2.4 StatisticalAnalysis............................. 36 A Annexes 39 A.1 PatternMap..................................... 39 A.2 ConceptsGrouping ................................. 40 A.3 InterviewGuide................................... 42 A.4 Companie Interviews Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 iii CONTENTS iv List of Figures 2.1 The DevOps onion [7] ............................... 7 3.1 Studiedcompaniessize............................... 14 3.2 Identified concepts and subsequent refining . . . . . . . . . . . . . . . . . . . . 14 A.1 Interview practices frequency extraction. . . . . . . . . . . . . . . . . . . . . . . 41 v State of The Art •On-demand self-service : A consumer can unilaterally provision computing capabilities, e.g. server time and network storage, automatically without requiring human interaction with each service provider [18]. •Broad network access : Users must be allowed to access resources through standard mechanism [18]. •Resource pooling : A multi-tenant model should be used in order to serve multiple users. Resources are allocated dynamically meaning that users do not know where, physically, the allocated resources are [12] •Rapid elasticity : Resources can be elastically allocated or deallocate. This should be done automatically [18]. •Measured service : The usage of resources should be measured providing transparency in the provider-client relation [18]. 2.1.2 Delivery methods In regards to their accessibility, it is common to identify three main categories of clouds [29] : •Public Clouds : Public clouds are a pool of resources hosted by cloud providers who rent them to the general public. This resources can be accessed over the Internet and are shared among users. •Private Clouds : Private clouds are usually administered and used by the same organization. Alternatively a third party can also be hired to manage the resources. The main difference between public and private clouds is the usage of the resources. Private clouds resources are only used by one company as opposed to public clouds were resources are shared. •Hybrid Clouds : Hybrid clouds combine both the private and public concepts. When using a hybrid clouds approach, infrastructure is divided by the two types of clouds meaning that some modules may be hosted in the private space and others on the public one. •Virtual Private Cloud : Virtual private clouds are an alternative to private clouds. This type of cloud are essentially a public cloud that leverages virtual private network (VPN) technology [29] allowing users to combine characteristics of both public clouds and private clouds. 2.1.3 Service Levels In terms of service levels cloud computing can be classified in regard to the provided abstraction. The main categories are the following [27] : 4 State of The Art •SaaS - Software as a Service (SaaS) gives users access to a platform usually through a web client without the need to download or install software. The user is able to use the provided software instantly and virtually everywhere. Applications of this model include messaging software, email services, collaborative platforms, etc. •PaaS - Platform as a Service (PaaS) allows its users to quickly deploy applications with little to no configuration. In this type of platform environments are usually setup previously or configurable. PaaS users should nevertheless expect only to be able to deploy applications or software supported by the provider. •IaaS - Infrastructure as a Service (IaaS) represents the lowest abstraction made available by cloud providers. In this model the user is able to configure and access a machine directly without constraints. This machine, usually a virtual server managed by the provider, can be configured and maintained by the user. This model is used when applications are complex and therefore need complex configurations. 2.1.4 Benefits The main advantages of cloud computing for its users can be summarized as following: •Monetary Efficiency - Cloud providers allow users to keep their resources to the needed minimum. By allowing users to quickly and easily increase/decrease the allocated resources amount and billing clients only for the resources used, cloud providers are good way to save money and spend only the needed amount [12,18]. •Scalability - usually through a public API of some kind most cloud providers allow for the quick increase or reduction of resources [18]. This enables businesses to quickly go from zero to millions of users with minimum overhead. Additionally, because processes related with the management and configuration of cloud servers can be automated, it is usually possible to manage large systems with small teams [17]. •Maintainability - Cloud Providers are responsible for the maintenance of all the hardware and infrastructure aspects. Cloud computing users therefore do not need to worry about updating the hardware or maintaining the physical infrastructure. This enables users to focus their resources in improving their product rather than improving the structure that supports it [12]. 2.2 DevOps In Why DevOps Is Like An Onion [7], Dave Sayers chooses to use the analogy of peeling an onion, with each layer representing a different concept, in order to describe DevOps. Further developing upon his initial idea, in this section, we will try to introduce DevOps using that same analogy. 5 State of The Art 2.2.1 The cultural layer Looking at some of the first DevOps efforts we see that a lot of them aimed to create a better articulation between Developers and Operations [9,2]. Being two distinct departments with different work methodologies and objectives, this meant that some cultural changes had to happen in order for the two departments to be able to create a common understanding of each other worlds [2]. The main cultural values associated with DevOps are, as a result, values that promote cooperation and communication. These are some of the ones that we identified: •Respect [8,2] •Trust [13] •Collaboration [8] •Sharing [28] 2.2.2 The methodology layer The DevOps movement does not define a specific methodology. Nevertheless, when we look for instance at CAMS 2.2.6 or at some of the practices like Continuous Deployment it becomes clear that, although a methodology is not defined, the one chose should enable an iterative and continuous improvement focused approach. 2.2.3 The practices layer Broadly speaking, DevOps practices gravitate around the automation of processes. From the setup of environments on the developers machine, up to the deployment phase, the capacity to automate repetitive tasks is a key feature of DevOps. Common practices associated with the DevOps movement include: •Continuous Integration - Regular integration of software helps discover risks associated with the integration of the software. [3] •Continuous Deployment - Deploying often means that less functionality are deployed each time. This makes deployments more manageable and in turn safer [2]. •Continuous Monitoring - Continuously monitoring infrastructure and applications allows for the early detection of bugs [6]. It can also be a way to identify areas that can be improved [28] •Defining infrastructure as code - By defining infrastructure as code it is possible to increase reliability and consistency [17]. 6 State of The Art •Using feature Flags - Feature flags are special flags that toggle features on and off. This can improve error handling by allowing faulty features to be turned off. Feature flags also enable more complex schemes of operation where certain functionality are launched, but are not displayed or are displayed just to certain users. Once it is observed that the new functionality works, the flag can be turned on and the new functionality is available [4]. •Others - As we will see in 2.2.6 there can be a great number of practices that can be considered DevOps practices. 2.2.4 The tool layer Tools allow for some practices to be more effective. In this section we will present some of the existent categories of tools. This categories are presented in [16] and are currently being maintained and increased by the DevOps community: •Source Control Management (SCM) •Continuous Integration (CI) •Deployment management •Cloud platform and infrastructure •Monitoring •Repository Management •Infrastructure Provisioning •Release management •Logging •Security •Build •Testing •Containerization •Collaboration •Database Management 2.2.5 The full onion Figure 2.1: The DevOps onion [7] As it was stated by Paul Hammon and John Allspawn at their 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr [2] presentation, tools and processes alone are not enough and the cultural change should be the first step to take towards DevOps. Because of this, the outermost layer in the onion (fig. 2.1) represents the most important layer and without peeling it will not be possible to take advantage of the inner layers [2]. 2.2.6 A DevOps definition DevOps vastness can be seen in the enormous amounts of tools categories identified in 3.1.1 related with DevOps. 7 State of The Art As one might expect from this vastness of areas that DevOps touches, there are still difficulties to properly define DevOps. In order to reduce this lack of definition, we will use throughout this thesis a conjugation of two definitions for DevOps. The first definition is from DevOps: A Software Architect’s Perspective [4] and it states that: DevOps is a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality. We chose this definition because it allows to easily classify something as being DevOps or not, i.e. one would only have to ask himself if a practice or cultural aspect will allow for the reduction of time since committing a change until that change is in production, if it does, then it is DevOps. Nevertheless, we find this definition to be a bit empty in the sense that by only reading it one would not be aware of the aspects that are associated with Devops. As a workaround for this problem we use a second definition based on the CAMS acronym. The CAMS acronym [28] defines DevOps as being a Cultural movement where Automation and continuous Measurement of processes and people are promoted and where the later can serve as an input for the Sharing of problems and new ideas. As new ideas appear to solve the identified problems, new measurements can be made to further identify problems further improving the overall process. 2.2.7 DevOps Benefits In a recent study [10] the following benefits were identified: •DevOps projects are believed to accelerate in 15%-20% the ability to delivery of capabilities to the client •Adopting DevOps allows business to practice Continuous Delivery. •The average cost of a critical application failure per hour is $500,000 to $1 million (DevOps can help reduce application failures). •The average cost percentage (per year) of a single application’s development, testing, deployment, and operations life cycle considered wasteful and unnecessary is 25% (DevOps can help automate some repetitive tasks and by doing so reducing some of the waste associated with those tasks) 2.2.8 Patterns Some previous progress has already been made regarding the identification of DevOps related patterns. We will summarize this progress by listing the identified patterns and briefly describing them: 8 State of The Art •Store Big Files in Cloud Storages[6] - Instead of creating and managing a system to store large files, or storing them in database columns, store them in a Cloud Storage2 •Queue based solution to process asynchronous jobs[6] - When there are tasks that take a long time to complete but users still expect a quick response, create a new Job instance in a queueing service and then have a service performing those tasks. When finished, post the result of that Job somewhere acessible to the user and notify the user that the task is done. •Prefer PaaS over IaaS[6] - For non technology companies, PaaS is usually preferable because it will give them lot of functionality without the need for configuration. This will allow them to simply focus on their core business. •Load Balancing Application Server with memcached user sessions[6] - Use a load balancer in front of your application servers. This severs will handle sessions using memcached which means that if a application server goes down or if a new application server is needed, it will be able to use the user session. •Email delivery[6] - Rather than implementing your own SMTP solution, use cloud mail delivery services which provicde REST API’s to send emails. •Logging[6] - Having multiple servers you need a way to consolidate your application logs. In order to do so, you should use a cloud based log service. •Realtime User Monitoring (RUM)[6] - Monitor user behaviour in order to find possible bugs or errors. •The isolation by containerization pattern [22] - Use a container to package the applications and its dependencies and deploy the service within it. •The discovery by local reverse proxy pattern [22] - Configure a service port for each service, which is routed locally from each server to the proper destination using a reverse proxy. •The orchestration by resource offering pattern [22] - Orchestrate services in a cluster based on each host’s resource-offering announcements. 2.3 The Portuguese startup scene Motivated by a recent investment in innovation and entrepreneurship, Portugal startups have been growing their position in the global startup scene [5]. A study from 2015 [23] in which the Portuguese startup scene was analyzed, revealed that there were already 40 technology scaleups 3operating in Portugal at the time. The same study stated 2A storage system provided by a cloud provider. 3Scaleups are companies that raised more than $1M funding (since foundation) and had at least one funding event in the last five-year period 9 State of The Art that this startups were able to raise a large portion of the received investment from international investors indicating, therefore, that the reach and scale of this startups was broader than just the national arena. Additionally, it is also indicated in the study that Porto and Lisbon are the main centers of innovation, encompassing 70% of the total of existing scaleups. In addition to the scaleups identified other smaller scale startups exist. Some of this startups are currently being incubated in incubators around the country like UPTEC 4and Startup Lisboa. This incubators had, at the time of this study, more than 300 companies [26,24] under their wing. 2.4 A pattern language Cristopher Alexander wrote A Pattern Language[1] in 1977 and in doing so, introduced the idea of representing knowledge on the form of a pattern language. Since then, pattern languages have been used in several areas [19] including software development where they were used to describe common architectural solutions [15] or recurring software design choices [14]. A pattern is a recurring solution to a problem that lives inside a specific context [19]. Pattern languages are collections of patterns that are related to each other by virtue of solving the same problems or parts of a solution to a larger, partitioned problem. [19]. There are some key characteristics that should be taken in consideration when writing patterns. In A pattern language for pattern writing [19], some of this characteristics are presented in a form of patterns themselves, they can be summed up as follows: •patterns should identify a pattern user operating in a context. This pattern user should have and identified problem that he solves using a solution that takes into consideration some of the constraints defined by the context. •patterns should be easy to read and understand and should not force the reader to read the entire pattern multiple times in order understand it. Because patterns represent a way to share recurring solutions to a problem [19], pattern languages help reduce re-discovery and re-invention of concepts and functionality as well as avoiding common pitfalls that are learned from experience [21]. 4Science and Technology Park of University of Porto 10 Chapter 3 Towards DevOps In this chapter we will look at the approach used in order to solve the problem described in 1.2. We will start by looking at how we defined the sample from which we extracted information and then how we handle that data in order to produce the final results. 3.1 Methodology In 2.2 we saw that the DevOps movement emerged in the software development1community. Having this factor into consideration and knowing that there is still a significant lack of scientific information regarding the subject, we choose to look for a solution within the development community. We believed, before conducting this study, that not only would they be able to provide us that information, but because they were using this techniques/tools in their daily activities, it would serve also an extra layer of assurance. 3.1.1 Adjusting Granularity The vast number of existing tool categories as well as the fact that there are a lot of possible practices that can be considered part of DevOps 2.2.6 means that fully capturing the ideals and ideas related with it would not only be a tremendously hard task but would also not fit in the length of this thesis. Taking this into consideration and in order to avoid over specializing this thesis in some particular area, we set the desired granularity for the study by defining that we would only focus on categories of practices and if needed tools e.g. we wanted to know what steps did the Continuous Integration server run, but we were not interested in fully documenting the tool or measuring the time it took to run a full integration cycle. 1software development should be seen in this context as both the development, maintenance and any other tasks related with the creation of software 11 Towards DevOps 3.1.2 Collecting the information Taking into consideration the vastness of topics that DevOps 2.2.6 touches, we knew that it would be hard to create a form that covered that extensively. It was theorized, at this point, that not all startups would have similar levels of maturity(we later observed this to be true A.4). This meant that even if we could create this form, it would be too extensive for some cases and we would still have to create an all englobing form to capture all the information. The chosen methodoly adopted was to do a more exploratory type of interview. This idea lead to the creation of a script rather than a form that would guide the interviews. This script A.3 had 5 major sections: •Product - The product section would try to understand, first what did the company do and secondly if there were any kind of special requirements that would influence the choices made by the company. •Team Management - Team sizes, interactions, project management techniques would be analyzed here. •Software delivery pipeline - In this section, we identified if teams did Continuous Integration, how did they handled the creation of environments for each of the pipeline states and what teams did what in each state. •Infrastructure Management - We tried to capture how the companies handled their infrastructer. Did they use the cloud? Which processes did they automate? •Monitoring & Error Handling - With this section we aimed to understand if the companies were monitoring their infrastructure, how did they do it and, when errors were detected, how were they responding? 3.1.3 Defining a sample As it was seen in 2.3, Portugal has a rich startup community. Startups have strict contraints regarding the ammount of resources they have at their disposal and have, as a result, aditional incentive to automate as many tasks as they can. Being small (in terms of staff), communication and cultural aspects are usually guided towards cooperation as this is a key factor in allowing small teams to handle large projects. Finally, the fact that startups have as their objective to scale, further highlighs the need for automation and cooperation. This conjugation of factors meant that the mindset of startups was aligned with the DevOps one and startups are therefore a good place to look for information that can be directly linked to DevOps. Having more than 300 startup companies 2.3 from which to choose, we needed a way to filter out companies that may not be relevant to our study. 12 Towards DevOps With that objective we first created a list of all the startups companies we knew that operated in Portugal. This list was create by looking at some of the Portuguese startup incubators and by extracting the list of companies. From those 300 startups, we attempted to identify which ones were doing software development. To do so we looked, when available, at the company web page and tried to determine if the company product was software related or not. This approach reduced the number of companies to 155. Then, we prioritize which companies were better or worst for our study. To do so we created a compound evaluation metric that would allow us to rank companies. We created 5 metrics to do so: •Cloud Usage - We created three possible values for this metric. If a company used cloud services, we would give the company 2 points. If we were not sure if a company was using cloud services, we would give it 1 point. If we know the company was not using cloud services, we would give it 0 points. With this metric we attempted to distinguish between, for instance, companies that were developing hardware solutions from those that were developing more software oriented solutions. •SaaS/PaaS offering - Having the same point attribution schema as the Cloud Usage metric we believed that if a company had a SaaS/PaaS product, it would need to have some sort of automation put into practice as it would need to be able to scale if there was a sudden increase in clients. •Company Size - We create four possible values for each metric. Companies could have 0,1,2,3 points if they had respectively less than 5 members, between 5 and 15 memebrs, more than fifteen members or more than fifteen members and several teams. •Subjective Appreciation - This metric would reflect the overall opinion of the company that we developed when searching for information for the other metrics. Some common factors that influenced this metric were for instance the fact that some companies had listed staff members working on software development task, or if the company website was down. The resulting ranking, created by summing all the factors, is not supposed to be seen as a precise way to accurately compare companies i.e. the second company maybe as relevant as the first one, but rather as a way to prioritize them, i.e. the first company surely is more intersting to study then the last one. In the end, we contacted a total of 60 companies from which we were able to interview 25. 3.1.4 Sample Characterization The sample, as seen in fig. 3.1 was mostly composed of small companies with no more than 35 employees and with half of the sample size to have less than 15 members. The companies also conducted a substancial amount of their business over the internet either through a browser application or a mobile app or a combination of both. 13 Patterns from the Portuguese Startup Scene It is possible to use the Code Review 4.13 pattern even without using any sort of version control. Using one, will nevertheless help organize the process. In the same way, Error Handling 4.12 can also be improved by using the correct Version Control Organization as it will allow you to, for instance, traceback what was the last live version or what was the code introduced in a hotfix. 4.4 Cloud CONTEXT Your company and/or your product needs to acquire computing resources in order to perform tasks like: •performing large complex operations. •supporting a website or a web platform. •any other kind of computing task This resources should be accessible and configurable and you believe that you do not need to physically connect to them it in order to control them. PROBLEM Owning computing resources is essential or at least advantageous to you or your business so the question is how do you acquire and maintain computing resources in a efficient way ? FORCES •Upfront Costs. Acquiring hardware may require significant upfront costs. •Maintenance. Depending on the ammount of hardware you have to manage, a person, team or department may be needed to maintain it. •Customization. Different services may provide different levels of customization/control. •Elasticity. You may want to scale the amount of allocated resources to match your needs. SOLUTION Solutions for this problem can be seen as belonging to three categories: 20 Patterns from the Portuguese Startup Scene •Purchasing and maintaining your own hardware allows you to have full control over your infrastructure. You can control, for instance, in which machine does a specific applications run, how that machine is configured, etc. This option represents therefore the highest level of customization customization. On the downside, this options usually means that you have to purchase hardware (upfront costs)and that you either acquire more resources than what you need or you risk not having enough resources to answer increasing computing needs. Additionally you will have to create and support a team or department to manage the maintenance of your infrastructure. •When using IaaS there is no need to purchase anything upfront. In this pay-as-you-go model you only pay for what you consume and you are able to elastically increase/decrease the size and/or number of resources you are using. With this model, the responsability for maintaining and setting up infrastructure is shifted to the cloud provider. IaaS providers usually allow you to have some degree of customization like choosing the operating system and resources available(CPU cores, memory, etc) but lower level configurations will not be available. As a matter of fact, most cloud providers use virtual machines to run their clients applications meaning that you will not be able to tweak network configurations or choose exactly wich machine runs what. IaaS reduces therefore the level of control in comparisson to hosting your own infrastructure. •Paas follows the same pay-as-you-go model as as IaaS meaning that you can also increase/decrease the size and/or number of resources you use. PaaS represents the smallest level of customization but at the same time allows you to use already pre-configured environments in which you can run your applications. RELATED PATTERNS When using the Cloud, choosing for instance a PaaS alternative may, depending on your choice, prevent you from being able to fully reproduce the production environment. Different levels of support exist, as well, for different Reproducible Environments 4.5 techniques. Some providers may, for instance, create pre-programed container ready environments while others may not allow you to run your own Virtual Machine. 4.5 Reproducible Environments CONTEXT When you have several environments (e.g. production,staging,devevlopment) or multiples instalations/instances of your software it is desirable to be able to guarantee that all instances work the same way. With this goal in mind you have identified that the environments where your instances 21 Patterns from the Portuguese Startup Scene run is a key factor when trying to antecipate how does the software behaves. This consistency is important because it will allow you to have reproducibility and will give you some guarantees when you desire to increase the number of instances of your software. PROBLEM How do you guarantee that the environment where you setup your application is consistent across instances? FORCES •Size. Having a complete copy of your environment (OS’s, libraries, etc) may create large files that may be hard to move around. •Parallelization. You may want to have several running instances of different environments in the same machine. •External Dependencies. Some of your dependencies may be fetched from external providers. •Update. You may want to update or change the environment. •Infrastructure. Depending on your choice for Cloud 4.4 you may have more or less access to your environment settings. SOLUTION •Using scripts usually means describing you environment in the form of a text file. This file is then executed/interpreted inside an environment in order to create the desired state. Because scripts do not contain the dependencies you need, you usually have to fetch external dependencies from external providers. If for instance one provider is down, your script will not be able to complete. In case you need to update your setup, depending on the change and the tool you use, you may need to run the script again, run only the part you modified or reset the machine and run everything again. Because scripts are just text files they often represent the most efficient alternative in terms of size. •Using virtual machines, an environment can be created by creating an image of the operative system with all dependencies installed. This image can then be replicated across different projects. If the need to change dependencies arise a new image can be created. Because of their size, usually only one or two VM’s can coexist in the same computer at the same time (parallelization). 22 Patterns from the Portuguese Startup Scene •Containers are a lightweight alternative to Virtual Machines. The setup process is pretty similar to VM’s but the generated representation/image is much smaller. This decrease in size comes from the fact that containers share resources with the host machine and even among containers. By doing so, it is usually possible to have multiple containers running in the same host. Some cloud providers already have options were they support containers natively. RELATED PATTERNS Having chosen to have environment consistency across your instances means that the when creating new instances(Deploying new instances 4.6) and building your software (Continuous Integration 4.8) the same choices should be used. 4.6 Deploying new instances CONTEXT You have decided to increase your computing resources horizontally in order to increase your hability to handle a bigger load of tasks. Depending on what type of Cloud you choose to use new resources were allocated but you still need to have your application running on those resources. PROBLEM How do you deploy your application in a reproducible and consistent way? FORCES •Reliability. Deployments must be reliable. •Speed. Deployments should not waist time. •Reproducibility. Deployments should correctly articulate with your environment setup method. SOLUTION In order have an efficient deploy both in terms of reliability and speed you should use the same approach used for setting up environments (Reproducible Environments 4.5). •If you have chosen Containers you can simply pull the container from a container registry and run it in your new instance. With this approach you will have a high degree of certainty that your instance will behave as you predict. Because containers are generally lightweight you will be able to download them fairly quickly. You will, nevertheless be dependant on you container registry service. 23 Patterns from the Portuguese Startup Scene •If you have chosen Virtual Machine you can create and image (AWS lets you crete a Virtual Machine using their services) and then deploy it to all your instances. •If you have chosen Scripts you can simply run the script in your target machine. RELATED PATTERNS Depending on your choice for Reproducible Environments 4.5, you should use the appropriate choice to deploy your code. You should be aware of the cost of this choice because if you want to scale horizontally, your performance maybe tied to the velocity with which you can provision new environments. 4.7 Scalling CONTEXT Having a 1:1 ratio between your needs and your resources may be easy to achieve if your needs are fixed in time. If, however, your needs fluctuate as a result of, for instance, new users accessing your application you would want to be able to increase or decrease (in case users numbers drop) the resource allocated. PROBLEM What strategy do you choose to increase your computing power ? FORCES •Costs. Costs are a factor. •Downtime. You want to change the allocated resources quantity without having to stop the existing application(s). •Scalability Limit. You may not have an upper limit for the amount of resources you will be using. SOLUTION Usually if your are using the Cloud 4.4 you can easily allocate new machines or increase the CPU cores, RAM, Disk Space, etc of your current machine(s). This two options represent the two existing approaches to scale your computing resources. The first one (increasing the number of machines) is usually refered to as Horizontal Scalling and second approach (increasing the resources of each machine) is usually refered to as Vertical Scalling. 24 Patterns from the Portuguese Startup Scene Horizontal Scalling usually is the cheaper option and allows for no downtime when upgrading (the existing machine can be put into production while the old one is running). This approach will also allow you to scale virtually without a limit. On the downside, this approach will force to have some considerations in mind concerning state keeping. If you have a need to keep sessions, for instance, and you are storing them in the machine, the new machines will not have access to that. Vertical Scalling is usually more expensive and depending on your Cloud 4.4 provider may have associated downtime. Verical Scalling also has a maximum amount of resources you can allocate to a single machine. On the upside if you scale vertically(and have only one instance) you can keep the state of your application inside your machine. Both approaches can be combined in order to accomodate your needs. 4.8 Continuous Integration CONTEXT There are several people contributing code to your application. PROBLEM Having several people collaborating into the same project can be challenging. If a developer, unaware that is introducing an error, pushes code into the team repository a long time may pass before the error is detected. Once detected, the error cause must identified and, because the code that introduced the error was pushed a long time ago, it may not seem obvious where the error is. FORCES •Running your entire test suit before pushing code may take to much time. •It can be challenging to setup an environment simillar to the production one in your local machine. •Your environment may need to be different from the production one(you may need some extra tools to aid you developing). SOLUTION Use (or develop) an automatic continuous integration(CI) system. This system should detect when code is pushed to your repository and then run the following steps: •Building your software consists in, depending on your choice for Reproducible Environments, building your environment, then fetching all required dependencies and finally compiling the code(if needed). 25 Patterns from the Portuguese Startup Scene •Test your build. When your build is successfull you should run your test suite against that build in order to check if everything is running according to plan. •Notify If any of the previous steps fails you should notify the developer that checked the code and any other people to whom the build integration status is relevant. Because this steps are run automatically, developers do not have to wait for tests to run or builds to complete. RELATED PATTERNS The Continuous Integration pattern can use the Communication 4.2 defined channels to send its messages. The test and build steps of this pattern, shoud be done in a environment equal to the production and development one and should follow the chosen Reproducible Environment 4.5 strategy. 4.9 Job Scheduling CONTEXT Sometimes there are tasks that, due their complexity may take a long time to finish. Cases may also exist when you have tasks that you want to schedule for later. Both this problems can be solved by scheluding jobs to be run when possible or later. PROBLEM How do you setup your infrastructure to handle this cases? FORCES •Having a fixed set of resources for dealing with scheduling tasks may not be cost effective. •Your load may vary during the day. SOLUTION You may launch new instances of your infrastructure to handle each of you tasks or batches of tasks. Each new instance receives the desired tasks and does the needed computation. When the task as been computed the new piece of infrastructure should be shutdown. Alternatively you can have a set of daemons running alongside your applications that handle this tasks. Tasks are generally fed through a queing system altough you can also store them in a database. 26 Patterns from the Portuguese Startup Scene RELATED PATTERNS When you are launching new pieces of infratructure to handle your jobs the effectiveness of that technique may be dependant on how fast you can Deploy new instances 4.6. 4.10 Auditability CONTEXT As applications grow identifying potential problems within your infrastructures will become increasingly difficult. If you have several machines and/or different possible points of failure you can not predict or assume that everything will always go without incident and you will therefore have to be prepared. Building a robust system may seem enough but is not. When problems appear (and they will appear) being able to identify them ,where and why they appear is essential for the resolution of those problems. PROBLEM What metrics should you extract and what should you do with them? FORCES •Extracting too many metrics may cluter your hability to effective analyse them. •You may want to keep an history of how your system behaved. •You want to have information about the current state of your service. SOLUTION Monitoring your application health can be done by using your own or external tools. Some cloud providers even provide you with a health view that tells you if your machines are healthy and running. Identifying some key indicators and some metrics is also important, by defining thresholds for each metric you can setup different levels of alerts for your teams. This way you can tackle problems as soon as they happen. Additionaly some indicators can also trigger automatic responses (e.g. if a platform is taking to long to answer requests you may launch new resources to distribute the traffic). RELATED PATTERNS You must be monitoring some metrics in order to create alerts. This mean that the Alerting 4.11 pattern will only exist if there are metrics being monitored. 27 Patterns from the Portuguese Startup Scene 4.11 Alerting CONTEXT You have defined a set of metrics for checking the health of your application. For some of those metrics when values reach a certain level a solution can not be automated (e.g. server repetitive failures). You still would want an immediate responese to that alert in order to make sure your services will not go down or in order to put them back on. PROBLEM Who should you inform? FORCES •People may not be available to answer alerts or may be unreacheable. •Alerting everyone may solve your problem quickly but may not be needed. SOLUTION Notifying can follow three main strategy. The first one is to wake everyone up. This approach is wastefull and as you usually do not need your entire team to solve the problem. The second one is always notifying the same person. This person should preferably be someone capable of diagnose the origin of your problem and then solve it or contact someone that can. The third option is two have a system were the responsability of handling errors rotates among the team members. Alerts can be sent using email, calling people, sending an sms and/or sending a notification to the person you want to notify. 4.12 Error Handling CONTEXT It is important not only to be able to detect errors but also to be able to respond to them in a proportional way. As someone involved in a team developing a product, being able to find the best way to handle a crysis may prove to be fundamental. PROBLEM In case of error what should the response be? 28 Patterns from the Portuguese Startup Scene FORCES •Impact. Errors have different degrees of impact. •Hard to find bugs. The cause of an error may not be easy to find. •Impossible rollbackSometimes you can not use a previous backup (e.g. when you have removed a column from your database). SOLUTION Handling and error is a delicate task. There are several things that need to be taken into account. If an error as a direct and significant impact in your applicaion (e.g. there is an error that allows people to login without checking the users passwords) you would want to respond as quickly as possible. In this types of cases you can rollback to an older version of your software. The rolling back effectiveness is nevertheless constrained by the speed with which you can do it and by the fact that you may not be able to do it. Strategies for rolling back you application can be of two types: •Deploying the previous version: You can order your system to deploy a version of your software that you know works. •Keeping a backup : You can keep a backup of your application/infrastructure and if an error is detected you can switch the DNS servers to point to your old infrastructure. When you cannot roll back and/or the error you detected does not have a substancial impact, you can try to find and fix the problem. Depending on your choice for Version Control Organization 4.3 you can create a newer hotfix branch and work on that. In the end, when you have found and fixed the problem, merging that branch with master and deploying the version will have fixed your problem. RELATED PATTERNS When you choose to rollback of to do a hotfix, you will be limited by the time it takes you to change the running version on your machines. If this change includes changing the environment you will need to update it. In order to do so, you will probably have to use the same method you defined for Deploying New Instances 4.6. Additionally, rollbacks may depend on the fact that you can detect what your last working software version was. This information can be store in your version control system if you used tags for instance. Information about the hotfix (if you chose to do one), can also be stored in you version control system(Version Control Organization 4.3) 29 Conclusions 6.2.4 Statistical Analysis A statistical analysis of the practices and methodologies employed by startup companies should also be developed in order to validate if the presented patterns are indeed a common practice among them. 36 Bibliography [1] C. Alexander. A pattern language: towns, buildings, construction. Oxford University Press, 1977. [2] J. Allspaw and P. Hammond. 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr. [3] S. S. And and B. Coyne. DevOps For Dummies, volume 53. John Wiley & Sons, Inc., 2015. [4] L. Bass, I. Weber, and L. Zhu. DevOps: A Software Architect’s Perspective. 1 edition, 2015. [5] A. Coleman. Portugal Discovers Its Spirit Of Entrepreneurial Adventure, 2015. [6] D. Cukier. DevOps patterns to scale web applications using cloud services. Proceedings of the 2013 companion publication for conference on Systems, programming, & applications: software for humanity - SPLASH ’13, (Figure 2):143–152, 2013. [7] Dave Sayers. Why DevOps Is Like An Onion, 2013. [8] J. Davis and K. Daniels. Effective DevOps, volume 1. 2015. [9] P. Debois. Agile infrastructure and operations: How infra-gile are you? Proceedings - Agile 2008 Conference, pages 202–207, 2008. [10] S. Elliot. DevOps and the Cost of Downtime: Fortune 1000 Best Practice Metrics Quantified. IDC Insight, (December), 2015. [11] F. Erich, C. Amrit, and M. Daneva. Report: DevOps Literature Review. (October):27, 2014. [12] G. Garrison, S. Kim, and R. L. Wakefield. Success factors for deploying cloud computing. Communications of the ACM, 55(9):62, sep 2012. [13] M. Hüttermann. DevOps for Developers. Apress, Berkeley, CA, 2012. [14] R. Johnson, E. Gamma, R. Helm, and J. Vlissides. Design patterns: Elements of reusable object-oriented software. Boston, Massachusetts: Addison-Wesley, 1995. [15] M. Kircher and P. Jain. Pattern-Oriented Software Architecture, Patterns for Resource Management, volume 3. John Wiley & Sons, 2013. [16] X. Labs. Periodic Table Of DevOps Tools. 37 BIBLIOGRAPHY [17] M. Loukides. What is DevOps? "O’Reilly Media, Inc.", 2012. [18] P. Mell and T. Grance. The NIST definition of cloud computing. NIST Special Publication, 145:7, 2011. [19] G. Meszaros and J. Doble. A pattern language for pattern writing. Pattern languages of program design, pages 1–36, 1997. [20] D. W. W. Royce. Managing the Development of large Software Systems. Ieee Wescon, (August):1–9, 1970. [21] D. C. Schmidt. Using design patterns to develop reuseable object-oriented communication software. Communications of the ACM Special Issue on Object-Oriented Experiences, 38(10):65–74, 1995. [22] T. B. Sousa, F. Correia, and H. S. Ferreira. DevOps Patterns for Software Orchestration on Public and Private Clouds. page 11, 2015. [23] Startup Europe Partnership. SEP MONITOR PORTUGAL RISING: MAPPING ICT SCALEUPS. 2015. [24] Startup Lisboa. Startups. [25] M. H. Syed and E. B. Fernandez. Cloud ecosystems support for internet of things and devops using patterns. In 2016 IEEE First International Conference on Internet-of-Things Design and Implementation (IoTDI), pages 301–304. IEEE, 2016. [26] Uptec. Companies. [27] L. M. Vaquero, L. Rodero-Merino, J. Caceres, and M. Lindner. A break in the clouds. ACM SIGCOMM Computer Communication Review, 39(1):50, dec 2008. [28] J. Willis. What Devops Means to Me, 2010. [29] Q. Zhang, L. Cheng, and R. Boutaba. Cloud computing: state-of-the-art and research challenges. Journal of Internet Services and Applications, 1(1):7–18, 2010. 38 Appendix A Annexes A.1 Pattern Map 39 Annexes A.2 Concepts Grouping 40 Annexes Abyssal Arealytics Atiiv Nmusic Zarco Company A Celfinet clickly Codacy Emailbidding EZ4U Hypelabs Imaginary Cloud Iterar Jscrambler Mindera Semasio Shiftforward Streambolico TOPDOX Ubiwhere Unbabel Virtus.ai 3Port Weduc Practice Frequency Pattern Related Practices Frequency Pattern Code Review ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 11 11 Code Review Jobs(Daemons) ✔1 Jobs(infrastructure) ✔ ✔ 2 Environment Setup (scripts) ✔ ✔ ✔ ✔ ✔ ✔ ✔ 7 Environment Setup (containers) ✔ ✔ ✔ ✔ ✔ ✔ ✔ 7 Environment Setup (VM) ✔ ✔ ✔ ✔ ✔ ✔ 6 Infrastructure Provisioning ( pull container) ✔ ✔ ✔ ✔ ✔ 5 Infrastructure Provisioning ( download VM) ✔ ✔ ✔ ✔ 4 Infrastructure Provisioning ( run script ) ✔ ✔ ✔ ✔ ✔ 5 Version Control (Git) ✔ ✔✔✔✔✔✔✔✔✔✔ ✔✔✔ ✔✔✔ ✔ ✔ ✔ 20 Version Control (Mercurial) ✔1 Version Control (SVN) ✔1 Version Control (Feature Branches) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 9 Version Control (Github flow) ✔ ✔ ✔ ✔ 4 Version Control (Master Only) ✔ ✔ 2 Automatic Scalling ✔ ✔ ✔ 3 Vertical Scalling ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 11 Horizontal Scalling ✔ ✔ ✔ ✔ ✔ ✔ ✔ 7 SCRUM ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 8 Teams (Specialized) ✔ ✔ ✔ 3 Teams(Size) 4-6 2 3 1-4 8 7-9 5-8 6 1-6 2-6 2 7 2-3 5 4 5-6 8 9 1-3 1-5 4 1-3 Hierarchical Teams Structure 0 Error Handling (Rollback) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 8 Error Handling (Rollback Dark) ✔ ✔ 2 Error Handling (Hotfix) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 12 Errors (thresholds) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 8 Errors (alert team) ✔ ✔ ✔ ✔ ✔ ✔ ✔ 7 Errros (alert defined group) ✔ ✔ ✔ 3 Alert channels Email,SMS PagerDuty SMS,Email Chat, Email 4 Cloud Usage ✔✔✔✔✔✔✔✔✔✔✔ ✔ ✔✔✔ ✔ ✔✔ ✔ ✔ ✔ 21 Clout Tools AWS Dig. Ocean, AWS AWS, PT Cloud AWS, own Azure AWS AWS AWS AWS AWS,Linode,.. Heroku AWS AWS AWS Azure Google Cloud AWS, New Relic, Bugsnag Azure AWS 19 CI chat Integration ✔ ✔ ✔ ✔ ✔ 5 CI email integration ✔ ✔ ✔ 3 CI ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 12 CI Build Step ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 11 Functional Tests ✔ ✔ 2 Unit Tests ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 12 Integration Tests ✔ ✔ 2 Monitoring tool ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 13 Monitoring(Response Time) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 13 Monitoring( Traffic) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 14 Monitoring( Health ) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 15 Communication(direct) ✔✔✔✔✔✔✔✔✔✔✔✔ ✔ ✔✔✔✔✔✔✔✔ ✔ ✔✔✔ 25 Communication(chat) ✔✔✔✔✔✔✔✔✔✔✔✔ ✔ ✔✔✔✔✔✔✔✔ ✔ ✔✔✔ 25 Communication(email) ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 12 3 Jobs 15 Reproducible Environments 11 Deploying New Instances 25 Version Control Organization 12 Scalling 9 Team Orchestration 15 Error Handling 9 Alerting 21 Cloud 16 Continuous Integration 17 Auditability 25 Communication Figure A.1: Interview practices frequency extraction. 41 Annexes A.3 Interview Guide Product •What type of product do you develop ? •What is the scale of that product ? Number of countries, number of users ? •Do you have some kind of SLA or some requirements that impacts your work ? Teams •How many teams do you have ? •What is the size of each team ? •Are teams specicalized or do they have multiple specializations working together ? •Do teams interact with each other ? •How are teams seen from an external perspective ? Are they autonomous? •How do you manage your workload ? Do you use SCRUM,Kanban or other ? •How do team members communicate among themselves. Pipeline •How long does your code take to go from idea to production ? •What are the states that you code goes through before reaching a production environment •What triggers the transition between states ? •What kind of tests do you develop? Which teams are involved in that process ? When do they run? •What happens in each of the pipeline states ? •In each state, which teams intervene and what do they do? •What processes did you automate ? Did you choose to not automate some state ? If so, why? •How do you handle your deployment process ? Which tools do you use ? Do you use containers or VMs ? 42 Annexes Infrastructure Management •How do you scale ? Horizontally or vertically? •Does scalling happen automtatically ? •What can make infrastructure scale up/down ? •How is infrastructure increased ? •How do you lift new instances of your infrastructure? Is it automatic? Monitoring & Error Handling •What metrics do you collect from running servers ? •What do you see as errors ? •What process do you follow to solve errors after they are detected ? •When errors are detected, who is notified ? How is the notification sent ? •If errors are detected before the software reaches production, what do you do ? 43 Annexes A.4 Companie Interviews Report Abyssal PRODUCT Abyssal is a startup company focused on developing Subsea Navigation Systems for Remotely Operated Vehicles (ROVs) operating since 1 de Fevereiro 2012. Their goal is to facilitate the access to these locations by developing intuitive and precise software solutions to the major problems affecting ROV piloting and supervision: poor navigation, reduced visibility and lack of spatial awareness. The current solutions has two major modules. One that renders the underwater obstacles in the live video feed from the ROVs and a web one that handles the transmission of data from the rig/ship to shore. TEAMS Teams at abyssal are mainly divided by area of specialization and have between 6-4 people. Nevertheless, roles inside each team sometimes change(see below). Teams communicate both directly and through Slack. Confluence is used as as a collaboration platform. In order to manage the project, Abyssal uses SCRUM. DEVELOPMENT The development process at abyssal focus on three languages : C++. C# and Python. In order to manage their code, Abyssal uses Git with feature branches. Unitary tests are developed by the development team, manual tests are performed by one member of the development team (the role is changed every sprint). This unitary tests are then used in the Continuous Integration phase. A code review process was also implemented. Every feature is analysed by one member of the development team. This member must not be the same one that did the feature. Continuous Integration is managed by Team City. Team City is responsible for creating a build and running the existent tests. OPERATIONS & INFRASTRUCTURE MANAGEMENT Abyssal uses AWS services in order to manage their infrastructure. Due to well defined upper and lower bounds for users numbers and load scaling is done only manually at Abyssal. Deployments and upgrades are mainly managed manually except for the Virtual Reality where a tool assists the the deployment process. Logs are stored in the running instances and retrieved manually when needed. 44 Annexes Monitoring relies on AWS services and the main metrics monitored are server load and and disk space. When errors are detected, Abyssal usually solves them by issuing a hotfix. 45 Annexes clickly PRODUCT clickly is a web startup that curates content from all over the web, and matches it with interactive ads from brands. Its technology monetizes online content by programmatically matching it to relevant advertisers through an interactive native ad unit. TEAMS The clickly team is not divided by speciality and they have between 6 developers. Teams communicate directly and by using Slack. DEVELOPMENT In order to manage their code, clickly uses Git with feature branches. OPERATIONS & INFRASTRUCTURE MANAGEMENT clickly uses AWS services in order to manage their infrastructure. Scalling is managed by the AWS Beanstalk service and occurs in case traffic increases. Deployments and upgrades are also managed by AWS Elastic Beanstalk. Logs are stored in the running applications and retrieved manually if needed. Monitoring relies on the AWS services as well and the some of the monitored metrics are the network traffic. When errors are detected, clickly usually solves them by issuing a hotfix. In cases where a hotfix is not possible the previous version is deployed. 52 Annexes Codacy PRODUCT Codacy is an automated code review tool that helps developers to save time in code reviews and to tackle technical debt. It centralises customisable code patterns and enforces them within engineering teams. Codacy tracks new issues by severity level for every commit and pull request. It provides advanced code metrics on the health of a project and on the performance of teams. TEAMS Teams at Codacy are mainly divided by specialization and have between 1 and 6 people. Teams communicate directly or using Slack. SCRUM is used to manage the project. DEVELOPMENT The development process at Codacy is done in Scala . In order to manage their code, Codacy uses Git. Unitary tests are developed by the developers. A code review process was also implemented. Every feature is analysed by a module technical owner(each module/service has someone responsible for guaranteeing the quality of that module). Continuous Integration is managed by Bamboo . This tool is responsible for creating a build and running the existent tests. Environments are defined using Docker. OPERATIONS & INFRASTRUCTURE MANAGEMENT Codacy uses AWS ElasticBeanstalk services in order to manage their infrastructure. Deployments and upgrades are managed by AWS and the rolling update strategy is used. Jobs are managed by launching a new instance/container to perform the job. Logs are stored and processed using the ELK stack. Monitoring relies on Ruxit. When an error is detected the IT member is notified. 53 Annexes EmailBidding PRODUCT Emailbidding is an Email Marketing Advertising Platform. A self-service, web-based platform for advertisers and agencies that allows them to segment and bid for the audience in opt-in publisher’s networks. TEAMS Teams at Emailbidding are mainly divided by speciality and have between 2 and 6 people. Teams communicate using Skype (to share files and links) or directly (for doubts and discussions). Teams at Emailbidding are divided in two groups: Developers and IT Operations. Nevertheless, there is a competence center that organizes activities that aim to provide the operations groups with some of the developers points of views and vice-versa. DEVELOPMENT The development process at Emailbidding focus on several languages: Javascript, PHP, Java, ... In order to manage their code, Emailbidding uses Git with a branching system where several environments are mapped. First there is one branch for each issue. On each commit, codesniffer is run and if it passes, a pull request is created for the master branch. A CI tool(Circle CI) runs unit and integration tests on every pull request and a code review process is also implemented. If everything passes, the code is then released to a stagin environment where a QA process is done. The code then goes to production if everything goes as planned. There are also additional branches for hotfixes. Environments are setup using Docker. OPERATIONS & INFRASTRUCTURE MANAGEMENT Deployments are managed using Fabric that manages asset building, clusters, CDN asset publication. Emailbidding uses a combination of AWS services with non cloud services. Scalling is managed both manually(for expected load increases) and automatic in case the load increases without warning. Logs are stored using Logentries, Stackdriver and Librato. When errors are detected an email is sent to the developers and a SMS is sent to the VP of Engineering and the CTO. They are usually fixed by issuing a hotfix. 54 Annexes EZ4U PRODUCT EZ4U’s platform allows sending SMS with global coverage. The platform allows: •Recurrent Contacts and / or massive marketing campaigns •Sending through the web or automated mechanisms •White Label solution for agencies and resellers •Sending to both national and international receivers •Inbox with automatic message processing •Integration with external systems: RESTful API - JSON/XML TEAMS The EZ4U development team has currently 2 developers. DEVELOPMENT The development process at EZ4U focus on PHP. In order to manage their code, EZ4U uses Git. Unitary tests are developed by the development team. OPERATIONS & INFRASTRUCTURE MANAGEMENT EZ4U uses AWS beanstalk services in order to manage their infrastructure. Scaling is possible altough it is not automated. Deployments and upgrades are managed by AWS Beanstalk. Logs are stored in the running instances and retrieved manually if needed. Monitoring relies on the AWS services and the main metrics monitored are the sms sending speed variance, server load, network traffic, etc. When an error is detected the entire development team is informed either by sms or email depending on the severity of the error. 55 Annexes Hypelabs PRODUCT HypeLabs develops a cross-platform communications SDK that uses multiple transport technologies, such as Wi-Fi or Bluetooth, to create local mobile ad hoc networks, making devices communication-capable even if there’s no Internet access. TEAMS The Hypelabs team is multidisciplinary and has 7 members. Teams communicate directly and using slack of facebook. DEVELOPMENT The development process at Hypelabs focus on three languages: C#,Java and Objective C . In order to manage their code, Hypelabs uses Git. 56 Annexes Imaginary Cloud PRODUCT Imaginary Cloud builds web and mobile applications. TEAMS Teams at Imaginary Cloud are mainly divided by project and have between 2-3 people. Teams communicate both directly and by using Slack. Projects are managed using SCRUM. DEVELOPMENT The development process at Imaginary Cloud focus on several languages including: Ruby, Java, Objective-C,... In order to manage their code, Imaginary Cloud uses Git with Feature Branches. Unitary tests, functional and acceptance tests are developed by the development team. A code review process was also implemented. Every feature is analysed by a senior developer before being accepted. Continuous Integration is managed by SemaphoreCI. OPERATIONS & INFRASTRUCTURE MANAGEMENT Imaginary Cloud uses (AWS|Digital Ocean|Linode..) services in order to manage their infrastructure. Scaling can be both vertical or horizontal. Monitoring relies on the some external tenchnologies like new relic and Mixpanel services. Some of the metrics monitored are the server load, server state, latency, response time and exceptions issued and the ratio between logged users and users that are not logged. When errors are detected the Imaginary Cloud usually solves them by issuing a hotfix or in case of a more severe case a roll back. 57 Annexes Iterar PRODUCT Iterar is a Porto based startup focused on web and mobile software development. TEAMS The Iterar team has 5 people. Everyone collaborates with each other and teams communicate directly or via Slack. Project tasks and progress are managed by Trello and the team is managed in a semi agile style. DEVELOPMENT The development process at Iterar focus on Ruby, Java, Objective-C and Javascript. In order to manage their code, Iterar uses Git. Some functional tests are developed by the development team that also performs some manual tests. OPERATIONS & INFRASTRUCTURE MANAGEMENT Monitoring is handling by New Relic(health and load) and Rollbar(exceptions). Iterar uses Heroku services in order to manage their infrastructure. Deployments, upgrades, workers launching are also managed by Heroku. Application logs and monitoring metrics values are stored in Rollbar and New Relic. When errors are detected the development team is notified. 58 Annexes Jscrambler PRODUCT Jscrambler is a Web startup that works on security products to protect Web and Mobile Applications. On of its products is a RASP solution to make apps self-defensive and resilient to tampering and reverse-engineering. TEAMS Teams at Jscrambler are mainly multidisciplinary and have 4 people. Teams communicate both directly (for doubts and discussions) and by Slack (to share files and links). DEVELOPMENT The development process at Jscrambler focus on Javascript. In order to manage their code, Jscrambler uses Git with feature branches. Unitary tests and functional tests are developed by the development team. Continuous Integration is managed by Jenkins . This tool is responsible for creating a build, running the existent tests and triggering the deployment process. OPERATIONS & INFRASTRUCTURE MANAGEMENT Jscrambler uses AWS services in order to manage, monitor and upgrade their infrastructure. The company also uses some proprietary monitors with the aim to enforce load and health monitoring. When errors are detected the the entire team is notified. Depending on the error severity emails or sms are sent to the devs. Errors are usually solved by changing the DNS to a previous version of the service. 59 Annexes Semasio PRODUCT Semasio develops a Web Platform that works analyzes internet users and their habits in order show them better and more customized ads. TEAMS Teams at Semasio are mainly divided by speciality and have between 2-5 members. Teams communicate directly (for doubts and discussions) or by using Slack (to share files and links). DEVELOPMENT The development process at Semasio focus on C# . In order to manage their code, Semasio uses Git with github flow. Unitary tests are developed by the development team. Manual and exploratory tests are performed by the QA team. Teams use a Code review process to both ensure code quality and also to increase knowledge sharing. Every commit is verified on pull request and by an additional developer and a QA. Continuous Integration is managed by Visual Studio Team Studio . This tool is responsible for creating a build and running the existent tests. Semasio uses CI in order to offload their developers with the responsibility of running the tests locally and also to detect errors early. Environments are pre-setup in a external server and developers develop on that servers. OPERATIONS & INFRASTRUCTURE MANAGEMENT Semasio does not directly manage its infrastructure. When errors are detected the depending on the severity an email or a call can be made to a on call engineer. 60 Annexes Shiftforward PRODUCT Shiftforward develops two web platforms that allow companies to better analyze and predict traffic to their sites. TEAMS The Shiftforward team has no division and has 8 members. Teams communicate both directly and using Slack. DEVELOPMENT The development process at Shiftforward focus on Scala. In order to manage their code, Shiftforward uses Git with feature branches. Unitary and functional tests are developed by the development team. A code review process was also implemented in order to improve knowledge sharing and maintain quality. Every feature is analysed by preferably a developer that fully understands the module in question. Continuous Integration is managed by Gitlab CI and is done to run all tests in a clean environment and mark pull requests with the build status and allow reviewers to have a way to know if the code is working without having to manually test it. As tests were beginning to take a while to run, the tool also allows developers to not have to run the tests locally. The CI tool is responsible for creating a build and running the existent tests. OPERATIONS & INFRASTRUCTURE MANAGEMENT In order to manage its infrastructure, Shiftforward uses Marathon on top of Apache Mesos hosted in AWS. Logs are collected using Logstash and are store in ElasticSearch. Monitoring relies on the Marathon and Pingdom services and the main metrics monitored are server response time and server health. When errors are detected the team is notified. 61 Annexes Weduc PRODUCT Weduc is a tool that allows schools to share informations and content with parents related with their child evoltuion. TEAMS Teams at Weduc communicates using email or skype. DEVELOPMENT The development process at Weduc focus on PHP and Javascript. In order to manage their code, Weduc uses Bitbucket. Unitary and acceptance tests are developed by the development team. A code review process was also implemented. Every feature is analysed by a senior developer before being accepted. OPERATIONS & INFRASTRUCTURE MANAGEMENT Weduc uses AWS services in order to manage their production infrastructure. Deployments and upgrades managed through scripts. Monitoring relies on Munin, Zabix and Pingdom. When errors Weduc usually solves them by issuing a hotfix. 68 Annexes Mindera PRODUCT Mindera is a software development company. TEAMS Teams at Mindera are multidisciplinary and have between 5 and 6 elements. Teams communicate directly (for doubts and discussions) or using Slack,Skype,gotomeeting and Zoom (to share files and links, to communicate with external clients and to talk with other team members). Projects are managed using SCRUM or Kanban. DEVELOPMENT The development process at Mindera focus on several languages which include: Java, javascript, Objective-C and C#. In order to manage their code, most projects at Mindera uses Git with feature branches. Unitary tests are developed by the development team. Continuous Integration is managed by Jenkins and in some projects by Go (https://www.go.cd/). This tools are responsible for creating a build and running the existent tests as well as checking the code coverage. Both this tools are also responsible for starting the deployment process. OPERATIONS & INFRASTRUCTURE MANAGEMENT Mindera uses AWS services in order to manage their infrastructure, scaling and monitoring. Deployments are made by lifting a copy of the current infrastructure and then switching the DNS server entries. Monitoring and auditing (log keeping and extraction) relies also in the AWS services and the main metrics monitored are the server health and response time. Some additional tools are also used in this context, including pingdom and CloudWatch. When errors are detected the the problem is usually redirected to the company through the client and Mindera usually solves them by issuing a hotfix or a rollback 69