A reusable web hosting control panel with billing system

A reusable web hosting control panel with billing system Marc Aymerich Gubern Universitat Politecnica de Catalunya A thesis submitted for the degree of Diploma in Computer Systems Fall semester 2011-2012 Contents 1 Introduction 1 1.1 Projectoverview ............................. 1 1.2 Context .................................. 3 1.2.1 WhatisPangea? ......................... 4 1.2.2 Pangea members description . . . . . . . . . . . . . . . . . . . 4 1.2.3 Pangea staff description . . . . . . . . . . . . . . . . . . . . . 5 1.2.4 Pangea software stack . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Requirements analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3.1 Contacts management . . . . . . . . . . . . . . . . . . . . . . 8 1.3.2 Services .............................. 8 1.3.3 Advanced pricing configurations . . . . . . . . . . . . . . . . . 10 1.3.3.1 Calculate the metric of the service . . . . . . . . . . 11 1.3.3.2 How to calculate the price . . . . . . . . . . . . . . . 12 1.3.4 BillingSystem........................... 13 1.3.5 Payment gateway . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.6 Resource limiting and accounting . . . . . . . . . . . . . . . . 15 1.3.7 Internationalization . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.8 Target software to support . . . . . . . . . . . . . . . . . . . . 16 1.3.9 Multi server support . . . . . . . . . . . . . . . . . . . . . . . 17 1.3.10 Easy to use for unskilled users . . . . . . . . . . . . . . . . . . 17 1.3.11 Easy to add new functionalities . . . . . . . . . . . . . . . . . 17 1.3.12Reusable.............................. 17 1.3.13 Open source friendly . . . . . . . . . . . . . . . . . . . . . . . 17 2 State of the art 19 2.1 Current related software at Pangea . . . . . . . . . . . . . . . . . . . 19 2.1.1 Members and billing management . . . . . . . . . . . . . . . . 19 2.1.2 User control panel . . . . . . . . . . . . . . . . . . . . . . . . 20 i 2.1.3 Administration scripts . . . . . . . . . . . . . . . . . . . . . . 21 2.2 Outstanding existing solution . . . . . . . . . . . . . . . . . . . . . . 21 2.2.1 SysCP............................... 23 2.2.1.1 Requirements fit . . . . . . . . . . . . . . . . . . . . 23 2.2.2 Domain Technologie Control (DTC) . . . . . . . . . . . . . . . 25 2.2.2.1 Requirements fit . . . . . . . . . . . . . . . . . . . . 25 2.2.3 ISPConfig2............................ 27 2.2.3.1 Requirements fit . . . . . . . . . . . . . . . . . . . . 27 2.2.4 Summary ............................. 28 2.3 Finaldecision............................... 29 3 Selected tools 31 3.1 Djangoframework ............................ 31 3.1.1 Framework advantages . . . . . . . . . . . . . . . . . . . . . . 31 3.1.2 Django advantages . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2 Celery distributed task queue . . . . . . . . . . . . . . . . . . . . . . 34 3.3 PisaPDFgenerator............................ 35 3.4 Djangoadmintools............................ 35 4 Design principles 37 4.1 Reusability ................................ 37 4.1.1 Separation of Concerns (SoC) . . . . . . . . . . . . . . . . . . 37 4.1.2 Loosecoupling .......................... 38 4.2 Orthogonality............................... 39 4.3 Adaptability................................ 40 4.4 Extensibility................................ 41 4.5 DRY Don’t repeat yourself . . . . . . . . . . . . . . . . . . . . . . . . 42 5 Architecture design and implementation 43 5.1 General considerations . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.2 Common.................................. 48 5.3 Daemons.................................. 50 5.3.1 DaemonsModel.......................... 50 5.3.2 DaemonsAdmin ......................... 53 5.4 Resources ................................. 55 5.4.1 Resources Model . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.4.2 Resources Admin . . . . . . . . . . . . . . . . . . . . . . . . . 58 ii 5.5 Contacts.................................. 59 5.5.1 ContactsModel.......................... 59 5.5.2 Scheduling cancellations and deactivations . . . . . . . . . . . 61 5.5.3 ContactAdmin .......................... 63 5.6 Ordering.................................. 68 5.6.1 OrderingModel.......................... 68 5.6.2 OrderingAdmin ......................... 69 5.7 Billing ................................... 72 5.7.1 BillingModel ........................... 72 5.7.2 BillingAdmin........................... 74 5.8 Serviceconverter ............................. 76 5.8.1 django.contrib.auth a succesfull example . . . . . . . . . . . . 76 5.9 Mail .................................... 79 5.9.1 MailModel ............................ 79 5.9.2 MailAdmin............................ 80 5.10Web .................................... 81 5.10.1WebModel ............................ 81 5.10.2WebAdmin............................ 82 5.11PHPPlugin................................ 83 5.11.1 PHP Plugin Model . . . . . . . . . . . . . . . . . . . . . . . . 83 5.11.2 PHP Plugin Admin . . . . . . . . . . . . . . . . . . . . . . . . 84 5.12DNS .................................... 84 5.12.1DNSModel ............................ 84 5.12.2DNSAdmin............................ 85 5.13Jobs .................................... 87 5.14JobsModel ................................ 87 5.14.1JobsAdmin............................ 87 5.15Extrafields ................................ 88 5.15.1 Extra fields Model . . . . . . . . . . . . . . . . . . . . . . . . 88 5.16ExtrafieldsAdmin ............................ 89 5.17Globalinteractions ............................ 90 5.17.1 Create, update and delete a service . . . . . . . . . . . . . . . 90 5.17.2Billacontact ........................... 92 iii 6 Evaluation 93 6.1 Developmenteffort ............................ 93 6.2 Global economic analysis . . . . . . . . . . . . . . . . . . . . . . . . . 96 6.3 Evaluation and testing of the project implementation . . . . . . . . . 97 6.4 Reusability evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 98 6.5 Requirements compliance . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.6 Generalevaluation ............................ 101 6.7 Futurework................................ 102 A Examples 105 A.1 Templateexample............................. 105 A.2 Invoiceexample.............................. 105 A.3 Feeexample................................ 105 B Manuals 109 B.1 Installation ................................ 109 B.2 Create and install new service . . . . . . . . . . . . . . . . . . . . . . 113 B.3 Adminuseguide ............................. 114 iv List of Figures 1.1 Pangeasoftwarestack .......................... 6 1.2 CaptionforLOF ............................. 7 1.3 Billlifecycle................................ 15 2.1 SysCPscreenshot............................. 23 2.2 DTCscreenshot.............................. 25 2.3 ISPConfig 2 screenshot . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.1 Celery architecture overview . . . . . . . . . . . . . . . . . . . . . . . 34 3.2 Django admin tools drag and drop detail . . . . . . . . . . . . . . . . 36 3.3 Django admin tools menu detail . . . . . . . . . . . . . . . . . . . . . 36 4.1 Django Model-Template-View overview diagram . . . . . . . . . . . . 40 5.1 Admin home page screenshot . . . . . . . . . . . . . . . . . . . . . . 44 5.2 Admin change list screenshot . . . . . . . . . . . . . . . . . . . . . . 45 5.3 Admin actions screenshot . . . . . . . . . . . . . . . . . . . . . . . . 45 5.4 Admin change form screenshot . . . . . . . . . . . . . . . . . . . . . . 46 5.5 Applications architecture overview . . . . . . . . . . . . . . . . . . . . 47 5.6 Screenshot of list add form with default looking . . . . . . . . . . . . 50 5.7 Daemons application model . . . . . . . . . . . . . . . . . . . . . . . 51 5.8 Daemons admin change list . . . . . . . . . . . . . . . . . . . . . . . 53 5.9 Django-celery admin change list . . . . . . . . . . . . . . . . . . . . . 54 5.10 Daemons admin change form . . . . . . . . . . . . . . . . . . . . . . . 54 5.11 Resources application model . . . . . . . . . . . . . . . . . . . . . . . 55 5.12 Monitoring dynamic inline . . . . . . . . . . . . . . . . . . . . . . . . 58 5.13 Monitor admin change list . . . . . . . . . . . . . . . . . . . . . . . . 58 5.14 Monitor admin change form . . . . . . . . . . . . . . . . . . . . . . . 59 5.15 Contacts application model . . . . . . . . . . . . . . . . . . . . . . . 60 5.16 Contacts application model . . . . . . . . . . . . . . . . . . . . . . . 61 v 5.17Diagramslegend ............................. 62 5.18 Related contracts graph . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.19 Related objects dependency graph . . . . . . . . . . . . . . . . . . . . 62 5.20 Scheduling cancellation and deactivation structure . . . . . . . . . . . 63 5.21 Contact link inserted on the service change list page . . . . . . . . . . 64 5.22 Contact and contract links inserted on the service change form . . . . 64 5.23 Filter objects by related contact . . . . . . . . . . . . . . . . . . . . . 64 5.24Addservicecycle ............................. 65 5.25 Contact admin change list . . . . . . . . . . . . . . . . . . . . . . . . 66 5.26 Contact admin change form . . . . . . . . . . . . . . . . . . . . . . . 66 5.27 Contract admin change list . . . . . . . . . . . . . . . . . . . . . . . . 67 5.28 Pack contraction page . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.29 Contract service page . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.30 Ordering application model . . . . . . . . . . . . . . . . . . . . . . . 68 5.31 Service admin change list . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.32 Order admin change list . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.33 Service admin change form . . . . . . . . . . . . . . . . . . . . . . . . 71 5.34 Billing application model . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.35 Billing admin change list . . . . . . . . . . . . . . . . . . . . . . . . . 74 5.36 Billing admin change form . . . . . . . . . . . . . . . . . . . . . . . . 75 5.37 Billing amend line page . . . . . . . . . . . . . . . . . . . . . . . . . . 75 5.38Adminloginform............................. 77 5.39Userchangelist.............................. 77 5.40Userchangeform............................. 78 5.41 Mail application model . . . . . . . . . . . . . . . . . . . . . . . . . . 79 5.42 Mail admin change list . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.43 Mail admin change form . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.44 Web application model . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.45 Web admin change form . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.46 PHP Plugin application model . . . . . . . . . . . . . . . . . . . . . . 83 5.47 PHP plugin dynamic form example . . . . . . . . . . . . . . . . . . . 84 5.48 DNS application model . . . . . . . . . . . . . . . . . . . . . . . . . . 85 5.49 DNS admin change form . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.50 DNS admin change list . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.51 Jobs application model . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.52 Jobs app admin add form . . . . . . . . . . . . . . . . . . . . . . . . 88 vi 5.53 Extra fields application model . . . . . . . . . . . . . . . . . . . . . . 89 5.54 Extra fields app admin add form . . . . . . . . . . . . . . . . . . . . 89 5.55 Extra fields dynamic form inserted on contacts change form . . . . . 90 5.56 Apps interaction on save() service . . . . . . . . . . . . . . . . . . . . 90 5.57 Apps interaction on bill() contact . . . . . . . . . . . . . . . . . . . . 92 6.1 LocandChurn .............................. 94 6.2 Commit Activity Index . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.3 FileCount................................. 94 6.4 Testenvironment ............................. 98 A.1 Invoiceexample.............................. 106 A.2 Feeexample................................ 107 B.1 Creating a chroot with deploy dev.sh . . . . . . . . . . . . . . . . . . 110 B.2 Configuring MySQL server with deploy dev.sh . . . . . . . . . . . . . 111 B.3 Screenshot of the last deploy dev.sh steps . . . . . . . . . . . . . . . . 111 vii 1.2.1 What is Pangea? Pangea.org or Pangea is a non-profit supporting organization, which has been working for social change since 1993. It “facilitates communication” through the use of information and communication technologies, such as e-mail, web content, free software and any other Internet service. Although it works with the Association for Progressive Communications in international projects its activity is focused on Europe / Spain / Catalonia following the principle of “act locally, think globally”. This network began with an idea in early 1993 with the goal to enable electronic communication and information dissemination before the Internet appeared in Spain. That was the time of modems, terminal emulators, email, conferences, data networks. Later on, Pangea offered, promoted and provided training on the use of Internet, Linux, free software, web pages and applications to civil society organizations, social movements and non-profits. Furthermore, Pangea promotes critical thinking about the use, development, participation and politics of the Internet, ICT4D, and the Information Society in general. Pangea offers is self-sustained by the membership.4 Today Pangea is formed by a group of more than 700 members, including NGOs and individuals, who contribute to this project and use their services. Its core is composed by 10 employees and voluntaries who take care of the daily work, including system administration, software developing, member support and entity’s accountancy. 1.2.2 Pangea members description The Pangea’s members are persons and organizations like associations, NGO’s and foundations. They usually work on the third sector, in topics such as development, peace, ecology, cooperation, education and so on. They are people from all ranges of age, including old people who usually has a huge lack of IT related basic knowledge, so they appreciate the good support provided by Pangea as well as services with simple and intuitive program interfaces. 4Source en.wikipedia.org/wiki/Pangea.org 4 1.2.3 Pangea staff description Currently there are 4 persons working at Pangea who are affected by the control panel subject in one way or another. Find below each person’s responsibilities, so the reader will get a general idea on how a control panel can improve and ease its everyday work. •Support (Lorena). Responsible of the help-desk support for the members via email and telephone. The usual members requests are changes on their services or ask for aid in any IT related topic, such as configuring their mail client or the best CMS choice for their new web space. She also manages their bills and membership fees, and it is on this point were the control panel can be very useful for she. •System administrators (Carlos and I). Our responsibility is improve and maintain up and running all the IT infrastructure. Also we have the responsibility of solving non trivial members requests that Lorena can not do by her own. For example it could be configure the Apache server to fit the needs of a member or create a new virtual machine. The control panel intends to change this situation and make this part of our work easy enough to be done by Lorena. •Accountant (Maite). She is responsible for reporting the Pangea’s financial results, in accordance with government and regulatory authority rules. She also measures, disclosure and provision assurance about financial information that helps make decisions about allocating resources, like buying new servers or reducing IT expenses. 1.2.4 Pangea software stack The Pangea software architecture is all based on Open Source and it runs on top of a Proxmox VE5cluster composed by three physical servers (dell.pangea.org, sol- .pangea.org and backup.pangea.org). Pangea uses OpenVZ6for the virtualization layer and all the services run on top of a Debian squeeze Linux distribution isolated inside OpenVZ containers. 5Proxmox Virtual Environment is an easy to use Open Source virtualization platform for running Virtual Appliances and Virtual Machines 6OpenVZ (Open VirtualiZation) is an operating system-level virtualization technology based on the Linux kernel and operating system. OpenVZ allows a physical server to run multiple isolated operating system instances, known as containers, Virtual Private Servers (VPSs), or Virtual Environments (VEs). 5 Figure 1.1: Pangea software stack Basically there are 3 containers that offer services directly to their members. These containers are the main target of the control panel. •web.pangea.org this server offers all the web-related functionalities providing shared hosting services through Apache2, SuExec, fcgid, php4-cgi and php5- cgi. There is a Wordpress and a Dokuwiki with multisite capabilities that allow members create their own blogs or wikis. Also this server provides the web interfaces of the services that are running on other servers, such as phpmyadmin to manage the MySQL databases and two webmails clients: RoundCube and Horde/IMP. •mysql.pangea.org is the MySQL database server. •mail.pangea.org is the main mail server of Pangea running with Postfix as a MTA7and MailScanner as an antipsam gateway. This server stores the mailboxes and provides IMAP and POP3 access through the Dovecot daemon. The mailing list service lives on this server too, and it is offered by Mailman. 7Mail transfer agent (MTA) or mail relay is software that transfers electronic mail messages from one computer to another using a clientserver application architecture. 6 1.3 Requirements analysis Figure 1.2: Requirements analysis parody8 An accurate analysis of the requirements is important if we want to satisfy the expectations of the users. Also it is one of the most complicated tasks during the development of a project since it involves identifying all the actors involved and understand their needs. So even for somebody, like in my case, that has been working for Pangea the last 4 years, it has not been a trivial matter since there are many underlying details which only the person who is daily working with them is aware of. Most of the efforts on the requirements analysis have been made on the accountancy and billing related functionalities, since the ones related with the system administrator’s tasks are much simpler for me. The work described on the following lines is the result of a lot of communication and discussions with the rest of the Pangea’s team. 8From http://www.projectcartoon.com 7 Please notice that the presented requirements are not limited to Pangea’s context. We have tried to identify them in a generic way, avoiding to exclude any potential use case that other organizations could have. The presented requirements can be applied to any commercial hosting company. 1.3.1 Contacts management Contact management is about managing the information stored on the system regarding the members (customers in a business context) that make use of the service provided. Typical information that is stored can be the phone numbers, email, post address, postal code, national ID, and so on. Also it can be possible to provide alternates contact information for technical contact and billing purpose. The basic necessary operations are register and unregister a contact and maintains each registered service related with a contact. 1.3.2 Services The most common hosting services that any hosting provider organization could offer are: 1. Shared web hosting9with optional SSL 2. Mail Accounts 3. MySQL databases 4. DNS10 5. Mailing Lists 6. Blogs and Wikis 7. Virtual Private Servers (VPS) 8. Internet transit consumption 9. Web development and maintenance 9Refers to a web hosting service where many websites reside on one web server connected to the Internet. Each site ”sits” on its own partition, or section/place on the server, to keep it separate from other sites. This is generally the most economical option for hosting, as many people share the overall cost of server maintenance. 10http://en.wikipedia.org/wiki/Domain Name System 8 10. Training courses 11. Packs of services The control panel must implement the mechanism that talks to the daemons11 behind these services (detailed ahead on requirement 8. target software to support) as well as provide a web interface for creating and managing these different services. These services can be differentiated based on two bussines model, (1) the subscription model and (2) the one time model. 1. The subscription model is applied in a service like shared hosting, databases, VPS or DNS. It implies that the customer must contract a subscription for a certain amount of time and when it expires the subscription must be renewed in order to keep using the service. For the subscription services it must be possible to schedule future cancellations and deactivations. 2. The one time model is used on those services that are contracted just for a one particular moment (eventual services). The services from the above list that fall into this category are Web development and Training courses. Two variants of the subscription services can be identified based on the service renewal point (the future date when a subscription service should be renewed). The renewal point can be (1) fixed or (2) variable. 1. A fixed renewal point means that the moment of renovation is fixed for all services and, for example, it should be renewed every April of each year. Most of the services offered by a hosting provider uses this approach since they used to invoice their subscription services only once a month or once a year. 2. A variable point means the renovation should happens just one month (or one year) after the moment when the service was contracted. Maybe DNS is the only service that uses this variable point since most DNS providers use it. There are some situations where it is desirable to be able to configure some discounts for the subscription services: •Discount on cancel or deactivation. In some service it is desired to discount the canceled or deactivated proportional part of the subscription period. For example we want to discount the time when a mail account was deactivated. 11A daemon is a computer program that runs as a background process, rather than being under the direct control of an interactive user. 9 •Discount on register. For some services with a fixed renewal point it is interesting to discount a quantity proportional to the time between the fixed renewal point and the register date. For example if we order a new mail account on February and the billing point is in April, maybe we want to discount April to February part of the entire year. We can identify two more variants of the subscription model depending if the service should be payed before (prepay) or after their use/consumption (postpaid). An example of postpaid service is the Internet transit consumption, since there is no way to know in advance how much transit a member consumes in a month. For prepay services such as shared hosting, databases or DNS it should be possible enable some additional flags: •When the subscription of the service is effective or scheduled for future cancellation or deactivation we might want to refund the proportional part of the payed period •When a discounted future cancellation or deactivation has been revoked we might want to recharge this proportional part 1.3.3 Advanced pricing configurations The pricing configuration defines how a price is calculated for a particular service. The hosting providers usually have multiple pricing options, most of them have (1) hosting plans where a pack or group of services are available for a cheapest price than picking these services one by one. (2) Tiered prices are other pricing option, with it the unitary price of a service decays according to the total of services of the same kind that a customer has contracted. The tiered price is useful to provide discounts based on bulk service contracting. Another pricing option is (3) the flat rate where a customer always pay the same price no matter how much resources consume. Since we want to provide a highly flexible control panel, we have came up as a requirement with a pricing mechanism that, even being pretty complicated, it will cover almost any possible configuration that can be thought of. The following subsections explain the different elements and parameters that should be taken into consideration by the control panel when calculating the prices. First we introduce the metric concept and how to calculate it for a given service, and then we will see how to calculate the price using this calculated metric. 10 1.3.3.1 Calculate the metric of the service A service can have two different metrics: (1) the number of contracted services per customer or (2) the amount of resources consumed by a service. Also we need to define which time period is relevant for calculating the metric, it can be a month or a year. And the start point of the period can be fixed or variable (same meaning as explained at the previous section). 1. In case it is a subscription service and the metric calculated is the number of contracted services we can calculate the metric based on the state of the service during the period. It can be: •Active or disabled •Registered (service registration moment) •Renewed (service renovation moment) •Registered and renewed 2. Otherwise in case we use the amount of resource consumed by a service we have two options: •Use the amount of resource for each order or, •use the amount of resource of all the contact orders of the same type of service. Moreover with a subscription service two different approaches for calculating the metric can be used: •Split the entire billing period in small pricing pieces and calculate the price based on the metric for each piece. For example this is possible with an annual billing period and a monthly pricing period, so even it is an annual service the metric used for calculate the price may vary each month. •Just use the first month of each renovation to calculating the metric of the entire billing period. 11 1.3.3.2 How to calculate the price Apart from the calculated metric there is another factor used to determinate the price of a service. It is the contracted hosting plans or packs that a customer or member has. The contracted packs are used for make different prices depending on, for example, the membership type. For instance the membership organization include more services with price 0 than a personal membership account. Or the members can contract for example 50GB Internet transit prepay pack, so with this pack the first 50GB of traffic are covered by the price of the pack. Additionally we have considered three different rating algorithms to determinate the price of a service based on its metric and the contracted packs. The different algorithms are (1) the best price approach, (2) the progressive approach and (3) the match price approach. The three different approaches are presented by examples using the following price definition. Table 1.1: Price definition example Service Pack Metric Unitary price Domain .ORG Default 1 - 4 15 e Domain .ORG Default 5 - 9 10 e Domain .ORG Default 10 - ∞5e Domain .ORG Organization member 1 0 e Domain .ORG Organization member 2 - ∞10 e Domain .ORG Personal member 1 - ∞10 e Now lets imagine that we have contracted an Organization member pack, so we need to consider the default pack and the Organization member pack. This is how each algorithm should obtain the price for 10 domains: 1. Best price method returns the best total price considering the given metric (10). It should return 1 at 0eand 9 domains at 5e 2. Progressive price returns the cheapest prices with accumulation of the pack entries. It is easy to understand with an example: With 10 domains it should return: 1 at 0e, and 9 at 10e. But notice that if we request 20 domains it should return a completely different values: 1 at 0e, 4 at 15e, 4 at 10eand 11 at 5e(total price: 145e) because this path is cheapest than the more ’intuitive’ 1 at 0eand 19 at 10e(total price: 190e) 12 3. Match price returns the cheapest price that match with the metric. For 10 domains it should return 10 at 5e, for 9 domains it should return 9 at 10eand for 1 domain 1 at 0e. 1.3.4 Billing System The billing system must be able to generate and manage five kinds of bill documents: 1. Invoice. It is a commercial document issued by the service provider (seller) to the customer (buyer) related to their contracted services. This document describes the contracted services, quantities, and agreed prices for the services. The invoice indicates how much the buyer must pay to the seller, according to some payment terms. The buyer has 60 days to pay for these services. The invoice must implement support for the Spanish invoicing regulatory rules12, but we need to provide hooks to allow future support for other regulations. You can see an invoice example on Appendix A.1. 2. Membership Fee. It is a document issued by a non profit organization and indicates that the receiver is a member of that organization and an amount of money should be payed for that reason. The membership fees have an independent numeration and no TAX should be applied. This kind of document does not have any regulatory rules. You can see a membership fee example on Appendix A.2. 3. Amendment Invoice. It is a document that indicates a rectification for a previous issued invoice with incorrect information, such as errors in the price, taxes, the services contracted or the contact information. An special request for the accountant of Pangea is that the amendment invoices should be adapted to the previous Spanish regulatory rules (RD 1496/200313) rather than the current ones RD 87/200514. We will provide both of them. 4. Amendment Fee. It is a document that describes a rectification for a previous issued membership fee, the amended error can be incorrect contact information or incorrect price value. This kind of document does not have any regulatory rules. 12descargas.sri.gov.ec/download/pdf/regfactura.PDF 13www.boe.es/boe/dias/2003/11/29/pdfs/A42537-42556.pdf 14www.boe.es/boe/dias/2005/02/01/pdfs/A03397-03401.pdf 13 •Each year the state of the orders of the subscription service must be updated via raw SQL queries •Very poor search options. GISP can not perform search over bills, services or anything else than contact information •Q19 bank procedure is not fully support, GISP can not handle the Q19 file with returned charges delivered by the bank entity Moreover the poor design makes it nearly impossible to maintain or reuse. For example these are some of the design principles applied on the GISP database (notice the irony): •Don’t trust on auto-incremental primary keys functionality of your database engine, it is way better create your own tables to do the job •If a relational database is ’not enough’ just create a text field and embed what you need in XML format •You don’t want an intermediary table for many-to-many relations, just use a sequence of primary keys in a typical varchar field 2.1.2 User control panel During 2006 a student, for his final thesis, tried to develop a control panel working on top of GISP. The project is called AGEC1(Auto Gesti´on de Clientes en Pangea, which means clients self-management for Pangea). Unfortunately the only functionality provided by AGEC is allowing members to list their invoices and their contact information. We tried adding new functionalities to this panel but in the end, only some basic functionalities were implemented: •Login system (the original release does not have a working one) •Web form for creating automated mail responses during holidays (vacation) using Sieve •Web form for creating blog accounts that worked under Wordpress and wikis with DokuWiki 1http://upcommons.upc.edu/pfc/handle/2099.1/5372 20 2.1.3 Administration scripts Currently we are using a set of Perl scripts developed by previous system administrators that helps Pangea’s work of creating and deleting accounts for FTP and email services. And thats it, so most of the rudimentary work must to be done manually by sysadmins. As result of the analysis of the current tools, since their design is not clear and extensible, we have considered that the best option is move forward with a completely new approach. So the next logical step seems to take a look at the existing open source control panels, which is detailed at the following section. 2.2 Outstanding existing solution As a web hosting control panel is a common need of any hosting company, so before deciding on implementing our own control plane, we should check out if there is an existing solution that fits the requirements or can be easily extended to fit them. A lot of control panels with different properties can be found with a quick search. Table 2.1 shows a comparison with basic information of the most popular existing control panels compatible with Linux servers. We are principally interested in the ones that are focused on hosting management (customers and billing support), support Linux servers and are released under an open source license. Among all of them we consider that there are 3 projects that stand out since they satisfy more or less the Pangea’s requirements: (1) SysCP, (2) Domain Technologie Control and (3) ISPConfig. We have installed these three candidates in our test servers and during a few months we have been made an exhaustive evaluation of them. The following sections detail the results and conclusions extracted after this evaluation. 21 Table 2.1: Main Linux web hosting control panels comparison Name License Billing System Language Plugin Support Multiserv Support Baifox GPL ? PHP, SQLite ? ? cPanel Proprietary Agora Perl Yes Partial Direct Admin Proprietary No C++, PHP Yes ? Domain Technologie Control GNU LGPL Yes PHP ? No Gnupanel GPL ? PHP ? ? H-Sphere Proprietary Miva Java Yes Yes HDE Controller X Proprietary ? PHP ? Yes Hosting Controller Proprietary No .NET, MSSQL Yes ? i-MSCP GPL ? PHP, MySQL Yes No InterWorx Proprietary No PHP, MySQL Yes Yes ISPConfig BSD No PHP, Perl Yes Yes ispCP GPL Yes PHP, MySQL No No Kloxo AGPL Planned PHP, C++ ? Yes OpenPanel GPL Miva C++ Yes Planned Plesk Proprietary Yes PHP, MySQL ? ? SysCP GPL Yes PHP, MySQL Yes Partial Froxlor GPL Yes PHP, MySQL Yes Partial Usermin BSD No PHP ? ? Virtualmin GPL Yes Perl Yes Partial Virtualmin Pro Proprietary No Perl Yes Partial Webmin BSD No Perl Yes Yes Main source from http://en.wikipedia.org/wiki/Comparison of web hosting control panels More at http://web-hosting-top.com/directory.control-panels 22 2.2.1 SysCP SysCP2is an open source web hosting control panel written in PHP and with MySQL as a storage backend. The project started in autumn 2003 by Florian Lippert, and two more developers. On 2010 the founder and another core developer left the project. The fact that only a single core-developer continues working on the project gives us little certainty about the future developments of this project3. Figure 2.1: SysCP screenshot 2.2.1.1 Requirements fit 1. Contacts management Ok The contact system fits our requirements. 2. Services Partial Supports most of the required services directly or indirectly through third-party pluggins, except for the complete lack of VPS service support. Also there is no support for shared services. A shared service is a property for some services 2Official SysCP web page http://www.syscp.org 3And we were right, no active development has seen since then. 23 that allows their use and exploitation for multiple different members. An example of this sort of service could be pangea.org domain name, if pangea.org is configured as shared service other members can create subdomains of it, like member1.pangea.org or member3.pangea.org. 3. Advanced pricing configurations Fail Syscp does not support all the pricing requirements. 4. Billing system Fail Syscp can be extended through a patch developed by a third-party developer that it supports billing. The billing system is very basic and does not support membership fees and amendment that follow the Spanish regulatory rules. 5. Payment gateway Fail There is no Q19 support. 6. Internationalization Ok SysCP fits the internationalization requirements. It’s translated to 8 different languages including Catalan and Spanish. 7. Resources limiting and accounting Ok Good resource control support including Internet traffic, web and mail account disk quota. 8. Target software to support Partial Except OpenVZ, all the other required software is compatible with SysCP. 9. Multi server support Partial Multi-server support is not natively supported in SysCP, but you can split services among different servers with little modifications, unfortunately the same service can not be split between multiple servers. 10. Easy to use for unskilled users Ok The interface is clear and simple. 11. Easy to add new functionalities Partial SysCP has a plugin system for including third-party plugins but it has some limitations since is not posible to change the behaviour of existing parts. 24 12. Reusable Fail The SysCP is designed for a very specific server’s architecture and could be difficult to adapt in an existing infrastructure. Also most core developers complain about the code quality (it is not object oriented) and at the time of the decision they were considering to perform a complete code refactoring. 13. Open source friendly Ok SysCP is released under GNU GPL licence. 2.2.2 Domain Technologie Control (DTC) Domain Technologie Control (DTC)4is a control panel aiming at commercial hosting. DTC is free software released under the GNU LGPL v2.1 license. It is also the first web hosting control panel that has reached inclusion in major distributions like Debian (since Lenny in 2009), Ubuntu (since 2008) and FreeBSD. Figure 2.2: DTC screenshot 2.2.2.1 Requirements fit 1. Contacts management Ok The contact system features fits our requirements. 4DTC project page http://www.gplhost.com/software-dtc.html 25 2. Services Partial Supports most of the required services except the installation of apps like Blogs or Wikis, and support for shared services is missing too. 3. Advanced pricing configurations Fail DTC does not support all the pricing requirements. 4. Billing system Fail The billing system is very basic and does not support membership fees and fee’s amendment according to Spanish regulatory rules. 5. Payment gateway Fail There is no Q19 support. 6. Internationalization Ok DTC is translated up to 11 languages including catalan and spanish. 7. Resources limiting and accounting Ok The resource control support seems to be enough. 8. Target software to support Partial As SysCP, OpenVZ support is missing on DTC. 9. Multi server support Fail Due to the directory structure used on DTC it is not possible to have different services distributed between multiple servers. 10. Easy to use for unskilled users Fail Despite the interface is well designed it has too much options and it is too complicated for most of Pangea’s members. 11. Easy to add new functionalities Fail There is no plugin-like architecture on DTC. 12. Reusable Fail DTC has strong server architecture needs and it is really hard to use in an existing infrastructure. Also the source code is a complete mess and most of the best practices on software development seems to be ignored: it is not object oriented, the directory structure is messy and it is impossible to find any kind of documentation oriented to developers. 26 13. Open source friendly Ok Licensed under LGPL. 2.2.3 ISPConfig 2 ISPConfig 25is a control panel for Linux server licensed under BSD license. The current release of ISPConfig is ISPConfig 3 and it has a lot more functionalities and it fits more or less the proposed requirements, unfortunately at the time of the decision it was on early alpha development stage and we did not consider it as an option. Figure 2.3: ISPConfig 2 screenshot 2.2.3.1 Requirements fit 1. Contacts management Ok The contact system is quite complete. 2. Services Fail Unfortunately, like the other valuated control panels, there is no support for shared services. 3. Advanced pricing configurations Fail DTC does not support all the pricing requirements. 5Official ISPConfig2 web page http://www.ispconfig.org/ispconfig-2 27 4. Billing system Fail There is no billing system support on ISPconfig2. 5. Payment gateway Fail Since there is no billing support, neither is payment gateway module. 6. Internationalization Partial ISPConfig2 is translated up to 13 languages but it is not translated to Catalan. 7. Resources limiting and accounting Ok The resource control is enough for most hosting providers. 8. Target software to support Partial Missing support for OpenVZ containers, Mailman mailing lists, Wordpress and Dokuwiki applications. 9. Multi server support Fail There is no multi server support for ISPconfig2. 10. Easy to use for unskilled users Ok One advantage of the lack of functionalities on ISPConfig 2 is that the user interface can not be much complicated. 11. Easy to add new functionalities Fail There is no plugin-like architecture. 12. Reusable Partial It is difficult to install on an existing infrastructure but the source code quality is pretty good and there is a lot of development documentation. 2.2.4 Summary None of the presented solutions fits all or nearly all requirements, therefore in case we decide on one of those solutions a lot of development should be done in order to achieve the objectives of the project. Table 2.2 summarizes how the requirements are fit by the reviewed control panels. The easiest project to contribute seems to be ISConfig 2 because of the good quality code and extensive development documentation, but instead is the solution with less features implemented. On the other hand, we have SysCP, wich has most of the functionalities already implemented but its future is completely uncertain because 28 two of three main core developers has left the project. Finally, DTC is the less preferred solution since it does not support multi server architectures and because the code is quite unfriendly. Table 2.2: Requirements compliance summary Requirement SysCP DTC ISPConfig2 1 Contacts management Ok Ok Ok 2 Services Partial Partial Fail 3 Advance pricing configurations Fail Fail Fail 4 Billing system Fail Fail Fail 5 Payment gateway Fail Fail Fail 7 Resource limiting and accounting Ok Ok Ok 8 Target software to support Partial Partial Partial 9 Multi server support Partial Fail Fail 10 Easy to use for unskilled users Ok Fail Ok 11 Easy to add new functionalities Partial Fail Fail 12 Reusable Fail Fail Partial 13 Open source friendly Ok Ok Ok 2.3 Final decision As the current solutions implemented at Pangea are poor designed and difficult to reuse, and at this point the available open source hosting control panels are not convincing, we decide to develop a new control panel that fits all the presented requirements and takes care of the common shortages of the reviewed panels, such as: •Limited pricing system •Hard to fit on an existing server infrastructure •No shared services •Lack of multi-server support •No membership fee support •No Spanish billing regulatory rules •Not scalable: not suitable for distributed architectures and no any interface seems appropriate for organizations with more than 40 customers 29 Figure 3.2: Django admin tools drag and drop detail •a customizable menu bar Figure 3.3: Django admin tools menu detail •tools to make admin user interface theming easier 36 Chapter 4 Design principles In this section we describe the main design patterns and decisions on which the implementation is based. As we early discuss in this document some of the design decisions has been already taken by choosing the Django framework. The design decisions have been made prioritizing the plugability and reusability of the system. We have designed the project with the idea of creating a framework for the easy creation of web hosting control panels more than designing a control panel itself. 4.1 Reusability The control panel design is highly focused on reusability. We understand reusability as the ability of the system, or some of their parts, to be used in other projects. We implement reusability using two design patterns: (1) Separation of concerns and (2) Loose coupling. 4.1.1 Separation of Concerns (SoC) Django has well integrated this pattern on its project design. With Django you are encouraged to split your project into more specific and focused applications, commonly known as a Django reusable applications. A reusable Django app, is an application that is easily plugged into any project, providing a very specific piece of functionality. They should be focused and follow the Unix philosophy of “do one thing and do it well”. James Bennett1likes to say that an application encapsulates some particular feature2. 1James is Djangos release manager, and a big contributor to the Django documentation 2James Bennett at DjangoCon 2008: Reusable Apps. youtu.be/A-S0tqpPga4 37 4.1.2 Loose coupling The different applications of the project should not know about others, unless it is absolutely necessary, that is, loose coupling and tight cohesion. If we write applications with this principle in mind we ensure that each app will be independent from each other. This independence gives us two potential benefits: 1. Easy development and future maintenance 2. Make it reusable for other projects Some underlying patterns of this principle that we widely use on this project are: •Signals (publish/subscribe) Django includes a “signal dispatcher” which helps on allowing decoupled applications to get notified when actions occur elsewhere in the framework. In a nutshell, signals allow certain senders (publishers) to notify a set of receivers (subscribers) that some action has taken place. They are especially useful when many pieces of code may be interested in the same events. •Registration pattern The registration pattern3is used on an application in order to notify their existence to other applications. That way the other application can take some decisions based on this information. For example, with this pattern we can enable or disable the admin interface for a certain application. •Backend The goal is to create a driver for an specific app that knows how to talk with the API of another application. This is used normally to decouple two applications that have a highly coupled behavior, such as, for example, Order and Billing apps. With the purpose of bill an order the Order application must know how to tell the billing system that it wants to create a bill, this should go on the backend file. If you want to switch your billing application the only thing that you should do is change the backend file in order to match with the new billing API. •Generic relations Common foreign keys4can only “point to” one other model, which means that 3More on http://charlesleifer.com/blog/looking-registration-patterns-django/ 4In the context of Django, a foreign key is a referential constraint between two models. It is implemented with a database field in a relational table that matches a candidate key of another table 38 if a model uses a ForeignKey it would have to choose one and only one model to point to. The Django ContentTypes5application provides a special field type (Generic- ForeigKey) which works around this and allows a relationship to point to any model. •Modify the behavior of another app at runtime Python is a language with a very high level dynamic classes, therefore it is very easy to modify classes and their methods at runtime. One application can deeply change the behavior of classes from another application, and without knowledge of it. The idea behind the reusability principle is to create a generic set of reusable applications that covers the different needs of a web hosting control panel. Then others can build their own control panel reusing a subset of these apps. 4.2 Orthogonality “Orthogonality is a system design property facilitating feasibility and compactness of complex designs. Orthogonality guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system.”6 The Django apps are designed using a variant of a model-view-controller (MVC)7 pattern called model-template-view (MTV). As the MVC patter does, MTV isolates the business logic from the user interface, permitting independent development, testing and maintenance of each part. Figure 4.1 shows the Django representation of the MTV design pattern. 5Django includes a contenttypes application that can track all of the models installed in your Django-powered project, providing a high-level, generic interface for working with your models 6http://en.wikipedia.org/wiki/Orthogonality#Computer science 7More at http://en.wikipedia.org/wiki/Model-view-controller 39 Figure 4.1: Django Model-Template-View overview diagram •The Model layer implements a description of the database table, represented by a Python class. It is the responsible of creating, retrieving, updating and deleting records in the database using simple Python code rather than writing repetitive SQL statements. This layer is the same as the MVC pattern and it is implemented on the models.py application file. •The View layer performs the business logic8for the web pages. It is implemented on the views.py application file. •The Urls module is the responsible to map each URL to a certain view function. It should be specified on the urls.py file. The Urls system together with View layer does the same job as the controller on the traditional MVC pattern. •The Template layer is implemented in different files within the template app directory. These files describe the design of the page so this layer is the same as the View layer on the MVC pattern. A template file uses the Django template language with basic logic statements. 4.3 Adaptability This project is designed with the philosophy of must be the control panel which suits IT infrastructure instead of infrastructure fitting the panel. The downside of this approach is that in most cases the sysadmins will need to modify the server management scripts in order to make it fully compatible with their software. This is not necessarily 8Business logic, or domain logic, is a non-technical term generally used to describe the functional algorithms that handle information exchange between a database and a user interface 40 a big deal since most system administrators love writing their own shell scripts, and moreover we will use the Django template system to ease this task. 4.4 Extensibility The control panel is designed to provide the basic and generic functionalities by default and move the specifics to extensions. Extensions can be through (1) the modification of existing functionality or through (2) the addition of new functionality. “The central theme is to provide for change, atypically enhancements, while minimizing impact to existing system functions.”9 (1) In systems architecture, extensibility means the system is designed to include hooks and mechanisms for expanding/enhancing the system with new capabilities without having to make major changes to the system infrastructure. We have been implemented a plugin mechanism that allows to easy extend the control panel services. For example, the web app only brings support for the basic VirtualHost configuration, and support for mod php or mod proxy configuration can be provided though a plugin. We provide two base classes that helps with the plugin creation task: PluginServiceModel and PluginServiceAdmin. Also the system must facilitate as much as possible the task of adding new services. We provide an inheritance based approach that makes easier the task of implementing support for new types of services. You only need to inherit from BaseService class and define the service specifics, BaseService does the rest. (2) Extensibility can also mean that a software system’s behavior is modifiable at runtime, without recompiling or changing the original source code. Our control panel has an admin interface that allows to modify the behavior of the control panel without touching any single line of code. For example it is possible to describe the resources used for a concrete service and the web interface forms will be modified at runtime in order to reflect these resource descriptions. Another example is the ability of our panel to dynamically add an arbitrary extra field to the services, useful when an organization has a very specific need for a concrete service, like store an additional field on the contact information. 9Quote from http://en.wikipedia.org/wiki/Extensibility 41 4.5 DRY Don’t repeat yourself This software development principle aims at reducing repetition of information of all kinds. The DRY principle is stated as “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” When the DRY principle is applied successfully, a modification of any single element of a system does not require a change in other logically-unrelated elements. Additionally, elements that are logically related all change predictably and uniformly, and are thus kept in sync. This principle is especially useful on model-view-controller architectures since both pursue guarantee that changes in one part of the system does not affect others. 42 Chapter 5 Architecture design and implementation In this chapter we are going to see how we have implemented the control panel requirements reviewed on section 1.3 using the development tools that we have presented on chapter 3 and following the design principles presented in the previous chapter. As one of the design principle was to split the project into small applications this chapter is also divided in the defined applications. Also an overview of different applications working together is presented on the last section: 5.17 Global interactions. 5.1 General considerations Before dealing with the different applications that compose the project we want to present some general points that are used on the implementation of several developed apps. •Commit on success. The views susceptible to perform any database change run under a commit on success database transaction scope. It means that if all the code runs successfully, the panel will commit the work done. Otherwise if the code raises an exception then the panel will roll back the transaction. •The admin interface relies entirely on django.contrib.admin. We reuse the django.contrib.admin application for the web interface by subclassing its ModelAdmin class. ModelAdmin class is very flexible, it has several options for dealing with customizing the interface1. All the admin interface related code lives on the app name/admin.py file. The automatic Django admin interface is composed by three main parts: 1Check available configurations at https://docs.djangoproject.com/en/dev/ref/contrib/admin 43 1. The admin home page, figure 5.1. This page lists all the available types of data (models) that can be edited through the admin site. 2. Admin change list, figure 5.2. Each type of data in the Django admin site has a change list that displays all the available objects in the database. The basic workflow of Djangos admin is “select an object, then change it.”, so the change list page lets you to perform a certain action over a set of selected objects as you can see on figure 5.3. 3. Change and add forms, figure 5.4. Used to modify existing objects and create new ones. Each field defined in the application model appears here, and fields of different types get different widgets. Also every change made through the admin interface is logged. You can examine this log by clicking the history link in the upper-right corner of the window. Figure 5.1: Admin home page screenshot 44 Figure 5.2: Admin change list screenshot Figure 5.3: Admin actions screenshot 45 •Model Level: The ContentType framework is used to identify which is the service model represented by the Deamon, webs.VirtualHost for our web hosting example. •Instance Level: When we have a service distributed over multiple servers multiple daemon instances must be defined too, one for each server. Therefore the system must be able to identify in what server lives each service instance. An expression approach is used to solve this problem. It gives the flexibility to make rules by service instance attributes combined with Python logical and arithmetic operations. Let’s introduce the expression mechanism with an example: O['pk']%3==0 Ois a model instance converted to a Python dictionary so we have access to their attributes using this syntax O[’attribute name’]. This expression example will match all VirtualHosts with primary key (pk) mod 3. So with other 2 daemons with expressions O[’pk’]%3==1 and O[’pk’]%3==2 we get nothing more than a round-robin distribution between those 3 daemons. 3. Execute the scripts This application performs the data modifications on the server side executing the save template4and the delete template. These Django templates can be executed using three different methods: (a) Execute as script over SSH. 99.92% of the *nix5servers around have the SSH service installed. So we take advantage of this, offering the ability to execute the scripts through SSH without the need of installing anything on the server side. We use paramiko6for managing SSH connections with Python. (b) Execute as python code. This option is interesting if we want to run some Python code that wants to access at the control panel model using Django ORM. (c) Execute the template as local script. This is useful if we want to run any other remote access different than SSH, 4See an example on Appendix A.1 5A Unix-like (sometimes referred to as UN*X or *nix) operating system is one that behaves in a manner similar to a Unix system 6http://www.lag.net/paramiko/ 52 or just run the script locally because the server that we want to manage is localhost. This approach is multi platform friendly because we can execute virtually anything. Bash, Perl, Ruby, Python, C, .NET. The only thing we need to do is to write the scripts using the Django template system, see appendix A.1. The script must be idempotent, that is, written in such a way that you may run the same operation more than once and achieve the same result. Otherwise you can run in troubles if the script fails and it is running partially multiple times. The execution of the templates is performed asynchronously using Django Celery message queue. So the user request does not get stuck waiting for the task’s end and the administrator can easily manage this task execution via the control panel interface [see figure 5.9]. 5.3.2 Daemons Admin The actions provided for the daemons are available on its admin change list page [fig. 5.8]: •Delete selected daemons •Enable selected daemons •Disable selected daemons Figure 5.8: Daemons admin change list 53 Figure 5.9: Django-celery admin change list Figure 5.10: Daemons admin change form 54 5.4 Resources This application is responsible of monitoring, recording and limiting the resource consumption of an specific service on the IT infrastructure. A common resource could be CPU cycles, memory usage, number of process, Internet transit or swap space. 5.4.1 Resources Model Figure 5.11: Resources application model 55 The resources application consist on two models: • Monitor . Defines the properties of the resource (like if it is limited or which is the algorithm used to postprocess the monitored data) and establishes how and when the monitorization should be performed. • MonitoringData . Maintains a record of the monitorization data. The resource abstraction on a single application is something intuitive but seems that it is not easy to come up with this concept, since there is no single existing control panel that does it. The existing control panels have hard-coded resources for each application susceptible to consume them, making the control panel more difficult to reuse. This lack of resource abstraction on the other control panels might be because a classic relational model does not have the concept of generic foreign key that allows us to have a model with foreign key relations with multiple models. Therefore we have this powerful tool allowing us to manage all the resources as a single model. The workflow of this application is: (1) execute the monitorization script periodically, (2) interpret and store monitorization data and (3) limit the resource consumption, by running predetermined actions. 1. Execute the monitorization script periodically We use django-celery for scheduling the execution of monitorization scripts, allowing crontab like configuration: minute, hour and day of week. Like daemons application we provide three methods for the template execution: (a) Execute as a script over SSH. (Relies on Daemon SSH connection details) (b) Execute as python code (c) Execute the template as a local script The templates are polled from ./resources/templates/scripts directory and the server where the template will be executed is determined using the daemons application. 2. Interpret and store monitorization data The Monitor configuration provides customizations for the monitorization data interpretation: (1) period, (2) algorithm and (3) block size. 56 •period. Is the time period used for calculating the current state of the resource consumption. •algorithm. Is the operation performed over the period. It can be: –Last monitorization result –Average during the period –Sum of the period values –Maximum or minimum of a period •block size. Defines the allowed tolerance on the measured value to avoid reiterative executions of the limit exceeded template when the measured unit presents an oscillatory behavior, creating a hysteresis7cycle for the thresholds that trigger the scripts execution. This is specially useful for a limit on monthly average disk usage. 3. Limit the resource consumption We can define a monitor with or without a limit. If we define a monitor with a limit two different ways of ensuring the resource limitation of a daemon are provided: (1) with the resource limit exceeded template and (2) with the Daemon save template execution. (a) Resource exceeded template. This approach is achieved by executing the exceeded template when a resource reaches the given threshold limit. For example, this is useful for limiting the monthly consumption of web traffic per VirtualHost. Imagine that you only want to allow 1GB consumption per month for an specific VirtualHost, so when at the middle of the month this VirtualHost has reached the limit, the exceeded template is executed and the VirtualHost is disabled. When a new month comes the recovery template will be executed and the VirtualHost will be reenabled. (b) Daemon save template execution. The limit is ensured by the daemon save template. As the resource application acts as a plugin for a Service, the service knows which is its limit value, so when we write the save template script of the related Daemon we can perform the limitation. A use case would be the web file system quota, because it is set by the command edquota. 7http://en.wikipedia.org/wiki/Hysteresis 57 5.4.2 Resources Admin Python’s metaclasses are used to dynamically create the resource control forms since their definition is stored on the database and we do not know how the monitors are defined until runtime. The Django forms are based on classes. With metaclasses we can create these classes only by defining how they look like. Once we have this form class created we insert it into the admin change list page of the related service. Figure 5.12 shows how a resource dynamic form looks like, and figure 5.45 on page 82 shows this form embedded on a service change form. Figure 5.12: Monitoring dynamic inline The available actions provided for the monitor on its admin change list page [fig. 5.13] are: •Delete selected monitors •Disable selected monitos •Enable seleted monitors Figure 5.13: Monitor admin change list 58 Figure 5.14: Monitor admin change form 5.5 Contacts This application stores the contact information and provides mechanisms to keep each service instance related to a certain contact. 5.5.1 Contacts Model The contacts system consists on the following models: • Contact information with optional BillingContact and TechnicalContact • Contract relations between contacts and services •Contract CancellationDate and DeactivationPeriod scheduling capabilities 59 Figure 5.15: Contacts application model Main contact information is stored on the Contact model class, and also there are two auxiliary model classes that make possible to have an arbitrary number of email and phone numbers per contact. Optional billing contact information is handled by the BillingData model. Contract relations are managed by the Contract model. This class uses generic relations provided by django.contrib.ContentType application. This generic relationship allows us to decouple the contacts application from the others, since it provides a Foreign Key emulation that can point to multiple models. This Contract abstraction is pretty intuitive but there is no existing control panel that takes a similar approach. 60 5.5.2 Scheduling cancellations and deactivations The scheduling of cancellations and deactivations is supported by CancellationDate and DeactivationPeriod models. Since services can depend on others, the data structure is a n-tree, where the root of the tree is a object without dependencies. This structure is generated and maintained by an algorithm that uses Django introspection methods provided by meta8attribute. The algorithm is able to discover by itself the underlying relationships between the services and follows them detecting which are the related services susceptible of being canceled or disabled. We explain on the lines below how the algorithm works with a practical example. Imagine that we have a Contact called Pangea that owns the Domain pangea.org with their two subdomains: www.pangea.org and carlos.pangea.org. Also the Contact Pangea has a VirtualHost that uses pangea.org and www.pangea.org and aSystemUser called pangea that is member of a SystemGroup called equipo. The figure 5.16 represents these relationships and figure 5.17 shows the legend of this and following pictures. Figure 5.16: Contacts application model The green boxes are services and the blue boxes are plugins of these services 8meta is a Django internal attribute used for model introspection 61 5.6 Ordering This application defines the different services that must be charged, keeps track of them, and determines the cost of each one existing on the system in a flexible and powerful way. 5.6.1 Ordering Model Figure 5.30: Ordering application model 68 The ordering application implements all the requirements discussed on chapter 1 sections 1.3.2 Services and 1.3.3 Advance pricing configurations. This app consists on the following data models: 1. Service , that defines precisely how a service is going to behave in terms of pricing and billing 2. Order , which keeps track of the instances that are going to be be charged 3. Pack , that defines service bundles9with included services and/or special offers 4. Rate defines the price of a service The Ordering app is conceptually quite coupled with billing application, so to solve this the backend approach has been used. In a file called billing backend.py we define how and with which application we want to perform the creation of the bills. Notice that it is not mandatory to use a Django related application for generating the invoices, since we can define on the backend file how to create and retrieve an invoice with any existing tool. 5.6.2 Ordering Admin The available actions provided for orders through its admin change list page, figure 5.32, are: •Ignore selected orders. The orders marked with an ignored bit will not be billed •Bill orders •Bill orders (default behavior). Will bill the orders with the behavior defined on settings.py •Budget selected orders. This will generate budgets of the selected orders. This action does not perform any change on the order status Actions provided for the services, figure 5.31: •Delete selected services •Disable selected services 9A services bundle is a package of services, it is used for offering several products for sale as one combined product 69 •Enable selected services •Check selected services, it update orders when a service has been updated or created Three screen-shots of the admin interface are presented below: (1) the service change list page on figure 5.31, (2) the order change list page on figure 5.32, and finally, (3) the figure 5.33 is an screenshot of the service add form. As you can see, it contains all the configuration details discussed on chapter 1 section 1.3.3 Advance pricing configuration. It has a lot of options, but, fortunately for the admin user, we provide a bunch of predefined services that will suit most of the use cases, at most only one or two parameters will need to be tweaked to precisely fit your needs: the billing or pricing period. Figure 5.31: Service admin change list Figure 5.32: Order admin change list 70 Figure 5.33: Service admin change form 71 5.7 Billing The Billing app generates and administrates budgets, invoices, membership fees and their amendments. This application implements all the requirements presented on chapter 1 section 1.3.4 Billing system. 5.7.1 Billing Model The billing application consists on the following models: • BaseBillLine , which is a bill line abstract model • BudgetLine is a bill line for budget • BillLine , which is a base class for the models above • FeeLine is a bill line for fees • InvoiceLine is a bill line for invoices • AmendedBillLine is a base class for amended lines • AmendedInvoiceLine , which is a amended line for invoices • AmendedFeeLine is an amended line for fees • BaseBill , which is a base bill abstract model • Bill is a base class for the models above • Invoice is an invoice • AmendmentInvoice , which is an amended invoice • Fee is a fee • AmendmentFee is an amended fee • Budget is a budget (inherit from BaseBill) An aggressive inheritance pattern is adopted on the implementation of the billing application models. We are not sure yet if this approach is better than one providing a bill type field. At least the adopted solution works really well with the Django automatic admin interface. 72 Figure 5.34: Billing application model 73 The billing system is heavily coupled with Contacts application because each bill needs to have a foreign key relation with a contact. To work around with this issue we provide a setting value allowing you to choose what is the model that you want to use for storing the contact information, as with the ordering app. By default it uses the Contact model from contacts application but the adopted approach lets you use any other application that encapsulates the same contact concept, like the User model from Django auth application. A bill can be created from a subset of contacts, contracts or orders. 5.7.2 Billing Admin The available actions on the admin change list page [fig 5.35] are: •Delete selected bills •Close selected bills •Send selected bills •Amend selected bills •Merge selected bills, must be bills issued to the same contact •Mark as returned •Mark as payd •Mark as irrecovrable Follows a bunch of screenshots of the billing admin interface is presented. Figure 5.35: Billing admin change list 74 The actions early described on table 1.3.4 are available on the billing change form through links on the right corer of the page, see figure 5.36. Figure 5.36: Billing admin change form The bill amendment is performed through the page displayed on figure 5.37. It supports two amend modes: (1) manual, that lets you specify the amended price value for each line, and (2) automatic, assuming the amendment of the entire value. Figure 5.37: Billing amend line page 75 5.8 Service converter Service converter is an application with the aim of providing full integration for third parties Django applications. That way we can pick up any interesting Django app and use it as a new service of our control panel without the need of touching a single line of code. All you need to do is adding the new application through two configuration parameters of the Service converter app settings.py file. Just like we have done for django.contrib.auth.User: # (module, class) BASESERVICE_CONVERTER = ( ("django.contrib.auth.models", "User"), ) # ((model module, model class), # (modeladmin module, modeladmin class)) SERVICEADMIN_CONVERTER = ( (("django.contrib.auth.models", "User"), ("django.contrib.auth.admin", "UserAdmin"),), ) At this point your new service application has the same control panel support as any service application that we have developed, like DNS, mail, web... At this moment this app will bring support for contacts, resources, daemons and orders, but this system is fully compatible with any future features of the control panel. 5.8.1 django.contrib.auth a succesfull example The control panel user accounting and authentication subsystem relies entirely on django.contrib.auth application. Please notice that this is not an application developed by this project, but an official Django reusable application. We have considered very interesting to present this application on this subsection because it is a successful example on how the control panel can perfectly integrate third-party apps without modifying their code. Screen-shots of the user interface are presented on the next two pages. Figure 5.38 shows the login form displayed when we try to access the panel without an active session. 76 Figure 5.38: Admin login form We can confirm that the UserAdmin class has been successfully adapted to the ServiceAdmin looking at the contact link on user change list, figure 5.39. Apart from this little detail of the contact support provided to the Auth application, other service-related support are also provided, such as support for resources, daemons, ordering and billing. Figure 5.39: User change list Finally on figure 5.40 we can see how the User support for SystemUsers and VirtualUsers has been provided through two forms dynamically inserted on the User change form, one (SystemUser) from web app and the other (VirtualUser) from mail application. 77 5.11.2 PHP Plugin Admin Figure 5.47 shows in detail the form inserted on web change form, figure 5.45, when the PHP plugin is installed. Figure 5.47: PHP plugin dynamic form example 5.12 DNS The DNS app is used to manage DNS zones and DNS server configurations. 5.12.1 DNS Model The DNS model application is composed of: • Domain , which is a main domain registered by a customer. It is usually is a second-level domain, for example, in the domain www.mydomain.org it will be mydomain. • Subdomain , which is an extension of a Domain. www in our previous example. • Name , which is the parent model class for Domain and Subdomain. The model Name (domain name) can be a Domain or a Subdomain. This inheritance pattern exists in order to allow no practical distinction between them. This is useful to allow domains and subdomains be part of the ServerName or ServerAlias statements of a VirtualHost definition. • Record . A DNS resource record is the basic data element in the domain name system. Each record has a type (A, MX, etc.), an expiration time limit, a class, and some type-specific data. Configuring a default set of records that will be created for each new domain is possible through dns/settings.py file. 84 Figure 5.48: DNS application model 5.12.2 DNS Admin The admin change form of the DNS is represented on the figure 5.49. The DNS records are filled automatically according to DOMAIN DEFAULT RECORDS settings value. Figure 5.50 shows the admin change list of the domains, notice that it is possible filter the domain list using the filters located on the right part of the page. 85 Figure 5.49: DNS admin change form Figure 5.50: DNS admin change list 86 5.13 Jobs Jobs application provides support for those services that have no representation on the IT infrastructure, such as courses, web maintenance, web development, complex service installations and so on. This application is just for recording and accounting purposes. 5.14 Jobs Model Figure 5.51: Jobs application model The jobs application models are: • Category model, which allows creating and deleting jobs type definitions • Job model, which are the job instances belonging to a category and composed by a description and the number of hours that the job took 5.14.1 Jobs Admin Figure 5.52 is how the jobs admin form looks like. 87 Figure 5.52: Jobs app admin add form 5.15 Extra fields The extra fields application is used to add custom extra fields on your service apps. Recall that the control panel service applications are designed with the philosophy of implementing the basic functionality on a base application and move the specifics to a plugin application. Extra fields is a plugin app that allows you to insert a custom field to a base application. This application uses dynamic forms pattern in order to insert the extra fields on the corresponding service at runtime. As an interesting concept, this application is not so far from an application that allows to entirely create new service applications with the admin interface. 5.15.1 Extra fields Model The extra fields application is composed by two models: • ExtraField model, which is responsible of defining an extra field for a concrete service. Letting to choose the field name, the description and the default initial value. • ExtraValue models, that stores the extra field instances. 88 Figure 5.53: Extra fields application model 5.16 Extra fields Admin Figure 5.54 shows the admin add form page and figure 5.55 displays the form that will be inserted on the change/add form page of the related application, in this case the one related is the contacts app. Figure 5.54: Extra fields app admin add form 89 Figure 5.55: Extra fields dynamic form inserted on contacts change form 5.17 Global interactions This section provides an overall workflow overview in order to give a better understanding of how the different applications work together. Two use cases, or situations, are presented (1) Create, update and delete a service and (2) Bill a contact. 5.17.1 Create, update and delete a service Figure 5.56: Apps interaction on save() service Considering that the global interactions performing a create, update or delete over a service are very similar with each other, only an example of updating a VirtualHost is analyzed. Figure 5.56 summarize this process. As we have seen before a service can send 3 different signals depending on the operation type: 1. service created() 2. service updated() 90 3. service deleted() In our example case servide updated() is sent because the object vHost is changed successfully using the web interface. At this point there are two applications listening to these signals: (1) contacts and (2) daemons app. 1. The contacts application behaves a bit different depending on the kind of signal that receives: (a) service created(): creates a new contract and sends contract created() signal (b) service updated(): sends contract updated() signal (c) service deleted(): cancels the related contract and sends a contractdeteled() signal These signals are sent basically for the ordering application who is listening to them in order to keep track of possible changes on the contract related orders. The ordering application will perform an update orders(contract) whenever recieves one of those contract signals. This operation creates new orders and cancels others depending on the changes performed on the service. 2. In the other hand when the daemons application recieves one of the servicexxx() signals it makes a lookup in order to determine what is the hosting server which manage the current service and then executes the save template or the delete templete on the given server. The save template will be executed on service created() and service updated() and the delete - template on service deleted(). This operations on the servers are executed asynchronously using django-celery, and in most cases over an SSH channel. 91 5.17.2 Bill a contact Figure 5.57: Apps interaction on bill() contact 1. When the admin user performs the billing action over a set of contacts the contacts application tells to the ordering application that some contacts must be billed. The date parameter is provided by the admin user and it indicates until what date wants to bill. This is useful to bill multiple periods in advance. The ordering application asks to its billing backend (billing app on our case) to create, first the bill lines, and then the bill with these bill lines. 2. Once the invoice is created the admin user can perform a send action over it. This will (1) send a mail with an electronic copy of the bill to the contact and (2) charge the bill value to the contact through the correct payment gateway. This last step is not yet implemented. It is planned to run all these operations with the task queue (celery). 92 Chapter 6 Evaluation In this chapter an evaluation of the whole project is presented: the dedicated work in terms of hours and economic costs, the implementation with tests and requirements fit (special attention to the reusability since is the main requirement of this project) and at the end of the chapter you can find the planned work to do beyond this thesis. 6.1 Development effort In this section we are going to show the whole development process in terms of dedicated work. From the very beginning a Subversion (SVN1) repository has been set up for tracking all the changes on the source code and other related files of the project. Now we can take advantage of this version control system creating some graphs, using svnplot2, for visual representation of the activity on the repository. The repository activity can show us different development phases, like learning, design, coding, refactoring and documentation. The presented graphs are: 1. Figure 6.1 Loc and Churn represents the lines of code (blue line) and code churn (red lines, and it means the lines added, modified or deleted to a file from one version to another). For instance, a high churn might be caused by a code refactoring; or a high increase on the lines of code may indicate that a new project branch has been created. 2. Figure 6.2 Commit Activity Index reflects the number of commits per day. 3. Figure 6.3 File Count shows the number of existing files. It might be the sign of the start of a new code branch or the creation of a new Django application. 1http://subversion.tigris.org 2http://code.google.com/p/svnplot 93 1. The billing system is designed to be compatible with Spanish billing regulatory rules but all of these regulatory specifics are moved on the settings file and each organization should be able to adjust these configuration parameters in order to satisfy the regulations of their country. 2. Also for each bill type the following attributes can be configured on the settings file: which billing template do you want to use, prefix and number of digits of your bill id’s, due date in days and the directory where to store the PDF version of your bills. Service applications •The overall reusable decisions for all the service applications is that we put the generic attributes on the main service application and moved the extra stuff to service plugins and the specifics to a settings file. This way we ensure maximum reusability of each component. •As we have seen in the 5.2 Common app section we have extremely reduced the complexity of creating a new service in such a way that you do not need to have advanced programing skills to be able to develop a new service or create an extension. Also the fact that we have moved the common and basic service functionalities on separate applications makes it not necessary to care about the resource control of the new application, neither the contract feature nor billing or ordering. You only need to focus on your application specifics. •Third party Django applications can be easily reused as a new service for our control panel by using our Service converter application, previously seen on page 76 section 5.8. 6.5 Requirements compliance The presented table 6.5 is an update of the requirements fit summary, table 2.2, presented before on chapter 2, page 29. Our panel is included in order to have a requirement compliance comparison with other existing open source solutions. 100 Table 6.1: Requirements compliance summary (with our solution) Requirement SysCP DTC ISPConfig2 Our 1 Contacts management Ok Ok Ok Ok 2 Services Partial Partial Fail Ok 3 Advance pricing configurations Fail Fail Fail Ok 4 Billing system Fail Fail Fail Ok 5 Payment gateway Fail Fail Fail Fail1 7 Resource limiting and accounting Ok Ok Ok Ok 8 Target software to support Partial Partial Partial Partial2 9 Multi server support Partial Fail Fail Ok 10 Easy to use for unskilled users Ok Fail Ok ?3 11 Easy to add new functionalities Partial Fail Fail Ok 12 Reusable Fail Fail Partial Ok 13 Open source friendly Ok Ok Ok Ok 1The Payment gateway is not yet implemented. 2Some software still remains without support, like mailing lists or databases 3The user interface is not yet implemented, so we can not say if it is easy to use or not :) 6.6 General evaluation As we have seen on the previous section 6.4 Reusablity evaluation, we have came up with a solution that can be reusable beyond Pangea organization. The solution can be easily adopted in many different contexts, such as (1) corporations that provide hosting services, (2) any non profit hosting organization like Pangea, e.g. Nodo507 or Greenet8, (3) for personal use and (4) also for reusing in other projects that have nothing to do with hosting services (think about how many organizations need contact management or a billing system). During the development of this project we have used some powerful concepts that have not been seen before on an open source web hosting control panel. Like dynamic resource control, truly multi server support, extremely easy creation of new services, message queuing for asynchronous tasks execution, oriented to be easy to integrate on an existing IT infrastructure, a lot of pricing configurations and so on. This project does not end with this thesis, I will continue working on it until the control panel becomes production ready. In the next section we discuss the planned work to do on the next months. 7http://info.nodo50.org/ 8http://www.gn.apc.org/ 101 6.7 Future work Unfortunately it still remains some work to be done before releasing an stable version of this software to the open source community. Below you can find a detailed list of the work that is planned to be done on the short run and in long term too. Planned work for the first stable release. 1. Some code and APIs should be re-factored and cleaned up before the release. 2. Write Python unittests for all applications. 3. Create the databases application which adds support for database management, also for database user management and permissions. 4. Implement the payment application as an extensible payment interface for multiple payment gateways, implement the Q19 bank procedure and provide hooks to reuse third party payment applications such as: •django-paypal9, which is a pluggable application that implements PayPal Payments Standard and Payments Pro. •django-authorizenet10, which implements Django integration with Authorize.NET payment gateway. Includes SIM and AIM implementations •Satchmo11 payment application. Satchmo is an ecommerce framework built on the Django framework, it includes a payment application with support for several different payment modules including Authorize.net, TrustCommerce, CyberSource, PayPal, Google Checkout, Sage Pay (Formerly Protx) and Sermepa. 5. Implement the APPS app. This application should implements the concept of SaaS “Software as a Service12” one click application installer. This application should be able to automatically install any kind of software, previously packaged. For example the packages provided by www.apsstandard.org. 6. Develop the Mailing lists app, which provides support for mailing lists. 9django-paypal project page http://github.com/johnboxall/django-paypal 10django-authorizenet project site http://github.com/zen4ever/django-authorizenet 11Satchmo project web page http://www.satchmoproject.com 12http://en.wikipedia.org/wiki/Software as a service 102 7. The VPS app, which provides support for creating and managing Virtual Private Servers, also known as virtual machine service. 8. Asynchronous tasks on billing system. 9. Create the remaining daemons and resource control scripts. 10. Give a name to this project. Long term interesting improvements. 1. Create an automatic interface for panel users like django.contrib.admin. 2. Integrate a ticket system. 3. Also it would be interesting to set up a web site for hosting the project and some community tools, such as a ticket system, source code browsing, mailing list, and so on. A sharing templates web page will be interesting too. 4. Reuse django-reversion which allow undo delete and change operations through the admin interface. 5. Provide REST API using django-piston or django-rest-framework in order to allow communication and management from other programs. 103 104 Appendix A Examples A.1 Template example #!/bin/bash echo "{{ object.name }}. IN SOA web.pangea.ORG. mail.pangea.ORG. ( {{ object.serial }} {{ object.slave_refresh }} ’{{ object.slave_retry }}’ {{ object.slave_expiration }} {{ object.min_caching_time }}) {% for record in object.record_set.all %}@ {{ record.type }} IN {{ record.value }} {% endfor %} {% for subobject in object.subobject_set.all %} {% for record in subobject.record_set.all %} {{ subobject }} {{ record.name }} IN {{ record.value }} {% endfor %} {% endfor %}" > /etc/bind/master/{{ object.name }} if [[ $(grep "^{{ object.name }}$" /etc/bind/primarios.conf ]]; then echo {{ object.name }} >> /etc/bind/primarios.conf; fi rndc reload A.2 Invoice example A.3 Fee example 105 Figure A.1: Invoice example 106 Figure A.2: Fee example 107 108 Appendix B Manuals B.1 Installation On our SVN repository you can find a bash script called deploy dev.sh1that automatizes the tedious work of installing and setting up a software project and their dependencies. Please notice that the deployment performed by this script is not suitable for production, it is only for development and testing purpose. As you can see on figure B.1 deploy dev.sh script provides support for three different deployment types (only for Linux platforms): 1. Deploy on your local Debian like distribution, most common are: Debian, Ubuntu and Linux Mint. This option will install the control panel on your own system. 2. Deploy on a Debian chroot (recommended). The control panel and all its dependencies will be installed inside a Debian chroot. A chroot jail is like a virtualized machine but only ensuring filesystem isolation, this allows to install a self contained environment where applications can run. This is useful when you want to install some software with a lot of dependencies and you do not want to mess your system. With this option two additional and optional scripts can be installed on your machine. One on /etc/init.d/ucp chroot that makes easier the task of start and stop the chroot. The second will be installed on /usr/sbin/ucp, this is useful to enter inside the chroot with all the environmental state ready. 1http://ucp.pangea.org/svn/ucp/trunk/ucp/scripts/deploy dev.sh 109