On evaluating health centers groups in Lisbon and Tagus Valley : efficiency, equity and quality

Ferreira et al. BMC Health Services Research 2013, 13:529 http://www.biomedcentral.com/1472-6963/13/529 RESEARCH ARTICLE OpenAccess On evaluating health centers groups in Lisbon and Tagus Valley: efficiency, equity and quality Cláudia Ferreira1*, Rui C Marques1and Paulo Nicola2 Abstract Background: Bearing in mind the increasing health expenses and their weight in the Portuguese gross domestic product, it is of the utmost importance to evaluate the performance of Primary Health Care providers taking into account both efficiency, quality and equity. This paper aims to contribute to a better understanding of the performance of Primary Health Care by measuring it in a Portuguese region (Lisbon and Tagus Valley) and identifying best practices. It also intends to evaluate the quality and equity provided. Methods: For the purpose of measuring the efficiency of the health care centers (ACES) the non-parametric full frontier technique of data envelopment analysis (DEA) was adopted. The recent partial frontier method of order-m was also used to estimate the influence of exogenous variables on the efficiency of the ACES. The horizontal equity was investigated by applying the non-parametric Kruskal-Wallis test with multiple comparisons. Moreover, the quality of service was analyzed by using the ratio between the complaints and the total activity of the ACES. Results: On the whole, a significant level of inefficiency was observed, although there was a general improvement in efficiency between 2009 and 2010. It was found that nursing was the service with the lowest scores. Concerning the horizontal equity, the analysis showed that there is no evidence of relevant disparities between the different subregions(NUTS III). Concerning the exogenous variables, the purchasing power, the percentage of patients aged 65 years old or older and the population size affect the efficiency negatively. Conclusions: This research shows that better usage of the available resources and the creation of a learning network and dissemination of best practices will contribute to improvements in the efficiency of the ACES while maintaining or even improving quality and equity. It was also proved that the market structure does matter when efficiency measurement is addressed. Keywords: Health center groups, Data envelopment analysis, Efficiency, Quality, Equity, Portugal Background Introduction Healthcare is one of the most important areas for citizens. It is also an area where countries spend a significant part of their resources. For this reason, measuring efficiency becomes relevant. In fact, in 2009 it was estimated that the waste of financial resources with the Portuguese Health System was about 25% of the amount allocated to health [1]. Furthermore, in line with other developed countries, an increase on health expenses, both public and private, hasbeenobserved.Thegrowthrateoftheseexpenses *Correspondence: [email protected] 1Instituto Superior Técnico, University of Lisbon, Lisbon, Portugal Full list of author information is available at the end of the article even exceeded the growth rate of the GDP and between 2000 and 2008 it rose almost 50%. This growth largely rests on the changes of the Portuguese demographic structure [2]. This situation has created a constant concern about the sustainability of the Portuguese health system [2] so that it becomes necessary to promote efficiency and innovation by adapting management practices and using financial resources in an optimized way. Despite the fact that the Portuguese health care model is still very dependent on the secondary and differentiated care, the idea that health care systems based on a solid structure of Primary Health Care (PHC) are more costeffective is, nowadays, fairly pacific [3]. Due to this fact, the Portuguese Primary Health Care has suffered many © 2013 Ferreira et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Ferreira et al. BMC Health Services Research 2013, 13:529 Page 2 of 17 http://www.biomedcentral.com/1472-6963/13/529 developments. Additionally, given the current economic situation of Portugal, there is an extra need to evaluate the performance of the public sector and provide value for public money. Health Centers Groups (ACES) productive behavior, and particularly, their efficiency might contribute to this purpose. As stated by Jacobs et al. [4], the efficiency study of health institutions must be a central objective to reduce the public expenditure. According to these authors, the non-parametric frontier technique of Data Envelopment Analysis (DEA) might be an extremely useful tool for estimating the PHC efficiency. Since the mid-1980s, efficiency in health care has been studied by many authors, both at national and international levels. The first application of DEA in the health sector dates back to 1983, in which Nunamaker and Lewin [5] measured the efficiency of the routine nursing service. Since then, DEA has been widely used in the assessment of health care services efficiency. However, the efficiency of PHC has not been a significant priority. Actually, in the literature review performed in this research, only 9 studies were found until 2002 regarding PHC efficiency [6]. The first study on the efficiency of PHC was published in 1989, when Sexton et al. [7] evaluated the managerial efficiency of veterans administration medical centers. The research concluded that around one third of the total number of health centers were inefficient and that the elimination of the referred inefficiency could reduce annual costs by 300 millions of dollars. After that, some studies were performed, including: Huang and McLaughlin [8], Pinillos and Antoñanzas [9], Linna et al. [10], Akazili et al. [11], Kirigia et al. [12], Amado and Santos [13], Sebastian and Lemma [14], Halsteinli et al. [15] and Nuti et al. [16]. A common feature of most of these studies, given the level of health care studied, is the variables chosen. The most frequent inputs are the staff and the expenditure, while the most adopted outputs are the different kinds of consultations related to PHC. Nevertheless, while the oldest studies only focused on efficiency, the most recent ones also take into account the factors that affect efficiency. Concerning quality, the relationship between efficiency and quality of care has had mixed results in prior studies. For example, Helling et al. [17] discovered that increasing efficiency would result in higher quality. On the other hand, Singaroyan et al. [18] found that improving quality of health care may not always lead to efficient operations. Regarding horizontal equity, it is concerned with fairness, which means equal treatment of patients [19]. In the healthcare field, it measures whether patients from different groups have similar access to the services they equally need. In Portugal, there are few studies about the performance of PHC, either related to efficiency, quality or equity. Most of the time, evaluation in Portuguese public services involves rankings, classifications and targets. In fact, Amado and Santos’ study [13] is one of the very few that uses DEA in this scope. So, this study contributes to the literature on several grounds. First, it provides an analysis of efficiency, equity and quality of the PHC in Portugal, which is still an area where big improvements must be made. Second, because at the date of the study of Amado and Santos [13], the PHC in Portugal was not organized as it is now, thus the study of efficiency, equity and quality of the ACES of Lisbon and Tagus Valley (LVT) is of the utmost importance when taking into account the current market structure. Primary health care in Portugal Overview Currently, the Portuguese healthcare system is characterized by three coexisting, overlapping systems: the National Health Service (NHS); public and private insurance schemes for certain professions and private voluntary health insurance. The majority of the population receives health care from the NHS, which was founded in 1979 and aims to provide health care, almost free at the point of delivery, with universal coverage being funded mainly by general taxation. Although centrally financed by the Ministry of Health, the NHS comprises five health administrations: North, Centre, Lisbon and Tagus Valley, Alentejo and Algarve. Their main responsibilities are the development of strategic guidelines; coordination of all aspects of health care provision; supervision of management of hospitals and PHC; establishment of agreements and protocols with private bodies and development of a long-term care network [20]. Now, it is important to distinguish between Primary Care and PHC. Primary Care provides entry into the system for all new needs and problems along with peoplefocused (not disease-oriented) care over time and care for all but with very uncommon or unusual conditions. It also co-ordinates or integrates care provided elsewhere by others [21]. PHC is a conceptual model which refers to disease prevention, health promotion, population health, and community development within a holistic framework, with the aim of providing essential community-focused health care [22]. This is the level of Health Care we are studying in this article. In Portugal, PHC has a history of about 40 years. Since there has been a constant need of implement reforms, it becomes necessary to understand how PHC evolved during the last 40 years. Therefore, it is important to distinguish two different periods: the first, between 1971 and 2004, when the first, second and third generation of health centers were created and the second one, from to 2005 up to now. Ferreira et al. BMC Health Services Research 2013, 13:529 Page 3 of 17 http://www.biomedcentral.com/1472-6963/13/529 From 1971 to 2004 The first legislation concerning health centers is from 1971, seven years before the Alma-Alta declaration. The “First Generation Health Centers” were created under the terms of the Decree-law 412/71. These Health Centers were responsible for ensuring public health: vaccination, women and childrend’s health, pregnancy care and health authority. At this time, the treatment of acute disease was not part of PHC activities. Such health care services were provided by the sickness insurance institutions [23]. Although seemingly paradoxical, regarding the health needs and patients’ expectations these two styles were complementary. That is the reason why a new legislation had been approved in 1983 (Decree-Law 97/83) which amended the preceding decree in order to integrate the health centers and the sickness insurance institutions, thus creating “Second Generation Health Centers”. However, this process was not totally peaceful, mainly due to the significant differences in the physical and human heritage. Because of that, consultations, home visits and health surveillance did not show the expected improvement. Additionally, these structures had very limited management autonomy [24]. To counteract this situation, a structural reorganization of PHC was provided for in 1999 (Decree-Law 157/99), which established an ideal model of a health center, the “Third Generation Health Centers”. However, in practice the situation of dependence was basically the same. This Decree-Law was repealed by the Decree-Law 60/2003, which aimed to improve the primary health care system by creating a network of healthcare provision. This network was supposed to improve the citizens’ evaluation capability. It was also an objective of the network to contribute to the reversal of the conservative policies and aversion to change apparently responsible for the inefficiency of the traditional health system. Nevertheless, this decree has never been implemented. Since 2005 A new phase of the Portuguese health sector started with the enactment of the Decree-Law 88/2005 which temporarily reinstated the one from 1999 and referred to a technical group specifically created to prepare the new health care reform. Since then the Government has begun studying ways of improving efficiency, equity and quality of PHC. During this period, some innovative ideas have arisen, the creation of ACES is an example of great importance. By definition, the ACES are amalgamations of resources and management structures and are composed of different functional units. Their mission is to ensure the provision of PHC in a particular geographic area, enhancing health gains accomplished by Family Health Units (USFs) and providing better management structures. Also, the development of epidemiological surveillance activities, the research on health and the control and evaluation of results are inherent to the ACES’s mission. According to “The Mission for Primary Health Care”, many factors are taken into account when defining the geographic area of influence of the ACES: the number of residents that should be between 50,000 and 200,000, the population structure, the aging index and the accessibility to the reference hospital. Each ACES is led by an Executive Director (DE) and consists of a Clinical Council (CC), a Community Council, a Management Support Unit (UAG) and five functional units, as described in Table 1. Other functional units might be considered, if the Health Regional Administration (ARS) decides so. Table 1 The 5 types of functional units that compose the ACES Functional unit Description USF Individual and Family Care Unit. USFs promote the training of multidisciplinary teams, comprised by doctors, nurses and administrative staff. USFs allow a closer relationship with users through constant and personalized contact. In addition, in USFs all the patients have assigned doctors. Personalized Healthcare Unit (UCSP) Individual and Family Care Unit. In terms of dimension, it is similar to an USF. However, USFs are regulated by specific legislation, whereas UCSP are bounded to rules approved by the Clinic Council. Community Care Unit (UCC) UCCs provide care to groups with special needs and community interventions. UCCs operate in the community and are able of mobilizing skills inherent to other functional units, to provide health care through specific interventions. Public Health Unit (USP) USPs are related to population and environmental and public health. USPs are responsible for the planning and divulgation in public health. They are also in charge of epidemiological surveillance and manages population-wide programs in the domains of prevention, health promotion and protection. Shared Assistential Resources Unit (URAP) URAPs provide and enhance specific support and advice to the functional units and health projects of each ACES. Their mission is to support the former functional units. Ferreira et al. BMC Health Services Research 2013, 13:529 Page 4 of 17 http://www.biomedcentral.com/1472-6963/13/529 Regarding the region of LVT (Figure 1), which corresponds to 13% of the Portuguese territory including the capital Lisbon, it has a population of 3,7 million (34% of the total population) and, according to the Centre for Regional Dynamics Observation (2009), it represents 44% of the national GDP. In this region, there are 22 ACES, organized according to the five existing NUTS III (subregions): •Grande Lisboa: Lisboa Norte (1), Lisboa Oriental (2), Lisboa Central (3), Oeiras (4), Odivelas (5), Loures (6),Amadora(7),Sintra-Mafra(8),Algueirão-Rio Figure 1 Location of Lisbon and Tagus Valley in the Portuguese territory. de Mouro (9), Cacém - Queluz (10), Cascais (11) and Vila Franca de Xira (12); •Península de Setúbal: Almada (13), Seixal - Sesimbra (14), Arco Ribeirinho (15) and Setúbal - Palmela (16); •Oeste: Oeste Norte (17) and Oeste Sul (18); •Médio Tejo: Serra d’Aire (19) and Zêzere (20); •Lezíria do Tejo: Ribatejo (21) and Lezíria II (22). In the next chapter, the adopted methodology is presented: the non-parametric full frontier technique of DEA to study the efficiency of the ACES and the partial frontier technique of order-mso as to adjust the results obtained to the exogenous environment of each ACES. Methods Data collection In this research, we analyzed the activity of the 22 ACES belonging to the LVT, mentioned at the end of Section ‘Background’, during the years 2009 and 2010. Most of the data used in this study was obtained from the ARS of LVT activity reports and the Project SimCidadão report. Both of the reports are freely available on-line. The data regarding staff was obtained from many contacts with the ARS of LVT. The use of this data required a previous authorization given by this organization. It is important to highlight that some inputs concerning costs were not available and that is the reason why most of the inputs are related to the staff. Table 2 shows the variables used as inputs and outputs,aswellastheenvironmentalvariables.Theinputand output variables as well as the environmental variables were chosen taking into account the literature review and the available data. All the environmental variables were collected by analyzing the activity reports of the ACES. Data envelopment analysis In order to study efficiency, we applied DEA, a nonparametric full frontier method based on linear programming technique with data from two consecutive years. This technique is used with the aim of evaluating the efficiency of the Decision Making Units (DMUs), by analysing the optimal combinations between inputs and outputs, i.e. between consumed resources and the resulting services or goods[25].Thismethodologyoptimizeseachindividual observation and builds a production frontier, constituted by the efficient DMUs. Several models might be applied and choices regarding the type of return to scale and the orientation must be made in agreement with the production process that is being analyzed [26]. Concerning the orientation, we decided to use the following three orientations to compare the results and because in healthcare we can consider both points of view: •Input orientation: this orientation is used when it is assumed that we have more control of the inputs, it is Ferreira et al. BMC Health Services Research 2013, 13:529 Page 5 of 17 http://www.biomedcentral.com/1472-6963/13/529 Table 2 Inputs and Outputs used in the DEA models to measure efficiencyand environmental variables Inputs Outputs Environmental variables x1 : doctors’ working hours y1 : number of adult health consultations EV1 : Population x2 : nurses’ working hours y2 : number of speciality consultations EV2 : Population density x3 : administrative staff working hours y3 : number of urgency consultations (SAP, CATUS) EV3 : Percentage of patients aged 65 years old or older y4 : number of home visits by doctors EV4 : Mortality Rate x4 : total costs y5 : total number of consultations EV5 : Percentage of patients without designated doctor EV6 : Distance to the nearest hospital y6 : number of group education sessions EV7 : Purchasing power y7 : number of consultations by nurses y8 : number of injections, vaccinations, curatives and other treatments y9 : number of home visits by nurses y10 : total number of nursing services y11 : total number of public health activities also intended to emphasize the reduction of excessive inputs and we think it is possible to reach the same outputs with fewer inputs [27]. •Output orientation: the output-oriented model is adopted if one considers to have more control on the outputs, for instance, by controlling the reputation or the quality of service and when it is considered to be possible to increase the outputs, without any proportional change in the inputs [27]. •Non-oriented model: this model considers to be possible to reduce the inputs and simultaneously increase the outputs. In this study, we used three possible orientations: inputorientation, output-orientation and also a non-oriented model, the additive one, which is based on slacks, excessive inputs or missing outputs that exists even after the proportional change in the input or the outputs [27]. These orientations were used in connection with both constant returns to scale (CRS) and variable returns to scale (VRS) technologies. In the interest of understanding why both CRS and VRS technologies were used, it is important to clarify both concepts. CRS technology assumes that scale of economies do not change as the size of the service increases while VRS assumes that scale of economies do not change with the size of the service. Thus, CRS technology is not present if a proportional increase in one input can cause greater than a proportionate increase in output. In healthcare, not all of the DMUs are producing atthesamescale,whichmightbeduetomanyfactors, such as financial constraints, technical constrains, or poor organization. The method was chosen depending on the perspective of the analyst. If we believe that the DMU is not operating at the optimal scale, the VRS technology is recommended. However, by calculating both efficiencies, we can determine a third efficiency measure: the scale efficiency, which is defined as the ratio between the efficiency computed with between VRS and CRS: θscale =θVRS/θCRS (1) The efficiency scores (θ)fortheDMUs(j=1, ...,n)are computed for the selected outputs (yrj)andinputs(xij). We used the dual model in order to observe benchmarks and their weights (λ). The (λ) values allow us to understand the return to scale for each DMU. If λ =1, it means that the DMU is working with CRS. If λ < 1, the DMU is working with increasing returns to scale. Finally, if λ > 1, the decreasing return to scale is the technology. In the objective function () is called the non-Archimedean, which is defined as infinitely small. s−and s+represent input and output slacks, respectively (Table 3). In contrast with the parametric techniques, such as the Stochastic Frontier Analysis, the non-parametric techniqueofDEAseemstobeabetteroptionsinceitdoesnot require the specification of the functional form that links the inputs to the outputs. Ferreira et al. BMC Health Services Research 2013, 13:529 Page 6 of 17 http://www.biomedcentral.com/1472-6963/13/529 Table 3 Mathematical formulation of the different models and orientations [27] Orientation CRS VRS Inputs Minimize θ−ε( m  i=1 s− i+ s  r=1 s+ r)subject to 1a. n  j=1 λjxij +s− i=θ∗xi0with i=1, ...,m2a. n  j=1 λjyrj −s+ r=yr0with r=1, ...,s3. λj≥0with j=1, ...,n Minimize θ−ε( m  i=1 s− i+ s  r=1 s+ r)subject to 1a, 2a, 3and4. n  j=1 λj=1 Outputs Maximize φ−ε( m  i=1 s− i+ s  r=1 s+ r)subject to 1b. n  j=1 λjxij +s− i=xi0with i=1, ...,m2b. n  j=1 λjyrj − s+ r=φyr0with r=1, ...,s3. λj≥0withj= 1, ...,n Maximize φ−ε( m  i=1 s− i+ s  r=1 s+ r)subject to 1b, 2b, 3and4. n  j=1 λj=1 Non-oriented Maximize m  i=1 s− i+ s  r=1 s+ rsubject to 1c. n  j=1 λjxij + s− i=xi0i=1, ...,m2c. n  j=1 λjyrj −s+ r= yr0r=1, ...,s3. λj≥0j=1, ...,n m  i=1 s− i+ s  r=1 s+ rsubject to 1c, 2c, 3 and 4. n  j=1 λj=1 Adopted models In the light of data constraints, 4 models were used: medicine, nursing, global and global with total costs. They are presented next: Model I (Medicine): it includes all the ACES’ medical services. The input used was x1andtheoutputswerey1, y2, y3andy4; Model II (Nursing): it comprises all the ACES’ nursing services. The input adopted was x2andtheoutputswere y6, y7, y8andy9; Model III (Global): this model aims to include all the ACES’ services. The input used was x1, x2andx3andthe outputs were y5, y10 and y11; Model IV (Global with total costs): model similar to Model III. However, the input adopted was y4 (total costs), instead of staff number resources. The outputs used were y5, y10 and y11. Thesemodelswereusedinordertoincludethetwo main services of the ACES (medicine and nursing service) and to calculate the global efficiency. The global efficiency is calculated by using two different models because the data regarding the costs was only available for the second year of study. The order-m methodology The environmental variables are exogenous factors that cannot be categorized as inputs or outputs but might affect efficiency [28]. As a result, they are not controlled by the DMU managers. It is extremely important to take these variables into account. If not, biased conclusions might be drawn. Although there is no agreement on the best technique to be used, in this study we used a recent methodology: the order-m[28], because, being a partial frontier method, order-m has proven to be less sensitive than DEA to extreme values and able to overcome the deterministic nature of traditional non-parametric techniques [28]. According to the order-mmethodology, the production process can be described by the joint probability of measure of (X,Y),onspaceRp +×Rq +according to 2. HXY (x,y)=Prob(X≤x,Y≥y)(2) where Xare the inputs and Yare the outputs. Afterwards, in an input orientation context, the likelihood function can be decomposed into two according to Bayes rule (3), and then, efficiency can be computed (4,5,6). HXY (x,y)=Prob(X≤x|Y≥y)Prob(Y≥y) =FX|Y(x|y)SY(y)(3) θ(x,y)=inf{θ|FX|Y(θx|y)>0} =inf{θ|HXY (θx,y)>0}(4) ˆ θ(x,y)=inf{θ|ˆ FX|Y(θx|y)>0}(5) ˆ θm,n(x,y)= ∞  0 (1−ˆ FX|Y,n(ux|y))mdu(6) with ˆ FX|Y,n(ux|y)= n  i=1 I(Xi≤ux,Yi≥y) n  i=1 I(Yi≥y) and I(k)being the indicator function that take the value I(k)=1whenkis true or I(k)=0otherwise. According to Daraio and Simar [28], the inclusion of exogenous variables can easily be done by limiting the Ferreira et al. BMC Health Services Research 2013, 13:529 Page 7 of 17 http://www.biomedcentral.com/1472-6963/13/529 production process to a given value of the exogenous variable (7, 8). HXY (x,y)=FX|Y,Z(x|y,z)SY|Z(y|z)(7) ˆ θm(x,y|z)= ∞  0 (1−ˆ FX|Y,Z,n(ux|y,z))mdu(8) where ˆ FX|Y,Z,n(ux|y,z)= n  i=1 I(Xi≤ux,Yi≥y)K(Z−Zi) h n  i=1 I(Yi≥y)KZ−Zi h,his the bandwidth and K(•)is the kernel function. With the aim of analyzing the influence of the exogenous variable on the production process, a nonparametric smoothed regression of the ratios between the order-m conditional efficiencies and the unconditional efficiencies is applied. If the regression has a positive slope, the exogenous variable has a negative effect on the production process because the environmental variable acts like an “undesirable” output to be produced, requiringtheusageofmoreinputs[28].Otherwise,itwillhave the opposite effect. The Kruskal-Wallis test An important study regarding the PHC is the analysis of the horizontal equity, which concerns fairness or justice in the treatment and measures whether patients from different groups have similar access to the services they equally need [13]. In order to analyze the horizontal equity of access regarding the NUTS III we studied the relationship between the NUTS III and (1) the efficiency and (2) the percentage of patients without a designated doctor. To test the hypothesis of relevant differences in equity, we used the non-parametric Kruskal-Wallis test, with multiple comparisons. The Kruskal-Wallis test is a non-parametric method used to determine whether three or more independent groups are the same or different on some variable of interest [29]. It calculates the probability of being wrong when concluding that there is no difference between three or more treatment groups [30]. The test will assume that the samples are identical and check if the differences found between the groups are genuine and if there are evidences to reject the null hypothesis. The ANOVA test is more powerful if we are studying a normal distribution. However, since we are using a nonparametric method, the Kruskal-Wallis is more appropriate because it does not assume that kind of distribution. Results Sample characteristics The summary statistics for the variables used in the research are described in Table 4. Efficiency and environmental variables Both CRS and VRS technologies concerning input orientation, output orientation and the non-oriented model were used in order to compare the results from each Table 4 Summary statistics of inputs and outputs (year 2010) Variables Average Evolution Median Std. deviation Maximum Minimum (2009 to 2010) x1 3583 -3.9% 3596 1191 6009 1750 x2 3537 -1.6% 3693 706 4655 2170 x3 6116 -5.8% 5828 1709 9905 2660 x4 34147052 ND 34817334 7503483 44872810 17898835 y1 351901 1.8% 355779 79532 517515 189954 y2 75365 -0.1% 73766 26072 158988 39745 y3 47693 -24.24% 47052 32488 105067 2503 y4 1879 3.4% 1788 932 4976 576 y5 476837 -1.9% 451655 109922 685802 276495 y6 918 -5.3% 726 868 1804 1 y7 212289 9.3% 181902 80977 485108 107244 y8 173392 0.7% 147439 73976 317073 78673 y9 50368 4.0% 42213 21870 103511 28695 y10 436967 5.1% 396031 148284 842544 237951 y11 13907 -0.9% 10060 10309 39012 3876 Ferreira et al. BMC Health Services Research 2013, 13:529 Page 8 of 17 http://www.biomedcentral.com/1472-6963/13/529 approach. Tables 5, 6 and 7 summarize the efficiency results for the three kinds of orientation. Tables 8, 9 and 10 shows the λresults analysis, i.e.,the benchmarks and also the returns to scale, as explained in Subsection ‘Data envelopment analysis’. Regarding the environmental variable, the graphic results of the order-m regression are illustrated in Figures 2 and 3. Equity of access Figures 4 and 5 show the results of the first KruskallWallis test: the relationship between the NUTS III and the efficiency. Figure 6 shows the results of the second test: the relationship between the NUTS III and the percentage of patients without a designated doctor. Quality In addition to the efficiency and the equity of access, it is also of utmost importance to evaluate the quality of the services provided. According to Fornell [31], the patients’ level of satisfaction on the PHC is very important to track the quality of service. Therefore, this analysis uses the ratio between the complaints and the total of ACES’ activities. These data are reported in the Project SimCidadão report and can be considered a good indicator of the patient’s level of satisfaction. Figure 7 shows the trade-offs between the VRS efficiency and the ratio addressed above, for the years 2009 and 2010. Discussion Efficiency and the influence of the environment on the efficiency By analyzing the general efficiency results referred to in Subsection ‘Quality’ (Tables 5, 6 and 7 some comments can be made. In general, we can see that the results were very similar for all types of used orientation. Regarding the results themselves, the ACES 17 (Oeste Norte) is one of the most efficient. As can be seen in Table 5, this ACES is referred to 7 times as the most common benchmark, i.e. the best practice for the other ACES. In fact, Model I (CRS) is the only model where this ACES does not provide a unitary efficiency. ACES 2 and 11 are the least efficient. In fact, these ACES have poor results regarding contractualization. The model with the lowest efficiency is Model II (Nursing), where the number of scores below 60% that can be found is maximum. A general improvement of efficiency between 2009 and 2010 can be observed. For instance, in 2009, VRS showed a result of 0.944 for Model III, while in 2010, the same Table 5 Efficiency results for 2009 and 2010 (input orientation) Model I Model II Model III Model IV Year 2009 2010 2009 2010 2009 2010 2010 θCRS average 0.793 0.816 0.764 0.769 0.913 0.935 0.918 min θCRS 0.581 0.569 0.469 0.465 0.761 0.817 0.745 DMUwithminθCRS 2 2 21 11 11 11 2 NumberofefficientDMUs(CRS)46 5688 6 DMUs with θCRS >95% 46 67911 12 DMUs with θCRS <60% 21 5300 0 Most common benchmarks (CRS) 22 22 17 17 17 17 8 θVRS average 0.852 0.857 0.839 0.849 0.944 0.961 0.928 min θVRS 0.585 0.595 0.581 0.520 0.775 0.835 0.754 DMUwithminθVRS 112 21151115 2 Number of efficient DMUs (VRS) 7 8 8 7 11 13 9 DMUs with θVRS >95% 98 891315 12 DMUs with θVRS <60% 11 1100 0 Most common benchmarks (VRS) 15 13 e 22 5 17 17 13 e 17 14 θScale average 0.935 0.954 0.905 0.903 0.968 0.972 0.990 min θScale 0.722 0.783 0.690 0.622 0.837 0.848 0.960 DMUwithminθScale 15 17 9 9 12 5 9 NumberofscaleefficientDMUs46 5688 7 DMUs with θScale >95% 11 17 8 10 17 17 22 DMUs with θScale <60% 00 0000 0 Ferreira et al. BMC Health Services Research 2013, 13:529 Page 9 of 17 http://www.biomedcentral.com/1472-6963/13/529 Table 6 Efficiency results for 2009 and 2010 (output orientation) Model I Model II Model III Model IV Year 2009 2010 2009 2010 2009 2010 2010 θCRS average 1,3010 1,264 1,383 1,380 1,104 1,076 1,099 max θCRS 1,722 1,758 2,132 2,149 1,314 1,224 1,343 DMUwithmaxθCRS 2 2 21 11 11 11 2 Number of efficient DMUs (CRS) 4 6 5 6 8 8 6 DMUs with θCRS <1, 05 4 6 6 7 9 11 12 DMUs with θCRS >1,6 3 365000 Most common Benchmark 22 22 17 17 17 17 8 θVRS average 1,154 1,134 1,307 1,292 1,049 1,037 1,084 max θVRS 1,585 1,590 2,125 2,017 1,306 1,188 1,3367 DMUwithmaxθVRS 5 5 21 11 11 11 2 Number of efficient DMUs (VRS) 7 8 8 7 11 13 9 DMUs with θVRS <1, 05 10 10 8 9 14 15 12 DMUs with θVRS >1,6 0 044000 Most common Benchmark 17 13 17 17 17 17 8 e 14 θScale average 1,132 1,115 1,062 1,070 1,053 1,037 1,014 max θScale 1,432 1,412 1,449 1,607 1,189 1,179 1,094 DMUwithmaxθScale 151991515 Number of scale efficient DMUs 4 6 5 6 8 8 6 DMUs with θScale <1,05 9 101312141621 DMUs with θScale >1,6 0 001000 technology showed a score of 0.96. One of the possible reasons is the start-up of 10 new USFs. Also, 2009 was the foundation year of the ACES and it is likely that they were not as organized as in 2010. Regarding the returns to scale, some important conclusions can also be drawn. By observing Table 8 (and also Tables 9 and 10), it can be seen that as far as Model I is concerned, more than half of the ACES are working with decreasing returns to scale, i.e. they are operating above the optimal scale. ACES 1 and 3 always present decreasing returns to scale for Model I. Actually, both ACES are serving a population over the theoretical limit of 200 000. Probably the effect of diseconomies of scale woulddecreaseifthe3ACESoftheLisbonarea(Lisboa Norte, Lisboa Oriental and Lisboa Central) were split into 4. In contrast, in Model II (nursing), more than 50% of the ACES have increasing returns to scale. This means that concerning the variables studied, in these ACES nursing is operating below the optimal scale. As stated before, adjusting for the environment in efficiency studies is very important. In this paper, the exogenous variables considered are the ones described in Table 3, respectively population density, percentage of patients aged 65 years old or older, mortality rate, percentage of patients without a designated doctor, distance to the nearest hospital and purchasing power. Table 7 Efficiency results for 2009 and 2010 (non-oriented model) Model I Model II Model III Model IV Year 2009 2010 2009 2010 2009 2010 2010 NumberofefficientDMUs(CRS) 4656886 Number of non efficient DMUs (CRS) 18 16 17 16 14 14 16 Most common Benchmark 22 22 17 17 17 17 8 Number of efficient DMUs (VRS) 7 8 8 7 11 13 9 Number of non efficient DMUs (VRS) 15 14 14 15 11 9 13 Most common Benchmark 13 13 17 17 17 17 14 Ferreira et al. BMC Health Services Research 2013, 13:529 Page 16 of 17 http://www.biomedcentral.com/1472-6963/13/529 the ACES Zêzere (20), Ribatejo (21) and also the ACES Oeste Norte (17), previously referred to as a benchmark. Conversely, the most problematic cases are the ACES Cascais (11) and Arco Ribeirinho (15), both located in the region of high complaints and low efficiency. Other examples of problematic cases are the ACES Algueirão - Rio de Mouro (9) and Lezíria II (22). These ACES are efficient only because they have a low level of human resources and because of that they receive massive complaints. As previously mentioned, satisfaction surveys regarding PHC are, at least for the time being, suspended. We intend to include them in this study as soon as they become available. Finally, regarding the exogenous variables, the distance to the nearest hospital has a positive influence on efficiency. In contrast the order-m test showed that the purchasing power, the percentage of patients aged 65 years old or older and the population affect efficiency negatively. In order to strengthen this study, it should be repeated with more information and more robust data in the future. For instance, it would be important to include more data regarding costs and health results, such as prevention and control of diabetes. The inclusion in the study of economies of scope would also be a good complement to this work. Also, it would be interesting to extend this study to all the ACES and even compare the efficiency of PHC before and after the reform of the PHC structure. Nevertheless, despite some difficulties and some work that can still be done, this research is important to understand how the ACES have been performing in their first years of activity. This study shows that by establishing a learning network there is room for improving the efficiency of ACES with a better usage of the available resources whilst simultaneously keeping quality and equity. Finally, we should note that an important aspect in this kind of studies is the fact that they allow the assessment of the current structure of the PHC in the country while taking into account not only the size but also other factors which are important for their performance, such as the proximity of hospitals. Competing interests There are no competing interests, financial or non-financial. Authors’ contributions CF was responsible for acquisition of data, carried out the study, interpreted the results and drafted the manuscript. RCM participated in the design and coordination of the study, provided technical help regarding the methodology and helped to review all the manuscript. PN participated in the analysis and interpretation of results and revised all the manuscript from a medical point of view. All authors read and approved the final manuscript. Acknowledgements The authors would like to thank ARS of LVT for its precious cooperation in providing some data. Author details 1Instituto Superior Técnico, University of Lisbon, Lisbon, Portugal. 2Faculty of Medicine, University of Lisbon, Lisbon, Portugal. Received: 20 April 2013 Accepted: 9 December 2013 Published: 21 December 2013 References 1. Nunes R: Regulação da saúde. Oporto: Vida Económic; 2009. 2. Fernandes AC, Costa FL, Barros PP, Ivo RS, Mello S: Saúde em análise: Uma visão para o Futuro. Lisbon: Delloite; 2011. 3. Biscaia A, Martins JN, Ferrinho P, Gonçalves I, Carreira M, Antunes AR: Cuidados de Saúde Primários em Portugal: Reformar para novos sucessos. Lisbon: Climepsi Editores PC; 2009. 4. Jacobs R, Smith PC, Street A: Measuring efficiency in health care: analytic techniques and health policy. New York: Cambridge University Press; 2006. 5. Nunamaker TR, Lewin AY: Measuring routine nursing service efficiency: a comparison of cost per patient day and data envelopment analysis models/comment. Health Serv Res 1983, 18:183–208. 6. Hollingsworth B: Non-parametric and parametric applications measuring efficiency in health care. Health Care Manage Sci 2003, 6:203–218. 7. Sexton TR, Leiken AM, Nolan AH, Liss S, Hogan A, Silkman RH: Evaluating managerial efficiency of veterans administration medical centers using data envelopment analysis. Med Care 1989, 27:1175–1188. 8. Huang YG, McLaughlin CP: Relative efficiency in rural primary health care: an application of data envelopment analysis. Health Serv Res 1989, 24:143–158. 9. Pinillos M, Antoñanzas F: La atención primaria de salud: descentralización y eficiencia. Gaceta Sanitaria 2002, 16:401–407. 10. Linna M, Nordblad A, Koivu M: Technical and cost efficiency of oral health care provision in Finnish health centres. Soc Sci Med 2003, 56:343–353. 11. Akazili J, Adjuik M, Jehu-Appiah C, Zere E: Using data envelopment analysis to measure the extent of technical efficiency of public health centres in Ghana. BMC Int Health Hum Rights 2008, 8:11. 12. Kirigia J, Emrouznejad A, Cassoma B, Asbu E, Barry S: A performance assessment method for hospitals: The case of municipal hospitals in Angola. J Med Syst 2008, 32:509–519. 13. Amado C, Santos S: Challenges for performance assessment and improvement in primary health care: the case of the Portuguese health centres. Health Policy 2010, 91:43–56. 14. Sebastian M, Lemma H: Efficiency of the health extension programme in Tigray, Ethiopia: a data envelopment analysis. BMC Int Health Hum Rights 2010, 10:16. 15. Halsteinli V, Kittelsen SA, Magnussen J: Productivity growth in outpatient child and adolescent mental health services: The impact of case-mix adjustment. Soc Sci Med 2010, 70:439–446. 16. Nuti S, Daraio C, Speroni C, Vainieri M: Relationships between technical efficiency and the quality and costs of health care in Italy. Int J Qual Health Care 2011, 23:324–330. 17. Helling DK, Nelson KM, Ramirez JE, Humphries TL: Kaiser permanente Colorado region pharmacy department: innovative leader in pharmacy practice. J Am Pharm Assoc 2006, 46:67–76. 18. Singaroyan R, Seed CA, Egdell RM: Is a target culture in health care always compatible with efficient use of resources? A cost-effectiveness analysis of an intervention to achieve thrombolysis targets. J Pub Health 2006, 28:31–34. 19. Kaplow L: Horizontal equity: measures in search of a principle. Nat Tax J 1989, 2:139–154. 20. Barros PP, Machado SR, Simoes JA: Portugal health system review. Health Syst Transit 2011, 13:1–156. 21. Starfield B: Primary Care: balancing health needs, services and technology. New York: Oxford University Press; 1998. 22. Declaration of Alma-Ata: International conference on primary health care. USSR: Alma-Ata; September 6–12. [http://www.who.int/publications/ almaata_declaration_en.pdf] 23. Pereira EA, Costa MM: Os centros de saúde em Portugal e o cuidado ao idoso no contexto domiciliário: Estudo de um centro de saúde. Texto contexto - enferm 2007, 16:408–416. Ferreira et al. BMC Health Services Research 2013, 13:529 Page 17 of 17 http://www.biomedcentral.com/1472-6963/13/529 24. Branco AG, Ramos V: Cuidados de saúde primários em Portugal. Revista Portuguesa de Saúde Pública 2011, 2:5–12. 25. Charnes A, Cooper WW, Rhodes E: Measuring the efficiency of decision making units. Eur J Oper Res 1978, 2:429–444. 26. Charnes A, Cooper W, Lewin A, Seiford L: Data Envelopment Analysis: Theory, Methodology and Applications. Massachusetts: Kluwer Academic Press; 1994. 27. Ozcan Y: Health Care Benchmarking and Performance Evaluation: An Assessment using Data Envelopment Analysis (DEA). New York: Springer; 2008. 28. Daraio C, Simar L: Introducing environmental variables in nonparametric frontier models: a probabilistic approach. JProdAnal 2005, 24:93–121. 29. Chan Y, Walmsley RP: Learning and understanding the Kruskal-Wallis one-way analysis-of-variance-by-ranks test for differences among three or more independent groups. Phys Ther 1997, 77:1755–1761. 30. Norheim ET: Kruskal-Wallis test: BASIC computer program to perform nonparametric one-way analysis of variance and multiple comparisons on ranks of several independent samples. Comput Methods Programs Biomed 1986, 23:57–62. 31. Fornell C: Productivity, quality, and costumer satisfaction as strategic success indicators at firm and national level. Adv Strat Manag 1995, 11:217–29. 32. Santos P: Os centros de saúde da terceira geração. Thesis. Universidade Fernando Pessoa: Departamento de Ciência Política e do Comportamento; 2008. 33. Alto Comissariado da Saúde: Estratégias para a Saúde; 2011. [http://pns.dgs.pt/files/2011/02/qs_18-03-2011.pdf] doi:10.1186/1472-6963-13-529 Cite this article as: Ferreira et al.:On evaluating health centers groups in Lisbon and Tagus Valley: efficiency, equity and quality. BMC Health Services Research 2013 13:529. Submit your next manuscript to BioMed Central and take full advantage of: • Convenient online submission • Thorough peer review • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit