The effect of vocational rehabilitation on the employment outcomes of disability insurance beneficiaries: New evidence from Canada

Campolieti, Michele; Gunderson, Morley K. L.; Smith, Jeffrey A. Article The effect of vocational rehabilitation on the employment outcomes of disability insurance beneficiaries: New evidence from Canada IZA Journal of Labor Policy Provided in Cooperation with: IZA – Institute of Labor Economics Suggested Citation: Campolieti, Michele; Gunderson, Morley K. L.; Smith, Jeffrey A. (2014) : The effect of vocational rehabilitation on the employment outcomes of disability insurance beneficiaries: New evidence from Canada, IZA Journal of Labor Policy, ISSN 2193-9004, Springer, Heidelberg, Vol. 3, pp. 1-29, https://doi.org/10.1186/2193-9004-3-10 This Version is available at: https://hdl.handle.net/10419/154684 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by/2.0/ ORIGINAL ARTICLE Open Access The effect of vocational rehabilitation on the employment outcomes of disability insurance beneficiaries: new evidence from Canada Michele Campolieti 1,2 , Morley KL Gunderson 3* and Jeffrey A Smith 4,5 * Correspondence: [email protected] 3 Department of Economics and Centre for Industrial Relations, University of Toronto, Toronto, Canada Full list of author information is available at the end of the article Abstract We estimate the effects of the vocational rehabilitation (VR) program run by the Canada Pension Plan Disability Program using administrative data. Identification relies on “selection on observed variables”plus careful comparison group selection and institutional knowledge regarding sources of conditional variation in participation. We employ several matching and weighting estimators and emphasize flexible conditioning on variables suggested by theory, the institutional setup and the literature. We find modest, and imprecisely estimated, impacts on employment outcomes for men and larger, sometimes statistically significant, impacts for women. A formal sensitivity analysis finds our results are quite robust to lingering selection on unobserved variables. JEL codes: I38; J08; J24 Keywords: Vocational rehabilitation; Disability; Evaluation; Matching 1. Introduction Recently, disability policy has placed an increased focus on the employment of persons with disabilities as a way to generate cost savings via reduced benefit payments. Increasing the return-to-work of disability beneficiaries is particularly important because of shifts in the characteristics of persons collecting disability benefits over the last several decades. First, the share of younger persons on the disability rolls is increasing in many countries (Bound and Burkhauser 1999). This demographic change makes it more difficult for disability beneficiaries to use disability insurance as a bridge to retirement benefits. Second, the nature of disabilities has shifted somewhat over the last several decades. Specifically, disability beneficiaries in Canada and the U.S. increasingly suffer from health problems such as musculoskeletal and soft tissue problems as well as mental disorders that tend to be more chronic in nature, rather than health problems associated with higher mortality rates and shorter claim duration (Bound and Burkhauser 1999; Campolieti 2002; Rupp and Scott 1998). These two changes in the nature of the disabled population both increase the time horizon over which investments in the skills of the disabled can pay off and increase the fraction of the disabled likely to see improvements in their medical conditions that would allow for a return to the labor market. © Campolieti et al.; licensee Springer. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. Campolieti et al. IZA Journal of Labor Policy 2014 2014, 3:10 http://www.izajolp.com/content/3/1/10 A few policy instruments can be used to increase the attachment of disability beneficiaries to the labor market. On the one hand, disability insurance programs can allow disability beneficiaries to engage in work activities and not terminate their benefits as long as they do not cross an earnings threshold. This sort of policy (referred to as benefit offset in the U.S. and an earnings disregard in Canada) can allow disability beneficiaries to earn some labor market income as well as test the waters to determine whether they can work in a sustained fashion. These sorts of initiatives are often bundled with trial work periods and automatic reinstatement options, where if an individual leaves the disability rolls but then determines they are unable to maintain their employment because of their disability they can resume receiving a disability pension. There is some evidence that such incentives can increase the employment of disability beneficiaries (Campolieti and Riddell 2012; Kostøl and Mogstad 2013) as well as their exit rates from the disability rolls (Kostøl and Mogstad 2013). On the other hand, the disability insurance program can also use vocational rehabilitation (e.g., training that upgrades existing skills or provides new skills, often in combination with job search assistance) as an alternative policy instrument to increase the employability of disability insurance beneficiaries. While used extensively by workers’ compensation programs for many decades (Allingham and Hyatt 1995), these programs are a relatively recent addition to disability insurance programs in many countries. For example, the Canada Pension Plan disability (CPPD) program only introduced a vocational rehabilitation program to facilitate the return-to-work of its beneficiaries in the 1990s, well after the CPPD program was created in the mid-1960s. The U.S. also expanded the potential for vocational rehabilitation in its disability policy via the Ticket to Work (TTW) initiative that provides disability beneficiaries with a voucher that can be used to purchase public or private sector employment services (U.S. Social Security Administration 2004). Generally, most countries use a combination of these sorts of policies in their disability strategies. Consequently, disentangling the impact of greater vocational rehabilitation services from financial and other incentives to encourage employment can be somewhat difficult because a client may have received vocational rehabilitation services and also have a number of other incentives influencing his or her decision to return to the labor market. Moreover, in some countries, like the United States, there is a complicated structure of programs available to persons with disabilities and disability beneficiaries that makes it difficult to design, test and implement initiatives (Wittenburg et al. 2013). In contrast, Canada offers a unique opportunity to study the effects of vocational rehabilitation on exit rates from the disability rolls as well as subsequent employment. Canada’s disability insurance program, the Canada Pension Plan Disability (CPPD) program, did not introduce its vocational rehabilitation program, the CPPD-VR program, until the mid-1990s. Moreover, when it did introduce this program it did not have any other incentives (e.g., allowable earnings, trial work periods, or automatic reinstatement options for disability beneficiaries who left the disability rolls) available to disability beneficiaries that might increase their employment. However, the CPPD program began to change its strategy and approach for facilitating the attachment of its beneficiaries to the labor market by beginning to introduce these incentives in 2001 (Campolieti and Riddell 2012). Consequently, there is a short window available in the Canadian data where only vocational rehabilitation services were available to disability beneficiaries. Campolieti et al. IZA Journal of Labor Policy Page 2 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 This paper presents estimates of the effect of the vocational rehabilitation program run by the Canada Pension Plan disability (CPPD) program on the labor market outcomes of disability insurance beneficiaries. Our identification strategy relies on “selection on observed variables”, bolstered in our case by careful selection of the comparison group and by the institutional knowledge that opportunities for participants in the CPPD VR program do not depend on local labor market conditions. We use administrative data from the CPPD program and obtain our estimates with propensity score matching estimators (kernel and local linear), an inverse probability weighting procedure and a genetic matching estimator. Finally, we also provide some cost-benefit computations from the perspective of the government. The remainder of the paper unfolds as follows: Section 2 presents background information on the CPPD program and its vocational rehabilitation program. Section 3 provides a detailed description of the data. Section 4 describes our treatment effect estimators. Section 5 presents our principal empirical findings, a sensitivity analysis and a comparison with previous research. Section 6 contains a calculation of the potential savings in transfer payments for the CPPD program resulting from the CPPD-VR program. We conclude the paper with a brief summary of our results and their implications for policymakers. 2. Background information and previous research The Canada Pension Plan Disability (CPPD) program is the disability component of the Canada Pension Plan (CPP), which was established in 1966. Quebec operates its own program, the Quebec Pension Plan, which also contains a disability component. The CPPD program is quite similar to the U.S. Social Security Administration’s Disability Insurance (DI) program. The CPPD program is available to individuals with severe (preventing the individual from working regularly) and prolonged (a long-term condition or condition likely to result in death) disabilities. In addition, applicants to the CPPD program must also satisfy a contribution requirement, which functions as a recency-of-work requirement (all workers in Canada must make contributions). In other words, applicants must have worked in some of the years leading up to an application for disability benefits. The CPPD program does not place any conditions on eligibility related to the source of the disability. This differs from workers’compensation programs, which focus on the compensation of disabilities arising in the course of employment. Benefits are paid until the maximum age of 65, at which point disability pensions are converted into retirement pensions. The CPPD program reevaluates or reassesses beneficiaries from timeto-time. If this reassessment indicates that an individual no longer has a disability, as defined by the CPPD program, then disability benefits would be terminated as a consequence. The Canada Pension Plan Disability program Vocational Rehabilitation (CPPD-VR) program was established in 1997 as a successor to the National Vocational Rehabilitation program, which evolved from a pilot program that operated from 1992–1997. The goal of VR under CPPD is to facilitate the client’s return to gainful employment and generate cost savings to the CPPD fund. The program is administered by Social Development Canada and was delivered to clients by about 30 case managers in the 1990s. These case managers screen CPPD beneficiaries for their suitability for vocational Campolieti et al. IZA Journal of Labor Policy Page 3 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 rehabilitation and manage the rehabilitation assignments. Specifically, after consulting with the client the case managers develop an individualized plan, which can include a vocational reassessment and planning, skills development and job search assistance. Third-party contractors provide the services and in some cases may also have input into the design of the client’s vocational rehabilitation plan. While the CPPD-VR program is relatively new, VR has been used by other disability programs much more extensively. For example, workers’compensation programs (especially in Canada) often have a strong emphasis on VR in their rehabilitation strategies (Allingham and Hyatt 1995). A number of studies have looked at the effects of VR provided by workers’compensation programs (primarily) and other programs on the labor market outcomes of individuals using data from the United States as well as Canada (e.g., among others, Dean and Dolan 1991a, b; U.S. General Accounting Office 1987, 1994; Gardner 1988; Skaburskis and Collignon 1991; Allingham and Hyatt 1995; Campolieti and Hyatt 2011; Dean et al. 2013a, b). This previous literature has produced a large range of estimates, e.g., some studies indicate that VR is quite effective in improving labor market outcomes, while others find that VR had little or no effect on the labor market outcomes of program participants. However, the interpretation of these findings is complicated by the differences in programs, which may also have different mandates and strategies for rehabilitation, across studies as well as the subject populations (e.g., disability insurance or worker’s compensation recipients versus persons with disabilities who do not collect disability benefits). Another potential concern in many of these earlier studies is that there may be identification problems since many were primarily descriptive (e.g., U.S. General Accounting Office 1987, 1994; Gardner 1988; Skaburskis and Collignon 1991) or utilized bivariate normal selection models or selection models using other distributional assumptions with no (or questionable) exclusion restrictions (Allingham and Hyatt 1995; Campolieti and Hyatt 2011). These identification problems may create biased estimates of the impact of VR on employment outcomes that make it more difficult to make conclusive statements about the effectiveness of VR initiatives, especially when combined with differences in programs and study populations. Wittenburg et al. (2013) review the results from a number of evaluations of demonstration and employment programs for disability beneficiaries and persons with disabilities in the United States, which include the Ticket to Work as well as earlier programs that provided vocational rehabilitation and other supports to various populations of persons with disabilities. They focus on studies using (in their view) rigorous evaluation methods (i.e., primarily relatively compelling non-experimental research designs along with some experiments) so as to minimize identification problems in the interpretation of estimates. At the same time, they consider a broad range of studies and interventions, some of which involve only counseling and case management while others include training. In addition, the programs they consider serve a broad range of study populations and groups. Overall, they did not find that there was much evidence of an effect of these interventions on program participants. However, their review also suggests that interventions targeted at some subpopulations, e.g., younger persons and persons with mental health impairments, are more successful. However, as we noted earlier, they also highlight the difficulties associated with evaluating specific programs in the context of a highly complicated institutional thicket involving multiple agencies and numerous programs and policies. Campolieti et al. IZA Journal of Labor Policy Page 4 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 Aakvik et al. (2005) is one of the few non-experimental studies in the existing literature on VR programs that addresses many of the problems in the earlier studies (see also the more structural Dean et al. (2013a, b) papers). Their identification strategy builds on the assumption that the availability of training slots in the VR program is not correlated with economic conditions in a program participant’s home district. Aakvik et al. (2005) study the VR program in Norway, which pays income support as well as providing training programs to individuals who are unable to return to work after 52 weeks on sickness benefits. Their findings suggest that women who received VR have higher employment rates than those who did not receive training, but their estimates are not very precise. They also conclude that the gains in employment for program participants would be increased if the least employable were encouraged to participate in the program rather than using the existing selection rule that encouraged persons who are deemed to have good employment prospects, i.e., cream-skimming, to participate in the program. 3. Data The data used in this analysis come from the administrative records of the Canada Pension Plan Disability program. The data were drawn from the following files: 1) Master Benefits File (MBF), which includes information on CPPD beneficiaries, such as individual characteristics; 2) Rules Based Reassessment System (RBRS) or reassessment file, which includes information on beneficiaries identified for reassessment, beneficiaries who have been reassessed and those who reported a return to work; 3) Rehabilitation Case Management System (RCMS), which includes a wide range of information on each individual (demographics, type of disability, etc.) who was enrolled in the CPPDVR program; and, 4) Record of Employment Master File (ROEMF), which contains annual information on total income, labor market earnings and contributions to the CPP program from 1990 until 2001. We excluded people who died or turned 65 during the years we focus on since they are no longer eligible for CPPD. The treatment group in our analysis is defined as the cohort of individuals who started the CPPD-VR program during 1998. Some of the individuals in this cohort dropped out of the program and so failed to complete their VR assignments. As we define treatment as starting VR, these individuals remain in our treatment group. We track the post-VR experiences for the treatment group up until 2001. As we discussed earlier, the CPPD program began introducing greater incentives to encourage return to work during and after 2001. Campolieti and Riddell (2012) found that these initiatives were associated with an increase in the employment of disability beneficiaries. Consequently, including data after 2001 would make it difficult to distinguish the effect of VR from these new initiatives. However, we do have a window, i.e., 1998 to 2001, to estimate impacts of the CPPD-VR program that are not contaminated by any other policy. Prior to 1997, VR was offered through the National Vocational Rehabilitation program. The CPPD-VR replaced this program in 1997, but was still ramping up from the previous program. We focus on the 1998 cohort because potential VR clients and case managers would have become more accustomed to the new program structure by 1998, relative to clients and case managers in 1997. As noted in Dean et al. (2013a, b), one could make a distinction between the short- (up to eight quarters after treatment) and long-run (more than eight quarters after Campolieti et al. IZA Journal of Labor Policy Page 5 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 treatment) effects of VR. The short window for our outcome measures (up to three years) means that we consider only the short-run effects of the CPPD-VR program. Viewed as proxies for the long-run estimates that we would prefer to have, our estimates may embody a downward bias if the VR treatment itself takes a long time, and/ or if treated individuals who return to employment only slowly find their footing in the labor market. We can rule out the first of these concerns because the VR program we consider provides services in a concentrated, contiguous block, rather than spreading them out over time as in Dean et al. (2013a, b). We draw our comparison group from the reassessment file (RBRS) and include individuals who entered the reassessment file in 1998. Individuals are reassessed in the CPPD if they are believed to have a high probability of returning to work. Individuals in the reassessment file fit into two broad classifications. The first type of reassessment includes those who had been flagged during their initial application for CPPD benefits as being likely to regain their earnings capacity and be able to return to work (i.e., those initially flagged for reassessment). The second type of reassessment includes persons who are reassessed because of their earnings (i.e., reassessed because of earnings). The CPPD program monitors its beneficiaries for earnings and for contributions to Canada’s unemployment insurance program (called “Employment Insurance”, or EI) through information sharing agreements with the EI program and with Revenue Canada (the government tax agency). Those who are deemed to have earnings or made EI contributions, based on information in these administrative sources, are reassessed. About 46 percent of the individuals in our comparison group got flagged for reassessment at the time of initial application while the remainder were reassessed due to their earnings. We focus on 1998 in order to temporally align the treatment and comparison groups. Temporal alignment ensures that treated and untreated individuals face the same general economic conditions and similar program environments. As individuals in the RBRS database have either returned to work, been reassessed for their potential to return to work or are being reassessed for their potential to return to work, we view them as more comparable to the VR treatment group than other CPPD beneficiaries. We think this substantially reduces the selection problem facing our “selection on observed variables”identification strategy; to the extent that selection bias remains after our conditioning, we expect our comparison group to look “too good”and so bias our estimated treatment effects downward. Like the treatment group, we only track our comparison group through 2001. Dean and Dolan (1991a, b) argue that the preferred comparison group in an evaluation of a VR program consists of clients who enroll in VR services but drop out prior to completion. They argue that using dropouts reduces concerns about selection bias because dropouts and completers share the motivation to apply for the VR program, satisfy the relevant eligibility criteria, and likely have similar levels of (unobserved) severity in their disabilities. We understand their reasoning, but note three concerns with this strategy in our context, one practical and two conceptual. At a practical level, we simply do not have enough dropouts to have sufficient statistical power to detect effects of reasonable size. More conceptually, using dropouts changes the nature of the estimand from the effect of starting VR to the effect of finishing VR. These differ to the extent that partial receipt of VR affects outcomes. In addition, using dropouts may accentuate some selection problems (why do the dropouts drop out?) while at the same Campolieti et al. IZA Journal of Labor Policy Page 6 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 time leaving aside potentially desirable comparison group members who do not attempt VR for reasons unrelated to their potential outcomes. The administrative records contain extensive information on CPP disability insurance recipients. This includes: the age at the onset of the disability, gender, pre-disability educational attainment, province of residence, earnings before entering the disability rolls, earnings after leaving the disability rolls (if they exited), and principal health problem (derived from ICD9 codes). For persons enrolled in VR we also have the total cost of the services provided to these individuals. We examine the effect of VR on several outcome measures. First, we consider the effect of VR on exit from the disability rolls during the period covered by our data. This indicator takes the value one if the individual left the disability rolls (i.e., stopped receiving CPP disability benefits). While we would worry about a “leaving the rolls”outcome in the context of other programs with frequent turnover, such as social assistance in Canada or food stamps in the US, individuals who exit the CPPD rolls rarely return, particularly in the short run. Second, we also consider employment-related outcomes based on two definitions of employment. The first simply captures any employment at any time during the period covered by our data, with no restrictions on the level of earnings. The second measure is defined as substantial gainful employment (SGO), where the individual must have earnings above a certain threshold in one of the calendar years covered by our data 1 .In both cases we code employment based on information in the ROEMF, which contains total calendar year labor market earnings as reported on the T4 tax form 2 . We use these definitions to define indicator variables corresponding to our two measures of employment. 4. Methodology 4.1 Estimating the treatment effect Following the standard notation, let Y 1 denote the outcome for someone who receives the treatment and Y 0 the outcome for someone who does not receive the treatment. Y 1 and Y 0 are potential outcomes because we only observe one for each individual. Let T= 1 indicate whether the person received the treatment, i.e., they received some VR, while T= 0 denotes that the individual is in the comparison group. The average treatment effect on the treated (ATET) in the population equals E[Y 1 −Y 0 |T=1]=E[Y 1 | T=1]−E[Y 0 |T= 1]. We estimate this treatment effect using two propensity score matching estimators, a reweighting estimator based on the estimated propensity score and a genetic matching estimator. We adopt a “selection on observed variables”strategy (Heckman and Robb 1985) to identify the ATET. This approach requires the conditional independence assumption (CIA), i.e., conditional on a set of observed covariates the untreated outcome is independent of participation in the VR program, which is expressed mathematically as Y 0 ⊥ T|X. Note that we need assume conditional independence only for the untreated outcome due to our interest in the ATET; as such, we need not rule out selection into treatment based on the treated outcome. The propensity score is the conditional probability of participating or receiving the treatment, i.e., P(X) = Pr[T=1 | X]. We also require overlap or common support in P(X). We assume this holds in the population and then restrict the analysis so it holds in the sample as well. Matching and reweighting Campolieti et al. IZA Journal of Labor Policy Page 7 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 estimators require that control variables (X) used in the analysis satisfy these conditions. We use economic theory, institutional considerations and a review of previous empirical research to determine the variables or factors that affect participation in VR and the labor market outcomes we consider for our study population, i.e., persons with disabilities who are collecting disability benefits. Our discussion provides a justification for the variables we include in our specification of the propensity score. In some cases, our data do not contain direct measures ofthesefactors;inthesecasesweattempttojustifytheproxiesweuseintheirstead. Following e.g. Smith and Todd (2005), estimating the ATET requires an estimate of the counterfactual E [Y 0 |T= 1]. Matching estimators estimate the counterfactual using some estimator of the form ^ EhY0j ^ PX i ðÞ i¼X J j¼1 w^ PX i ðÞ; ^ PX j  Y0j;ð1Þ where ^ PX i ðÞis the estimated propensity score for individual iin the treated group, ^ PX j  is the estimated propensity score for individual jin the comparison group, jindexes all the individuals in the comparison group, and w(•,•) is a weighting function based on the distance between propensity scores for individual iand the individuals in the comparison group. The form of the weights depends on (one might even say defines) the particular matching estimator used. We use three matching estimators - kernel matching, local linear matching and genetic matching, along with Inverse Probability Weighting (IPW). Using multiple estimators with different strengths and weaknesses but all based on the same identifying assumption of “selection on observed variables”provides a clear indication of the sensitivity of the empirical findings to details of the estimation strategy. Local linear and kernel matching estimators are special cases of the general class of estimators that uses local polynomial regression to estimate the (conditional) expected value of the counterfactual outcome (Heckman et al. 1997). Both the kernel and local linear matching methods take locally weighted averages of the observations in the nontreated group to construct the counterfactual. For example, the kernel estimator computes the weights in equation (1) as w^ PX i ðÞ; ^ PX j  ¼ G ^ PX i ðÞ− ^ PX j ðÞ aN ! XkG ^ PX k ðÞ− ^ PX j  aN ! ð2Þ where G (•) is a kernel function, ^ P•ðÞ is the estimated propensity score, and a N is a bandwidth. The literature suggests that the choice of kernel does not matter much in practice; we use the Epanechnikov kernel. In contrast, the literature clearly indicates that bandwidth selection does matter; we selected our bandwidths using leave-one-out cross-validation as in Black and Smith (2004). We present the bandwidths we use in the kernel and local linear matching approaches in Table 1. As noted in e.g. Todd (2008), local linear matching has some advantages over standard kernel matching. In particular, the local linear approach has a faster rate of convergence near boundary points, which is important when the data feature many propensity scores near zero or one (which is not the case in our application). More Campolieti et al. IZA Journal of Labor Policy Page 8 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 level of educational attainment and a higher proportion of persons with children for both males and females. The primary medical problems differ somewhat between the treatment and comparison groups as well as by gender. For example, in Table 2, males with diseases of the nervous system comprise 26 percent of the treatment group, but nine percent of the comparison group; mental disorders are 11 percent of the treatment group, but 25 percent of the comparison group; and musculoskeletal and soft-tissue disorders are about 21 percent of both the treatment and comparison groups. In Table 3, the summary statistics for women indicate some different patterns: diseases of the nervous system are four percent of the treatment group, but 10 percent of the comparison group; mental disorders are 26 percent of the treatment group, but 38 percent of the comparison group; and musculosketal and soft-tissue problems are 30 percent of the treatment group and 27 percent of the comparison group. The average earnings before entering the CPPD program are lower in the treatment group than the comparison group. Moreover, the differences in average earnings between the treatment and comparison groups are smaller for women. Regional Table 3 Descriptive statistics, women Treatment Comparison Mean Standard deviation Mean Standard deviation Age (at onset of disability) 36.552 7.361 43.423 8.668 Married 0.507 0.504 0.618 0.486 Have children 0.522 0.503 0.410 0.492 [Less than high school] High school 0.552 0.501 0.325 0.469 Post-secondary 0.164 0.373 0.143 0.350 University degree 0.149 0.359 0.097 0.296 [Other] Infectious and parasitic diseases –––– Cancer 0.030 0.171 0.082 0.275 Blood diseases 0.000 0.000 0.000 0.000 Mental disorders 0.254 0.438 0.390 0.488 Diseases of the nervous system 0.045 0.208 0.107 0.309 Circulatory diseases 0.090 0.288 0.052 0.222 Respiratory diseases –––– Diseases of digestive system 0.090 0.288 0.017 0.131 Genitourinary system diseases 0.030 0.171 0.010 0.100 Musculoskeletal and soft-tissue disorders 0.299 0.461 0.263 0.441 Congenital diseases 0.015 0.122 0.013 0.113 Zero earnings 1-year prior to application 0.313 0.467 0.251 0.434 Zero earnings 2-years prior to application 0.254 0.438 0.228 0.420 Average earnings 1-year prior to application (dollars) 10796.25 13260.08 13778.91 15603.86 Average earnings 2-years prior to application (dollars) 13730.81 14902.95 16427.55 16576.09 Provincial unemployment rate 8.012 2.604 8.101 2.350 Duration on CPPD (days) 1242.254 587.261 1024.588 551.253 Number of observations 67 692 Campolieti et al. IZA Journal of Labor Policy Page 15 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 unemployment rates are relatively similar between the treatment and comparison groups. The average time on the CPPD program is also similar for men (1276 days for the treatment group, and 1260 days in the comparison group) and somewhat similar for women (1282 days for the treatment group and 1154 days for the comparison group). We present the standardized differences between the treatment and comparison groups before and after matching in Tables 4 and 5. We present these estimates based on the IPW and the genetic matching algorithm for both men and women. Lee (2013) notes that there is no consensus on how to best show balance and that there are a number of ways to test for balance. However, standardized differences are straightforward to implement and relatively standard in the literature, so we use them. Not surprisingly, the genetic matching algorithm tends to improve covariate balance relative to IPW for many of the observed covariates. Table 6 presents coefficient estimates from the propensity score models for males and females. For males, higher levels of educational attainment are associated with increases in the probability of participating in the CPPD-VR program. A few of the controls for health problems are associated with decreases in the probability of participating in CPPD-VR, relative to the omitted health problems category (endocrine Table 4 Standardized differences in treatment and comparison group, inverse probability weighting Men Women Before matching After matching Before matching After matching Age −67.9 12.7 −61.2 1.8 High school 66.3 0.7 35.7 4.1 College 13.6 3.8 5.5 −4.8 University 3.3 −6.9 13.7 −1.1 Married −4.6 6.1 −8.1 −7.8 Have children 6.7 −4.6 20.8 3.5 Infectious and parasitic diseases 8.1 −1.6 –– Cancer 8.2 −2.5 −14.0 −1.8 Mental disorders −14.6 −0.7 −17.8 −6.8 Diseases of the nervous system 16.9 −2.5 −20.8 −2.0 Circulatory diseases −18.8 8.8 −1.9 −4.8 Respiratory diseases 0.0 −7.6 –– Diseases of digestive system −5.3 −13.4 20.3 23.5 Genitourinary system diseases −9.3 −1.3 5.9 −6.0 Musculoskeletal and soft-tissue disorders −4.4 −1.5 14.2 6.2 Congenital diseases 9.5 14.1 4.7 2.9 Zero earnings 1-year prior to application 9.5 6.2 21.8 10.8 Zero earnings 2-years prior to application 11.6 21.6 14.5 2.4 Earnings 1-year prior to application −18.7 1.0 −9.5 −0.4 Earnings 2-years prior to application −22.7 −13.9 −3.6 5.8 Unemployment rate 0.2 −0.2 −19.0 −8.2 Duration on CPPD program −9.7 1.7 18.2 −2.2 Notes: Entries in the table are standardized differences between the treatment and comparison groups. Campolieti et al. IZA Journal of Labor Policy Page 16 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 disorders, ill-defined causes and injuries and poisonings). Many of the other variables in the propensity score specification are not individually statistically significant, although they are jointly significant. The estimates for women in Table 6 resemble those for men. In particular, higher levels of educational attainment are associated with increases in the probability of participating in the CPPD-VR program and some health problems are associated with decreases in the probability of participating in the CPPDVR, relative to the omitted category. Also like the estimates for males, most of the variables in the propensity score model are not individually statistically significant but the likelihood ratio test statistic for the model as a whole is quite large. 5.2 The average treatment effect on the treated for the CPPD-VR program We present our estimates of the ATET for VR in Table 7 for men and Table 8 for women. We consider three outcomes (leaving the disability rolls, employment and substantial gainful employment) and present estimates of the ATET based on propensity score matching (kernel and local linear), inverse probability weighting and genetic matching. As we noted earlier, we present estimates for three ranges of the propensity score, with our preferred sample containing individuals with propensity scores that lie in [0.001, 0.40] for men and [0.001, 0.35] for women. Table 5 Standardized differences in treatment and comparison group, genetic matching Men Women Before matching After matching Before matching After matching Age −67.9 3.0 −61.2 −9.8 High school 66.3 3.6 35.7 0.0 College 13.6 5.4 5.5 0.0 University 3.3 0.0 13.7 5.8 Married −4.6 7.4 −8.1 −4.1 Have children 6.7 0.0 20.8 −4.1 Infectious and parasitic diseases 8.1 0.0 –– Cancer 8.2 0.0 −14.0 0.0 Mental disorders −14.6 0.0 −17.8 0.0 Diseases of the nervous system 16.9 0.0 −20.8 8.4 Circulatory diseases −18.8 0.0 −1.9 0.0 Respiratory diseases 0.0 0.0 –– Diseases of digestive system −5.3 0.0 20.3 0.0 Genitourinary system diseases −9.3 0.0 5.9 0.0 Musculoskeletal and soft-tissue disorders −4.4 −4.9 14.2 −4.3 Congenital diseases 9.5 13.5 4.7 14.3 Zero earnings 1-year prior to application 9.5 −3.7 21.8 0.0 Zero earnings 2-years prior to application 11.6 −3.7 14.5 −13.1 Earnings 1-year prior to application −18.7 −9.3 −9.5 −5.1 Earnings 2-years prior to application −22.7 −5.8 −3.6 0.7 Unemployment rate 0.2 2.9 −19.0 −9.8 Duration on CPPD program −9.7 −4.4 18.2 −9.6 Notes: Entries in the table are standardized differences between the treatment and comparison groups. Campolieti et al. IZA Journal of Labor Policy Page 17 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 Table 6 Estimates of propensity score, logit model Men Women Coefficient estimate Standard error Coefficient estimate Standard error Age (at onset of disability) 0.519 0.761 −2.667** 1.067 Age 2 −0.009 0.021 0.076*** 0.030 Age 3 0.000 0.000 −0.001*** 0.000 Married 0.413 0.351 −0.427 0.331 Have children 0.334 0.321 0.188 0.339 [Less than high school] High school 2.673*** 0.354 1.766*** 0.428 Post-secondary 1.592*** 0.492 1.311*** 0.529 University degree 2.203*** 0.544 2.169*** 0.590 [Other] Infectious and parasitic diseases −0.885 0.703 –– Cancer −1.250* 0.729 −1.810** 0.877 Mental disorders −1.580*** 0.467 −1.613*** 0.518 Diseases of the nervous system 0.804* 0.423 −1.806** 0.768 Circulatory diseases 0.021 0.548 0.175 0.702 Respiratory diseases 0.530 1.409 –– Diseases of digestive system 0.480 0.785 0.992 0.787 Genitourinary system diseases −1.396 1.129 −0.071 1.046 Musculoskeletal and soft-tissue diseases 0.036 0.447 −0.398 0.496 Congenital diseases −0.158 1.391 −0.509 1.217 Zero earnings 1-year prior to application 0.365 0.451 −0.097 0.524 Zero earnings 2-years prior to application −0.273 0.505 −0.529 0.609 Earnings 1-year prior to application (‘0,000 dollars) 0.000 0.000 0.000** 0.000 Earnings 1-year prior to pplication 2 (‘000 dollars) 0.000 0.000 0.000** 0.000 Earnings 1-year prior to pplication 3 (‘000 dollars) 0.000 0.000 0.000* 0.000 Earnings 2-years prior to application (‘000 dollars) 0.000 0.000 0.000 0.000 Earnings 2-years prior to application 2 (‘000 dollars) 0.000 0.000 0.000 0.000 Earnings 2-years prior to application 3 (‘000 dollars) 0.000 0.000 0.000 0.000 Earnings 1-year prior to application × Earnings 2-years prior to application 0.000 0.000 0.000 0.000 Provincial unemployment rate −0.231 3.602 −1.398 1.502 Provincial unemployment rate 2 0.095 0.418 0.119 0.145 Provincial unemployment rate 3 −0.006 0.016 −0.003 0.004 Duration on CPPD Program (‘000 days) −0.003 0.003 −0.003 0.003 Duration on CPPD Program 2 (‘000 days) 0.000 0.000 0.000 0.000 Duration on CPPD Program 3 (‘000 days) 0.000 0.000 0.000 0.000 Likelihood ratio test statistic 227.4 {<0.001} 163.9 {<0.001} Value of log likelihood −198.15 −179.51 Sample size 979 759 Notes: ***denotes statistically significance at the 1 percent level. **denotes statistically significance at the 5 percent level. *denotes statistically significance at the 10 percent level. Square braces contain omitted reference category for dummy variables. Braces contain p-value for likelihood ratio test statistics. Campolieti et al. IZA Journal of Labor Policy Page 18 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 The estimates for men in Table 7 indicate that VR has a relatively small effect on the outcomes we consider. Putting aside the genetic matching estimates for the moment, the other estimators suggest impacts on leaving the disability rolls and on gainful employment of about five or six percentage points. They indicate a smaller impact on substantial gainful employment of about two percentage points. None of these estimates differ statistically from zero, though of course the point estimates remain the preferred estimates. Mucking around with the common support does little to change the story. The estimates for women in Table 8 differ from those for men: they are larger in magnitude and more often attain conventional levels of statistical significance. Looking across estimators, the estimates of the ATET for the leaving the disability rolls and gainful employment outcomes average a bit over ten percentage points, while the substantial gainful employment ATETs average around 16 percentage points. The former estimates do not attain statistical significance, while the latter do, sometimes at the five Table 7 Estimates of the ATET of VR on individual outcomes, men Matching estimator Inverse probability weighting Genetic matching Kernel Local linear Sample restricted to propensity score values [0.001, 0.40] Leaving disability rolls 0.057 0.073 0.039 −0.036 (0.082) (0.113) (0.076) (0.096) Gainful employment 0.062 0.070 0.030 −0.073 (0.080) (0.099) (0.077) (0.096) Substantial gainful employment 0.025 0.035 0.001 −0.055 (0.078) (0.097) (0.074) (0.098) Sample size 786 786 786 786 Sample restricted to propensity score values [0.001, 0.35] Leaving disability rolls 0.014 0.015 0.003 −0.067 (0.083) (0.082) (0.084) (0.111) Gainful employment 0.022 0.025 0.007 0.000 (0.081) (0.079) (0.083) (0.109) Substantial gainful employment 0.009 0.026 0.013 −0.022 (0.083) (0.078) (0.083) (0.111) Sample size 774 774 774 774 Sample restricted to propensity score values [0.001, 0.50] Leaving disability rolls 0.033 0.044 0.018 0.000 (0.082) (0.085) (0.072) (0.101) Gainful employment 0.024 0.027 −0.008 −0.017 (0.081) (0.088) (0.073) (0.097) Substantial gainful employment −0.008 −0.005 −0.033 −0.049 (0.078) (0.085) (0.073) (0.100) Sample size 805 805 805 805 Notes: ATET denotes average treatment effect on the treated. Observations with propensity score values outside the range in square brackets are omitted from the sample used to estimate the ATET. Standard errors in parentheses. The kernel and local linear matching estimates were obtained with the Epanechnikov kernel, with bandwidths computed using cross-validation; see Table 1for the bandwidths. Genetic matching estimates are based on single nearest neighbor matching with replacement. Standard errors for kernel and local linear matching and inverse probability weighting are obtained by bootstrapping with 1000 replications. Standard errors for genetic matching are based on Abadie and Imbens (2006). Campolieti et al. IZA Journal of Labor Policy Page 19 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 percent level and sometimes at the 10 percent level. Changing the imposed common support region does not change the overall picture. Looking across estimators in Tables 7 and 8, the genetic matching estimates end up outliers in both cases: on the high side for women and on the low side for men. The genetic matching estimates have the feature that they optimize balance and the feature that they build on a single nearest neighbor matching estimator that does not use as much of the information available in the comparison group as the other three estimators. For this reason, we tend to discount the estimates from the genetic matching estimator relative to the other three. Substantively, the estimates reveal a large difference in impacts between men and women and, not unrelated, estimates for women large enough to cast doubt on the validity of our identification strategy. The estimates for women seem a bit large relative to a casual prior based on the nature of the treatment and of the participants’underlying conditions. To address our concerns about the magnitude of some of the estimates, and because we think it represents good empirical practice more generally, we turn now to an analysis of the sensitivity of our estimates to lingering selection on unobserved variables not accounted for by our choice of comparison group and our Table 8 Estimates of the ATET of VR on individual outcomes, women Matching estimator Inverse probability weighting Genetic matching Kernel Local linear Sample restricted to propensity score values [0.001, 0.35] Leaving disability rolls 0.104 0.104 0.095 0.122 (0.086) (0.084) (0.080) (0.111) Gainful employment 0.110 0.109 0.108 0.143 (0.082) (0.081) (0.078) (0.104) Substantial gainful employment 0.169** 0.158** 0.144* 0.184* (0.079) (0.079) (0.078) (0.103) Sample size 631 631 631 631 Sample restricted to propensity score values [0.001, 0.30] Leaving disability rolls 0.105 0.085 0.096 0.071 (0.088) (0.088) (0.087) (0.120) Gainful employment 0.133 0.120 0.130 0.143 (0.087) (0.087) (0.084) (0.122) Substantial gainful employment 0.180** 0.156* 0.159* 0.167 (0.083) (0.085) (0.084) (0.118) Sample size 615 615 615 615 Sample restricted to propensity score values [0.001,0.40] Leaving disability rolls 0.109 0.093 0.093 0.077 (0.086) (0.081) (0.078) (0.105) Gainful employment 0.147 0.137 0.138 0.154 (0.085) (0.076) (0.074) (0.099) Substantial gainful employment 0.195** 0.175** 0.171** 0.192** (0.081) (0.075) (0.074) (0.098) Sample size 642 642 642 642 Notes: * denotes statistical significance at the 10 percent level; ** denotes statistical significance at the 5 percent level. See notes for Table 7. Campolieti et al. IZA Journal of Labor Policy Page 20 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 conditioning variables. Later in the next section, we place our estimates in the broader context of the literature. 5.3 Sensitivity analysis and comparison with previous and related research This section presents the results of a sensitivity analysis using the approach in Ichino et al. (2008). Their approach builds on earlier work by Rosenbaum and Rubin (1983) and Rosenbaum (1987), who consider the robustness of estimates of the ATET to assumptions with respect to a binary unobserved variable associated with both the treatment and outcome variables. Within the sensitivity analysis, the CIA holds when the variable is included and fails to hold when it is not included. Consequently, estimating the ATET including and excluding this variable from the conditioning set determines the sensitivity of the estimates of the ATET to the unobserved variable. Ichino et al. (2008) build on this framework and extend it so they can provide point estimates of the ATET under different assumptions about the distribution of the unobserved variable, rather than bounds as in the earlier literature. We present the estimates from the sensitivity analysis in Tables 9 and 10. These tables present the estimates of the ATET by outcome measure. We present these estimates for various assumptions about the unobserved variable or confounder, denoted U. Each panel of estimates includes an estimate for the case with “no confounder”, which means that the CIA holds and that no relevant variable has been excluded from the conditioning set and a “neutral confounder”, which has a distribution that is calibrated so that it has no net effect on the untreated outcome and no effect on selection into treatment and so functions as a sort of placebo test. We also selected a few variables to simulate as confounders, which are listed in the rows of the tables. Unfortunately, the methodology we use restricts us to using discrete variables for the sensitivity analysis. We present estimates of the ATET as well as the quantities referred to as the “outcome”and “selection”effects in Ichino et al. (2008). The outcome and selection effects are odds ratios from logit models that estimate P(Y=1|T=0,X,U) and P(T=1| X, U), where Uis the confounder and the other variables are as previously defined. The outcome effect measures the effect of Uon the untreated outcome controlling for observed covariates. The selection effect measures the effect of Uon assignment to treatment controlling for observed covariates. The outcome and selection effects are used in the sensitivity analysis to benchmark how strong the confounder needs to be to change the substantive importance of the ATET or its statistical significance. The estimates we examine for the sensitivity analysis are the kernel matching estimates 3 . The sensitivity analysis for men in Table 9 indicates that the estimates of the ATET with the simulated confounders are the same as those with no unobserved confounding, but the precision of the estimates varies somewhat with the confounders. As with our estimates in Table 7, none of the estimates of the ATET are statistically significant. For women, we also see very robust findings, i.e., there is not a great deal of sensitivity of our estimates to the simulated confounder. For women, the ATET do not vary by the confounders we consider, but the precision of the estimates is affected somewhat for some confounders 4 . As we noted earlier, we are restricted to using binary variables in our sensitivity analysis. One explanation for our finding of no sensitivity is that the variables we selected are not strongly correlated with both treatment choice and outcomes. We feel that some of the continuous variables in our conditioning set, such as Campolieti et al. IZA Journal of Labor Policy Page 21 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 pre-program earnings and spell duration on the CPPD program, would be better candidates for confounders since they are proxies for otherwise unobserved variables. Still, we find a remarkable lack of sensitivity. Table 9 Sensitivity analysis, calibrated confounders for kernel matching estimate of ATET, men Outcome effect Selection effect ATET SE Leaving disability rolls No confounder ––0.065 0.073 Neutral confounder 1.008 1.051 0.065 0.071 Confounders High school 1.703 3.508 0.065 0.081 College 3.194 2.213 0.065 0.090 University 1.566 1.325 0.065 0.083 Married 0.601 0.862 0.065 0.074 Mental disorders 0.750 0.812 0.065 0.084 Musculoskeletal disorders 0.660 0.972 0.065 0.081 Zero earnings 1-year prior to application 0.641 1.322 0.065 0.081 Zero earnings 2-years prior to application 0.713 1.515 0.065 0.074 Gainful employment No confounder ––0.074 0.081 Neutral confounder 1.011 1.028 0.074 0.084 Confounders High school 1.981 3.640 0.074 0.071 College 2.990 2.175 0.074 0.083 University 1.247 1.278 0.074 0.072 Married 0.560 0.868 0.074 0.078 Mental disorders 0.862 0.805 0.074 0.082 Musculoskeletal disorders 0.599 0.994 0.074 0.083 Zero earnings 1-year prior to application 0.554 1.304 0.074 0.071 Zero earnings 2-years prior to application 0.647 1.509 0.074 0.077 Substantial gainful employment No confounder ––0.051 0.075 Neutral confounder 1.017 0.051 0.069 Confounders High school 1.852 3.634 0.051 0.069 College 2.946 2.196 0.051 0.065 University 1.352 1.298 0.051 0.087 Married 0.569 0.872 0.051 0.062 Mental disorders 0.085 0.797 0.051 0.073 Musculoskeletal disorders 0.639 0.974 0.051 0.069 Zero earnings 1-year prior to application 0.510 1.300 0.051 0.087 Zero earnings 2-years prior to application 0.593 1.532 0.051 0.070 Notes: ATET denotes average treatment effect on the treated; SE denotes standard error. The outcome and selection effects are odds ratios. The rows in the tables denote the confounding variable, the first two rows in each panel of estimates show the ATET with no cofounding factor and with a neutral confounder. The kernel and local linear matching estimates were obtained with the Epanechnikov kernel, with bandwidths computed using cross-validation. All standard errors are obtained by bootstrapping with 1000 replications. Campolieti et al. IZA Journal of Labor Policy Page 22 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 Our estimates indicate that VR has a larger effect on the likelihood of leaving the disability rolls or finding employment for women. Our findings are not unlike the findings from the literature examining active labor market programs that has generally found that such programs have a larger effect on the labor market outcomes of women than those of men (e.g., Heckman et al. 1999). However, the large difference in the size of the estimates for men and women, as well as the absolute size of the estimated ATET for women, make Table 10 Sensitivity analysis, calibrated confounders for kernel matching estimate of ATET, women Outcome effect Selection effect ATET SE Leaving disability rolls No confounder ––0.103 0.068 Neutral confounder 1.018 1.034 0.103 0.078 Confounders High school 1.295 2.235 0.103 0.067 College 1.464 1.219 0.103 0.078 University 1.358 1.703 0.103 0.077 Married 0.854 0.885 0.103 0.074 Mental disorders 1.045 0.727 0.103 0.069 Musculoskeletal disorders 0.652 1.433 0.103 0.088 Zero earnings 1-year prior to application 0.468 1.712 0.103 0.090 Zero earnings 2-years prior to application 1.064 1.496 0.103 0.082 Gainful employment No confounder ––0.141** 0.069 Neutral confounder 1.016 1.043 0.141 0.089 Confounders High school 1.166 2.232 0.141** 0.069 College 1.624 1.198 0.141** 0.070 University 1.434 1.689 0.141** 0.068 Married 1.031 0.889 0.141** 0.063 Mental disorders 0.904 0.707 0.141** 0.07 Musculoskeletal disorders 0.569 1.456 0.141 0.088 Zero earnings 1-year prior to application 0.288 1.747 0.141* 0.080 Zero earnings 2-years prior to application 0.704 1.466 0.141** 0.070 Substantial gainful employment No confounder ––0.200*** 0.066 Neutral confounder 1.019 1.031 0.200** 0.079 Confounders High school 1.176 2.202 0.200** 0.082 College 1.551 1.214 0.200** 0.077 University 1.592 1.678 0.200** 0.081 Married 0.961 0.899 0.200** 0.077 Mental disorders 0.998 0.716 0.200** 0.075 Musculoskeletal disorders 0.529 1.431 0.200*** 0.069 Zero earnings 1-year prior to application 0.241 1.703 0.200*** 0.066 Zero earnings 2-years prior to application 0.643 1.455 0.200** 0.079 Notes: See the notes to Table 9. Campolieti et al. IZA Journal of Labor Policy Page 23 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10 us somewhat cautious about drawing strong conclusions about the improvement in employment outcomes of women after receiving VR. While we use a large and very relevant set of conditioning variables motivated by theory, institutions and the prior literature, in combination with a flexible functional form, some selection bias may remain in our estimates. We would have liked to condition on a measure of the severity of each individual’s disability, on some measure of ability (i.e. a test score), on measures of non-cognitive skills, and so on. We think that using pre-disability labor market outcomes captures much of what we would want from these variables, butitmaynotgoalltheway.WealsoworrythatdurationonCPPDmaynotfully capture the variation in human capital depreciation across individuals. Alternatively, as we noted earlier, drawing our comparison group from the reassessment file may not accomplish what we intend it to accomplish if individuals in the reassessment file differ in, say, unobserved motivation to return to work. Most of the previous literature looking at VR has examined programs with different rehabilitation strategies as well as different client bases and many of these studies may also be sensitive to specification errors or contaminated by other problems, as they sometimes do not use very transparent identification strategies. However, our results do have some parallels in some more recent empirical work studying VR programs with more clearly defined identification strategies. For example, Aakvik et al. (2005) find an (imprecisely estimated) increase in the employment of women in their study of the effects of the Norwegian VR program, though it is not as large as ours. Aakvik (2001) also finds positive (but imprecisely estimated) effects of the Norwegian VR program on employment for a pooled sample of men and women, with a point estimate of 6.3 percentage points, but interprets his estimates with caution due to lingering worries about selection bias resulting from his bounding analysis. We also estimated the ATET using an IPW estimator with a pooled sample of men and women and obtained estimates similar to those in Aakvik (2001). More specifically, we obtained an ATET of 0.061 for the gainful employment outcome and 0.052 for the substantial gainful employment outcome, but these estimates did not statistically differ from zero. Our estimates of the effect of VR are also quite interesting in comparison to estimates from studies looking at financial and non-financial incentives to increase the attachment of disability beneficiaries to the labor market. Campolieti and Riddell (2012) found that the introduction of the CPPD earnings disregard (similar to the benefit offset in the SSDI program in the U.S.) increased the employment of disability beneficiaries. Campolieti and Riddell’s preferred difference-in-difference estimates show that the introduction of the earnings disregard and automatic reinstatement option increase the employment of men by 5.1 percentage points and women by 9.5 percentage points, relative to the Quebec Pension Plan Disability program. One concern about the introduction of such incentives is the increased uptake of benefits, which is also referred to as the induced entry effect (e.g. Hoynes and Moffitt 1999). However, Campolieti and Riddell (2012) did not find any evidence of the increased receipt of disability benefits in their analysis of flows onto the disability rolls. Kostøl and Mogstad (2013) looked at the introduction of a benefit offset in Norway, which is very similar to the benefit offset used in the SSDI program, exploiting a discontinuity created by the eligibility rules for these new incentives, and found that there was a five to six percentage point increase in employment after the introduction of the benefit offset. They also found that the Campolieti et al. IZA Journal of Labor Policy Page 24 of 29 2014, 3:10 http://www.izajolp.com/content/3/1/10