USING VOUCHERS IN GOVERNMENT TRAINING PROGRAMS: FINDINGS FROM THE INDIVIDUAL TRAINING ACCOUNT EXPERIMENT

Sheena McConnell, Irma Perez-Johnson, and Paul Decker

October 2005

This research was funded under a contract from the US. Department of Labor (contract number N-7731-9-00-87-30). The views expressed herein do not necessarily reflect the policies or opinions of the U.S. Department of Labor.

Vouchers are increasingly being used to deliver government-funded goods and services.

Vouchers meet the government’s desire to target consumption by limiting the consumers’

choices to a specified set of goods or services; at the same time vouchers allow consumers to

choose the type of goods or services and their provider that is best for them. Examples of

government-funded goods and services delivered by vouchers include food (such as food stamps

and WIC), childcare, housing to low-income families (such as Section 8 housing), K-12

schooling, job retention services (such as clothing, gas, and computer training), and prescription

drugs (Posner et al. 2000). Since implementation of the Workforce Investment Act of 1998

(WIA), vouchers have also been used for government-funded occupation training.

The effectiveness of any voucher program depends on how the vouchers are administered.

Vouchers vary in how they are administered in three main ways. First, they differ in the amount

of information and assistance consumers are provided to help them make informed choices. In

some programs, little information is provided; while in others, extensive information is provided.

In childcare programs, for example, the government usually provides families a guide to local

childcare providers. The second way that the administration of vouchers varies is the extent to

which the program ensures a minimum level of the quality of goods and services “purchased”

with the vouchers. Typically, the choice of goods and services are restricted to those that have

met certain minimum quality standards. The U.S. Department of Housing and Urban

Development, for example, requires that housing units that can be rented using Section 8

vouchers meet certain standards for health and safety. The third source of variation is how the

vouchers are used to ration goods or services among consumers. Thus vouchers vary in their

value, whether the value is the same for all consumers or varies by consumers’ needs or other

factors, and the extent to which the consumers are required to contribute payment for the goods

or services.

2

This paper explores the impacts of different approaches to administering vouchers—or

individual training accounts (ITAs)--for occupation training under WIA. As WIA requires that

local workforce agencies determine their own approach to implementing ITAs, this study is of

particular policy interest. It is also interesting given the Bush Administration’s interest in a new

but similar voucher initiative known as Personal Reemployment Accounts (PRAs). PRAs are

vouchers that can pay for a wide range of goods and services for unemployed workers or to pay a

cash bonus for rapid reemployment.

Using a classic experiment implemented in eight sites, we examine the impact of three

different approaches to administering ITAs. These approaches vary in how they ensure that the

consumer, or “customer” in the language of WIA, has made an informed choice, whether the

customer’s choice can be rejected, and how the value of each ITA is determined. We examine

the impact of each approach on the amount of counseling customers receive about their choices,

the likelihood the customers will use the training voucher, the price of selected training programs

and the cost of the ITA to the workforce agency, and the training program and provider selected.

The rest of this paper is organized into seven sections. Section A provides the policy

context for the use of vouchers in occupation training. Section B describes issues that arise in

the administration of ITAs. Section C describes the three tested approaches. The experimental

design and data collection are described in Section D. Section E describes the characteristics of

the customers in the sample. The main study findings are described in Section F. The paper

ends, in Section G, with a discussion of the implications of the findings and a description of

further research.

3

A. POLICY CONTEXT

The mandate to use vouchers under WIA reflects a trend toward promoting customer choice

and provider competition in government training programs that has been ongoing for several

decades. The first large-scale government-funded training program, established in 1962 under

the Manpower Development and Training Act (MDTA), contracted directly with service

providers for classroom training. The program directed trainees to one of these contracted

providers. Under the Comprehensive Employment and Training Act (CETA), which replaced

MDTA in 1973, local service providers competed to receive CETA funding disbursed by local

workforce agencies or “prime sponsors.” Under the Job Training Partnership Act (JTPA),

which superseded CETA in 1982, local workforce agencies still contracted with providers to

provide blocks of classroom training. However, local workforce agencies could also refer

individual customers to courses already offered by service providers if preferred by the customer

(Barnow 2000). And indeed many local workforce agencies chose to move away from the use of

contracted training and toward individual referrals (D’Amico et al. 2001). Some local workforce

agencies even began to use vouchers for training (Barnow and Trutko 1999).

One of the key objectives of WIA, which replaced JTPA in 1998, was to provide customers

more choice in employment and training services. WIA mandated that ITAs be given to nearly

all persons eligible for WIA training. This included persons funded under both the dislocated

worker program and those funded under the adult program (for persons 18 years of age or older

who are not dislocated workers). An ITA has a dollar value and can be used to purchase

training services up to that dollar value. Customers can purchase more expensive training by

supplementing the ITA with their own funds or funds from other sources. However, any balance

remaining in the ITA after the training program is paid for is kept by the workforce agency.

4

WIA placed two restrictions on how ITAs can be used. First, the customer must choose a

program that will provide training for an occupation “in demand” in the local area. However,

WIA left up to local discretion how occupations in demand were to be identified. Second, the

training program that the customer chooses must be on the eligible training provider (ETP) list.

Training providers need to apply for each of their training programs to be on the ETP list by

submitting information about their program and student outcomes to the local workforce

agencies. To promote informed choice, WIA also required states to make available to customers

a Consumer Report System which provides information on training programs including student

outcomes, program cost, program length, and certifications available on completion of the

program.

Another key objective of WIA was to provide flexibility for local workforce agencies to

design their own policies. Hence, local workforce agencies decide how to implement their ITA

program--the amount of each ITA, counseling requirements, the criteria under which a training

program is approved, and the criteria for whether an occupation is in demand.

B. DESIGNING VOUCHERS FOR TRAINING

The theoretical advantage of using vouchers for delivering training is that they promote

economic efficiency. Vouchers promote efficiency in two ways. First, informed customers

choose training programs that maximize their individual well-being and hence social welfare (as

defined by the government and its goals). Second, customer choice increases competition among

training providers, which may lead to better quality or less costly training programs.

1. Providing Information

Underlying the theoretical advantage of vouchers is the assumption that customers are

informed. However, the efficiency benefit of vouchers may not be realized if customers are

5

inadequately informed about their own abilities, the quality of the training provider, or the

expected wages and employment prospects in the occupation for which they are training

(Barnow 2000). Customers may be uninformed because it is too costly to become informed or

they may not have the ability to become informed (Steuerle 2000). To the extent that customers

are not informed, local workforce agencies may need to provide information to customers or

require that customers show they are informed.

Concerns about uninformed customers have led many workforce agencies to require

customers to receive counseling and conduct research on the selected occupation and training

program before they are issued a voucher. Studies of voucher programs under JTPA and WIA

found that customers are typically required to undertake an assessment of their skills and

abilities, to show that they have engaged in extensive research on training programs and the labor

market, and to have extensive discussions with the counselors before a voucher is issued

(Barnow and Trutko 1999; D’Amico and Salzman 2004). In some sites, customers are required

to submit a formal application in which they justify their choice of training program.

2. Constraining Choices

The efficiency benefit of vouchers may also not be realized if individual customers choosing

training programs that maximize their own well-being do not also maximize social welfare

(Barnow 2000). For example, it may be that the government places a higher emphasis on the

financial return to training than the customer. As public assistance removes some of the cost of

unemployment, customers may place a lower emphasis on the probability of job placement after

training than is socially desirable. Some customers may evaluate training programs based on

factors that the government views as less important, such as whether they have friends who will

also attend the program.

6

If individuals maximizing their own well-being do not maximize social welfare, it may be

optimal for workforce agencies to place more restrictions on customers’ choices. WIA already

places some constraints on customers’ choices: they can only choose providers that are on the

ETP list (and so meet some minimum quality standards) and choose occupations that are “in

demand.”

Local workforce agencies vary in how they determine whether an occupation is in demand

(D’Amico and Salzman 2004). Some draw on lists of occupations with job vacancies and

relatively high wages developed locally or by the state’s labor market research unit. Others just

rely on employment counselors to make the determination.

Local workforce agencies can place further constraints on customers’ choice of training. In

a study of training voucher programs under JTPA, Barnow and Trutko (1999) found that local

workforce agency staff felt that the authority to reject a customer’s training choice was important

to prevent customers making poor training choices and wasting resources. In practice, however,

few agencies constrained customers’ choices any further. D’Amico and Salzman (2004) found

that employment and training counselors in the workforce agencies in their study rarely rejected

a customer’s choice of training program.

3. Rationing Resources—The Voucher’s Value

Funds for training are limited and hence need to be rationed among customers. One

approach is to provide each customer the same value voucher. This approach has the advantage

of being equitable and simple. However, if the return to training differs by customer, this equity

and simplicity may be gained at the cost of a loss of social welfare. If the local workforce

agency can identify those customers who receive the largest return to training, it may be better

for the agency to provide a larger voucher to those customers who will receive a higher return on

their training.

7

In practice, most local workforce agencies provide an ITA with the same fixed value to all

its customers (D’Amico and Salzman 2000). Some local workforce agencies, such as the one in

Jacksonville, Florida, however, set the ITA value at three different levels, providing a higher ITA

for training for occupations with higher wages.

C. THE APPROACHES TESTED IN THE ITA EXPERIMENT

While some workforce agencies had experience with administering vouchers for training

after WIA was passed, the vast majority of workforce agencies did not. But, as described above,

they are faced with key decisions about how to administer ITAa—how much information to

provide, the constraints placed on customer choices, and how to set the value of the ITA. To

help workforce agencies make informed decisions about how to administer the ITAs, the U.S.

Department of Labor sponsored Mathematica Policy Research, Inc. to conduct the ITA

Experiment.

The ITA Experiment tested three approaches to administering ITAs (Table 1). The three

approaches we tested vary along the three key implementation dimensions: (1) the intensity of

mandatory counseling about the training choice; (2) whether the counselor can reject the

customer’s choice; and (3) whether the ITA amount is the same for all customers or is set by the

counselor depending on the expected return to training.

Table 1: Summary of Approaches

Approach 1:

Structured Choice

Approach 2: Guided Choice

Approach 3: Maximum Choice

Counseling Mandatory, Intensive

Mandatory, Less intensive

Voluntary

Counselors Can Reject Choices?

Yes No No

Award Amount Customized Fixed cap Fixed cap

8

The approaches tested reflect a spectrum of different approaches to administering ITAs. At

one end of the spectrum, we tested an approach in which counselors play a major role in the

choice of training program (the “structured choice” approach). At the other end of the spectrum,

we tested an approach with few constraints or requirements (the “maximum choice” approach).

The approach in the middle of the spectrum, the “guided choice approach,” approximates the

approach that many workforce agencies currently use. Under this approach, counselors have

some limited role in the training decisions.

1. Approach 1: Structured Choice

Approach 1 places considerable emphasis on the counselors’ role in guiding customers to

make wise investments in training. The assumption underlying this approach is that without

counseling customers would not always choose the training program with the highest expected

return to the government’s investment. Hence, the counselors provide extensive counseling and

require that customers show that they are informed. They can also reject a training choice that is

not expected to have a high economic return. The counselors also tailor the amount of the ITA to

each customer, giving a larger amount for training with higher returns.

The counseling activities required before an ITA is issued under Approach 1 are highly

structured. Customers are required to identify an occupation to train for from a list of high-wage,

high-demand occupations developed by the site. Customers are also required to show that they

have thoroughly researched the various training programs available for their chosen occupation.

For example, customers are required to complete a form for each of two or three training

programs with detailed information about the program, such as typical class size, beginning date

of the course, and the time of the classes. They are also required to list all the costs of the

training including books, supplies, and any necessary childcare. Counselors are required to

discuss the customers’ family budgets during the time they will be in training to ensure that they

9

have sufficient resources to complete training. Finally, counselors are required to walk through

with the customer a benefit-cost analysis of the customer’s training choice. This involves

considering all the costs of the training (even those incurred by the government), including the

opportunity cost of the customers’ time, and comparing these costs with the expected benefits

from increased earnings from receiving the training.

Approach 1 is the only approach in which the counselor has the ability to reject a training

choice made by the customer. Potential reasons for rejecting a choice includes that the expected

benefits from the training do not exceed its costs, the net benefits of another training program are

higher than the one chosen by the customer, or the counselor thinks that the customer has

insufficient financial resources to live on while participating in the training program. If the

counselor rejects a choice, the customer can make another choice, but the counselor is not

required to accept any choice.

Under Approach 1 the counselor tailors the value of the ITA to each customer. Counselors

are directed to provide an ITA of greater value to people choosing training with higher net

benefits and to provide an ITA of lower value or to deny an ITA to people choosing training with

lower net benefits. The goal was that the workforce agency should spend on average the same

amount on customers under Approach 1 as it does on customers under Approaches 2 and 3. In

response to requests from the sites, we placed a cap on the amount of the ITA a counselor could

award under Approach 1. However, the cap was set high enough that in most cases it was not

expected to be binding. The cap on the ITA award was set at $8,000 in six sites, $7,000 in one

site, and $6,000 in one site.

2. Approach 2: Guided Choice

10

Approach 2 approximates the approach currently used by many workforce agencies. Like

Approach 1, it assumes that without counseling some customers would not be sufficiently

informed to make good decisions. Hence, like customers under Approach 1, Approach 2

customers are required to conduct some counseling activities. The counseling requirements are

similar to those under Approach 1, except that the customers do not need to consider the benefits

and costs of the training (although the counselors would discuss these if requested to) nor do they

need to choose an occupation from the list of high-wage/high-demand occupations.

Unlike Approach 1, however, this approach assumes that well-informed customers will

make decisions that are in line with maximizing social welfare. Hence, under Approach 2,

counselors cannot reject the customer’s choice as long as the program is on the ETP list.

Following the usual practice of workforce agencies, each Approach 2 customer receives an

ITA of the same value. The value of the ITA was set by each site at a level that was sufficient to

pay for many occupation training programs in the area, but low enough that they did not expect

to run out of training funds before the end of the year. The ITA value was $3,000 in five sites,

$4,000 in two sites, and $5,000 in one site.

3. Approach 3: Maximum Choice

Approach 3 is the least restrictive voucher approach. This approach assumes that customers

will voluntarily acquire the information they need to make and informed training decision and by

maximizing their own well-being, they also maximize social welfare. Hence, customers under

Approach 3 are not required to participate in any particular counseling activities, but can

voluntarily choose to participate in any counseling activities available to customers under

Approaches 1 and 2. As under Approach 2, counselors cannot reject the customer’s choice as

long as the program is on the ETP list and each customer is provided an ITA of the same value.

11

The ITA value was set at the same value for Approaches 2 and 3 customers. Hence, the only

difference between Approaches 2 and 3 is the mandatory counseling required under Approach 2.

D. EVALUATION DESIGN

The three ITA approaches were implemented side by side in each of eight study sites: (1)

Phoenix, Arizona; (2) Maricopa County, Arizona; (3) Bridgeport, Connecticut; (4) Jacksonville,

Florida; (5) Atlanta, Georgia; (6) Northeast Region, Georgia; (7) North Cook County, Illinois;

and (8) Charlotte, North Carolina. These study sites were chosen purposively by the U.S.

Department of Labor from among the workforce agencies that applied to participate in the study.

The enrollment period for the study differed slightly by site. It began on a rolling basis

between December 2001 (in North Cook County) and August 2002 (in Bridgeport) and

continued for about 18 months in each site. The first month of enrollment was considered a

pilot; data from that time period were not included in the analysis.

All customers who were found eligible for training during the enrollment period were

randomly assigned to one of the three approaches. They were assigned with a probability of one-

third to each approach. Before being randomly assigned, customers gave their written consent

for participation in the study and completed a baseline information form that requested

demographic and socio-economic information. In total, about 8,000 people were randomly

assigned to the three ITA approaches.

After random assignment, all customers were sent a letter notifying them of their assigned

approach. The letter provided a brief description of their assigned approach, including the value

of the ITA under Approaches 2 and 3 (but not the cap on the value of an ITA under Approach

1). Customers were then required to attend an approach-specific orientation, which provided

details about the ITA approach to which they were assigned. The site assigned the study

12

participant to a counselor in the usual way and counselors were notified of the customer’s

assignment.

The use of random assignment ensures that customers assigned to each of the three

approaches will have similar baseline characteristics on average. As a result, any observed

differences in participant outcomes across the approaches can be directly attributed to differences

in the ITA approaches with a known degree of statistical precision. Hence, most of the findings

in this paper involve comparing average outcomes across the three approaches.

All counselors in the study worked with customers under all three approaches. Hence, none

of the differences in outcomes across approaches can be attributed to differences in the

counselors across approaches. Our monitoring of counseling suggested that counselors were

successful in administrating a different approach to different customers. One potential limitation

of testing all three approaches in each site, however, is that we are not able to test whether the

three approaches differ in their effects on the training providers. It is possible that the way the

vouchers are implemented could affect the pricing and quality decisions of the training providers.

A computerized information system was designed specifically for the study. This study

information system was used (by both site staff and researchers) to track study participants,

including their participation in counseling, whether they received training, the training program

they chose, and the value of the ITA. Researchers also conducted three rounds of in-depth site

visits to each site (in 2002, 2003, and 2004).

E. CHARACTERISTICS OF THE STUDY SAMPLE

As expected, the sample of nearly 8,000 customers was evenly distributed across the three

ITA approaches. Table 2 shows the characteristics of these customers using data collected on the

baseline information form just prior to random assignment. For the most part, the random

assignment process yielded statistically equivalent groups of participants under each approach.

13

There was one exception. Relative to Approach 2 customers, Approach 1 customers were

slightly more likely to be employed at the time of enrollment in the study. These customers are

more likely to be funded under the adult WIA program rather than the dislocated worker

program. This difference does not affect the overall findings of the paper as we found the

impacts similar for customers funded by the adult and dislocated worker programs.

14

Table 2. Characteristics of ITA Study Participants, by Approach (Percentages Unless Otherwise Noted)

Characteristic All Approach 1 Approach 2 Approach 3 Gender

Female 54 54 53 53 Race

One race White 48 48 48 48 Black or African American 39 38 39 38 American Indian or Alaskan Native 1 1 1 1 Asian or Pacific Islander 6 6 6 6 Other 5 4 5 6

Two or more races 1 1 1 1 Hispanic or Latino 10 10 10 11

Average Age at Enrollment (Years) 40 41 41 40 Years of Regular Schooling

Less than 12 10 9 10 10 12 38 38 37 38 13 to 15 27 27 28 27 16 16 17 16 17 More than 16 9 9 10 9

Employment Status at Enrollment

Unemployed 91 90 92 90 Employed 9 10** 8 10 Had never worked 0 0 0 0

Years Worked at Current/Most Recent Job

Less than 1 year 34 34 34 34 1 to 5 years 45 44 44 46 6 to 9 years 8 8 9 8 10 or more years 13 14 13 13 Average 4 4 4 4

Funding Source Dislocated worker 69 67*** 71 69 Adult 31 33*** 29 31

Received Public Assistance at Enrollment 20 20 21 20

Sample Size 7,922 2,646 2,649 2,627 Source: Baseline information form. */**/*** = Difference relative to Approach 2 is statistically significant at the .10/.05/.01 confidence level.

15

The customers in the ITA study are quite diverse in their characteristics (Table 2). While

nearly all study participants had more than 12 years of education, 10 percent had fewer than 12

years of education, and 9 percent had more than 16 years of education. While their average age

was 40, some were as young as 18 and others as old as 81. Just over half the customers were

female; just under half were white. Most customers were unemployed when they were randomly

assigned; only about nine percent were employed. Most customers had less than five years of

experience in their current or most recent job. About two-thirds of the sample members were

funded through the dislocated worker program.

The study sample is not evenly distributed across sites (Table 3). One site, North Cook

County, accounts for 23 percent of the study sample, while Northeast Region, Georgia accounts

for only 2 percent of the sample. These differences reflect the differences in enrollment periods

(the enrollment period was about four months longer in North Cook County), differences in the

size of the populations served, and differences in the funds allocated to training in each site. The

distribution across sites makes little difference to our findings, as the estimated impacts of each

approach are remarkably similar across sites.

Table 3. Participation in the ITA Experiment (Percentages)

Overall Approach 1 Approach 2 Approach 3 Total Enrollment 100.0 33.4 33.4 33.2 Phoenix, AZ 8.2 33.1 33.9 33.0 Maricopa County, AZ 8.5 33.3 33.0 32.7 Bridgeport, CT 13.0 33.3 33.4 33.3 Jacksonville, FL 9.8 33.8 33.4 32.9 Atlanta, GA 17.8 33.6 33.3 33.1 Northeast Region, GA 2.2 33.3 31.8 31.9 North Cook County, IL 22.8 33.4 32.7 33.3 Charlotte, NC 17.7 33.3 33.4 33.4 Sample Sizes 7,922 2,646 2,649 2,627 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04).

16

F. FINDINGS

The impact of mandatory counseling and the ability of the counselors to reject customers’

choices depends on the customers’ prior knowledge of occupations and training programs, the

time and burden cost to the customers of counseling, the knowledge and expertise of the

counselors, and how additional information changes the customers’ decisions. The impacts of

tailoring the value of the ITA to the needs of the customers depends on the extent to which the

value of the ITA is a binding constraint to the choice of training program and the ability of the

counselors to tailor the ITA appropriately.

This paper examines the impacts of the administration of ITAs on four outcomes: (1)

receipt of counseling, (2) receipt of an ITA, (3) the price of training programs chosen by ITA

recipients and the expenditures on ITAs, and (4) the customers’ selections of training programs

and providers. Before describing our estimated impacts, however, we describe how the planned

approaches were actually implemented by the sites.

1. Implementation of the Approaches by the Sites

Our monitoring of counseling activities and discussions with workforce agency staff

suggests that, in general, the sites implemented Approaches 2 and 3 as planned. Discussions with

counselors in the study sites revealed that they were most comfortable with Approach 2--the

approach most similar to the approach used by the sites prior to the study. Counselors were less

comfortable with Approach 3. They were concerned that without counseling, customers would

make poor decisions. Despite this concern, counselors implemented Approach 3 as planned and

did not require customers in Approach 3 to participate in counseling.

Counselors did not, however, implement Approach 1 as planned. While they enforced the

counseling requirement and tailored the ITA amount to each customer, they found it very

difficult to steer customers to high-return training, and rarely, if ever, denied training to anyone.

17

They also found it difficult to control the expenditures on ITAs. As shown later, expenditures on

ITA under Approach 1 were much higher than under Approaches 2 and 3. Instead of taking the

more directive approach required under Approach 1, counselors tended to defer to the customer.

Many customers already had chosen a training program when they were found eligible for

training and counselors rarely attempted to change the customer’s choice.

Counselors gave several reasons for the lack of fidelity to Approach 1. Many counselors felt

that counseling should be a collaborative process and it was important to respect customers’

wishes. They felt that if they steered customers to training programs that were not their first

choice, the customers would be less likely to complete the training. Many felt the information

that was available to calculate the net benefits of a training program was too inaccurate to use as

a basis for decisions. And for some specialized occupations, such as information technology,

counselors felt unqualified to prescribe a specific training strategy.

This deviation of actual from planned implementation of Approach 1 implies that, in

practice, the three approaches differed mainly along only two dimensions: the amount of

required counseling and whether the ITA cap was fixed or customized.

2. Receipt of Counseling

One of the key ways in which the approaches differed was the extent of mandatory

counseling. Counseling was mandatory under Approaches 1 and 2, with the counseling

requirements being more extensive under Approach 1. Counseling was voluntary under

Approach 3.

While this paper discusses counseling received after random assignment, most of the study

sites also required counseling prior to determining eligibility for training. As this occurred prior

to random assignment, the amount of counseling received by customers would on average be the

same under each approach.

18

The counseling requirements under Approaches 1 and 2 seem to discourage customers from

pursuing ITA-funded training. To receive an ITA, all customers had to attend a mandatory

approach-specific orientation after random assignment. The proportion of invited customers who

attended this orientation varied by approach (Table 4), even though all that had differed between

approaches at this point was the content of the letters notifying the customers of their

assignment. Approach 3 customers were significantly more likely to attend the orientation than

Approach 2 customers (74 percent compared with 67 percent). Although the overall rate at

which study participants attended an orientation varied considerably across sites, being assigned

to Approach 3 increased the probability of the participant attending an orientation in every site

and the impact was statistically significant in five sites (Table 4). This suggests that just the

knowledge of the counseling requirements discouraged customers from pursuing training.

On the other hand, there was no significant difference between the likelihood of customers

in Approaches 1 and 2 attending an orientation overall. The proportion of Approach 1

participants who attended to an orientation was comparable to the proportion of Approach 2

participants who attended (68 percent compared with 67 percent). Approach 1 customers were

slightly more likely to attend an orientation in three sites, although the difference was significant

at a 5 percent confidence level only in Maricopa County.

19

Table 4. Participation in ITA Counseling (Percentages Unless Otherwise Noted)

Overall Approach 1 Approach 2 Approach 3 Attended Approach-Specific ITA Orientation Overall 69.4 68.2 66.5 73.6*** Phoenix, AZ 80.7 76.6 76.3 89.2*** Maricopa County, AZ 69.1 71.0** 60.7 75.9*** Bridgeport, CT 89.1 78.5 86.7 93.0*** Jacksonville, FL 80.4 75.7* 81.9 83.6 Atlanta, GA 78.9 77.0 78.0 81.8 Northeast Region, GA 69.6 68.4 67.9 72.4 North Cook County, IL 62.2 61.1* 59.4 66.3** Charlotte, NC 43.7 45.0* 38.9 47.2** Attended Counseling Beyond the ITA Orientation 42.5 65.2*** 58.2 3.8***

Average Number of Counseling Sessions

All ITA study participants 1.3 1.7*** 1.5 0.8*** Participants who attended an ITA

orientation 1.9 2.5*** 2.2 1.1***

Sample Size 7,922 2,646 2,649 2,627 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). Notes: Percentages may not sum to 100 due to rounding. */**/*** = Difference relative to Approach 2 is statistically significant at the .10/.05/.01 confidence level.

After the orientation, any further counseling for Approach 3 customers was voluntary and

Approach 3 customers rarely requested it. Less than 4 percent of all customers assigned to

Approach 3 participated in any counseling beyond orientation, compared with more than half of

customers under Approaches 1 and 2 (Table 4). Even when Approach 3 customers did request

further counseling after the orientation, they attended just over one additional counseling session

on average (Table 4). This result was found consistently across our sites.

After attending the ITA orientation, Approach 1 customers were significantly more likely to

attend some counseling than Approach 2 customers even though some counseling was

mandatory for customers under both Approaches 1 and 2. Of all customers randomly assigned,

65 percent of Approach 1 customers attended some counseling beyond the ITA orientation

20

compared with 58 percent of Approach 2 customers (Table 4). This suggests that information

provided at the orientation was more likely to encourage Approach 1 customers than Approach 2

customers to continue the process toward receiving an ITA. While the letter to Approach 1

customers did not tell them they could receive a higher ITA than under the other approaches,

anecdotal evidence suggests that many customers were aware of this difference. We suspect that

the perception that they could receive a higher ITA offset the burden of any additional

counseling under Approach 1.

Approach 1 customers were required to complete more counseling to receive an ITA, and

they in fact did. On average, Approach 1 customers attended 1.7 counseling sessions compared

with 1.5 for Approach 2 customers (Table 4). Those Approach 1 customers who attended the

ITA orientation participated in an average of 2.5 counseling sessions; in comparison Approach 2

customers who attended an ITA orientation participated in an average of 2.2 counseling sessions.

3. Receipt of ITAs

Although all customers found eligible for training are eligible to receive an ITA, some may

decide not to receive one. This may be because they decide after being found eligible for

training that they do not want to pursue training, perhaps because they find a job. Alternatively,

they may decide to pursue training but finance it from their own funds or other sources.

The likelihood of a customer pursuing ITA-funded training depends on its expected costs

and benefits, both of which may be affected by the approach to administering ITAs. The costs of

ITA-funded training include the burden of completing the activities necessary to receive an ITA.

It also includes earnings forgone when training or waiting to receive an ITA. The benefits of

ITA-funded training depend on the dollar value of the ITA received and the restrictions placed

on how the ITA can be used.

21

Overall, only about two-thirds of all customers who were randomly assigned received an

ITA (Table 5). This proportion varied by site. It was particularly low in two sites—Atlanta,

Georgia and Charlotte, North Carolina. In Atlanta, the relatively low ITA training rates may

reflect the availability of state-funded alternatives to ITA. We have no explanation for the low

rate in Charlotte. Most of the attrition occurred prior to the mandatory orientation—87 percent

of those who attended the orientation received an ITA.

Approach 3 customers were significantly more likely to receive an ITA than customers

under Approaches 1 and 2 (Table 5). About 65 percent of all Approach 3 customers who were

randomly assigned received an ITA compared with 59 percent of Approach 1 customers and 58

percent of Approach 2 customers. Compared to Approach 2 customers, Approach 3 customers

were more likely to receive an ITA in every site, and the difference was statistically significant

in four of the eight sites.

The difference between the rate of ITA receipt under Approaches 2 and 3 was mainly due to

differential attrition between the receipt of the letter notifying the customer of his/her assignment

and the orientation. Of those customers who attended an orientation, the proportion of Approach

3 customers who received an ITA (88.5 percent) was not significantly different than the

proportion of Approach 2 customers who received an ITA (86.6 percent). This suggests that it is

the expected, rather than actual, burden of the counseling that explains the lower rate of ITA

receipt under Approach 2.

22

Table 5. Receipt of ITAs (Percentages)

Overall Approach 1 Approach 2 Approach 3 All Study Sites 60.6 59.0 57.6 65.1***

Phoenix, AZ 67.5 62.1 61.6 78.9*** Maricopa County, AZ 65.8 67.0** 56.3 74.5*** Bridgeport, CT 73.4 68.3 71.9 79.9** Jacksonville, FL 77.2 72.2 77.3 82.0 Atlanta, GA 39.8 37.4 38.6 43.3 Northeast Region, GA 69.6 68.4 67.9 72.4 North Cook County, IL 70.1 70.9 68.7 70.8 Charlotte, NC 43.7 44.8* 38.7 47.6***

All Study Sites, Among those who Attended an Orientation 87.3 86.5 86.6 88.5 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). */**/*** = Difference relative to Approach 2 is statistically significant at the .10/.05/.01 confidence level

Overall, Approach 1 and 2 customers had comparable rates of receipt of ITAs. This does,

however, mask some difference across sites. For example, the proportion of customers who

received an ITA was significantly higher under Approach 1 in Maricopa County, Arizona and

Charlotte, North Carolina.

The similar overall rate of ITA receipt between Approach 1 and 2 customers suggests either

that the differences between the two approaches had little effect on ITA receipt or that opposing

factors were offsetting. We expect that the possibility of a higher value ITA, to the extent that

the customers were aware of it, may have encouraged ITA receipt. However, the additional

counseling requirements may have discouraged ITA receipt. After random assignment, it took

ITA recipients under Approach 1 about one week longer than Approach 2 ITA recipients to

receive an ITA (Table 6). The additional time involved also increases the likelihood that the

customers will receive a job offer and decide against pursuing training. As the ability of the

counselors to reject the customer’s choice was rarely exercised, we doubt that this ability had

much impact on ITA receipt.

23

Table 6. Average Number of Weeks Between Random Assignment and ITA Receipt

OVERALL APPROACH

1 APPROACH

2 APPROACH

3 7.0 8.1*** 7.0 6.0

Source: Study tracking system for the ITA experiment (data extract as of 5/17/04).

*/**/*** = Difference relative to Approach 2 is statistically significant at the .10/.05/.01 confidence level.

4. Price of Selected Training Programs and ITA Expenditures

We examined two approaches to determining the value of each ITA. The first approach was

to tailor the amount to each customer—providing a larger amount to customers who pursue

training with a higher expected economic return and a lower amount to other customers

(Approach 1). The second approach was to provide the same fixed ITA to all customers

(Approaches 2 and 3).

We found that customers assigned to Approach 1 chose significantly higher-priced training

programs on average than those assigned to Approaches 2 and 3 (Table 7). Approach 1

customers chose training programs that were on average $4,800—about $1,600, or over 50

percent, more expensive than those chosen by Approaches 2 and 3 customers. On average,

Approach 1 customers chose more expensive programs than the other customers in all sites and

the difference was statistically significant in six of the eight sites.

Although the fixed value of the ITA was the same under Approaches 2 and 3, the required

counseling under Approach 2 may have affected the program chosen. However, we found that

the receipt of counseling under Approach 2 had little impact on the price of the program chosen.

Within each site, Approach 2 and 3 customers chose programs of similar prices, on average

$3,116 and $3,133 respectively.

24

Table 7. Average Price of Selected Training Programs (Dollars)

Study Site

Approach 1 Cap

Approach 1 Price

Approaches 2 and 3

Fixed ITA Value

Approach 2 Price

Approach 3 Price

Phoenix, AZ 8,000 4,791*** 3,000 2,489 2,518 Maricopa County, AZ 8,000 5,659*** 3,000 3,204 3,089 Bridgeport, CT 7,000 3,810*** 3,000 2,784 3,034 Jacksonville, FL 6,000 4,834*** 3,000 3,388 3,300 Atlanta, GA 8,000 4,429 5,000 4,210 4,220 Northeast Region, GA 8,000 3,931 4,000 3,556 3,547 North Cook County, IL 8,000 5,210*** 3,000 2,536 2,578 Charlotte, NC 8,000 5,055*** 4,000 3,245 3,229

Overall 7,626 4,764*** 3,430 3,116 3,133 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04).

*/**/*** = Difference relative to Approach 2 is statistically significant at the .10/.05/.01 confidence

As customers can contribute other sources of funding to pay for the training program, the

workforce agencies’ expenditures on ITAs may be less than the price of the programs. On

average, for those who received an ITA, ITA expenditures were about $1,800 to $1,900 (66

percent) higher for customers under Approach 1 than under Approaches 2 and 3 (Table 8). The

average expenditure per ITA recipient under Approach 1 was significantly higher in all sites

except Northeast Region, Georgia (not shown). In contrast, there was no statistically significant

difference in average expenditures per ITA recipient between Approaches 2 and 3 overall, or in

any site.

25

Table 8. Average Expenditures of Workforce Agencies

Approach 1 Approach 2 Approach 3 Average Price of Selected Programs $4,764*** $3,116 $3,133 Average Expenditures per ITA Recipient $4,731*** $2,849 $2,857 Average Expenditure per ITA Recipient as a Percent of Average Price 99.3% 91.4% 91.2% Percentage of Customers who Receive ITAs 59.0% 57.6% 65.1%*** Average Expenditures per Customer $2,791*** $1,641 $1,853*** Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). Notes: Estimates do not include WIA administrative costs or the costs of ITA-related counseling.

*/**/*** = Difference relative to Approach 2 is statistically significant at the .10/.05/.01 confidence level.

Customers assigned to Approaches 2 or 3 were more likely to use non-ITA funds to help pay

for training than were customers assigned to Approach 1. In total, customers financed about 9

percent of the cost of their training under Approaches 2 and 3 from some other sources (Table 8).

In contrast, under Approach 1, the cap on the ITA value was set so high that few customers

chose a training program that cost more than the cap. Customers financed less than 1 percent of

the cost of the training programs under Approach 1 from other sources (Table 8). This suggests

that the more generous ITA funding available under Approach 1 tended to discourage customers

from finding other sources of training financing.

Our findings in Table 8 also suggest that the ITA approach has a substantial effect on the

workforce agencies’ expenditures on ITAs averaged over all their customers (including those

who do not receive an ITA). The average training cost per customer is 70 percent higher under

Approach 1 than under Approach 2 ($2,791 under Approach 1 compared with $1,641 under

Approach 2). This higher cost is mainly because Approach 1 customers choose more expensive

programs and in most cases this additional expense was entirely paid for by an ITA.

26

The average training cost per customer under Approach 3 is also significantly higher than

under Approach 2 ($1,853 under Approach 3 compared with $1,641 under Approach 2). This is

because, although the ITA expenditures per ITA recipient is similar under Approaches 2 and 3,

Approach 3 customers are significantly more likely to receive an ITA than Approach 2

customers.

While the expenditures on ITAs are a large part of the cost of the training to the government,

the provision of counseling is also costly. As the costs of counseling are likely to be slightly

higher under Approach 1 than under Approach 2, if counseling costs are taken into account, the

difference in cost between Approaches 1 and 2 would increase. However, as the counseling

costs are relatively small for Approach 3, once the counseling costs are included, the difference

in total costs between Approaches 2 and 3 would decrease and perhaps disappear or become

negative.

5. Training Program Selections

Ultimately, workforce agencies are concerned about the training choices made by customers

and the employment they obtain after training. The way ITAs are administered could affect the

choices made by the customers. Whether counseling is mandatory, whether the counselor can

require the occupation to be on the high-wage/high demand occupation list and can reject the

customers choices, and the value of the ITA may all affect the selection of training programs.

The only difference between Approaches 2 and 3 is the counseling requirement. If

counseling provided customers more information or made them think more about their choices,

we would expect to see some differences in the training programs selected by Approach 2 and 3

ITA recipients.

The occupations chosen by ITA recipients when counseling was mandatory were

remarkably similar to the occupations chosen by ITA recipients when counseling was voluntary

27

and few received counseling. Table 9 shows the top 20 occupations corresponding to the training

programs chosen by the ITA recipients in our sample.1 Approach 3 ITA recipients were slightly

less likely than Approach 2 ITA recipients to choose accounting or bookkeeping as an

occupation, but the difference between the proportion of ITA recipients in Approach 2 and 3 who

chose this occupation was only statistically significant at the 10 percent level.

1 Table 9 shows only the occupations chosen by those sample members who received ITAs. As the approach

affected the likelihood of receiving an ITA, differences between the occupations chosen by the recipients under each approach may be due to underlying differences in the characteristics of ITA recipients rather than the effect of counseling on occupation choice. We found few differences in the observed characteristics of ITA recipients under each approach.

28

Table 9. Occupation Choices by Recipients of ITAs (Percentages)

Occupation Overall Approach 1 Approach 2 Approach 3

Top 20 Occupation Choices:

1. Computer networking and information technology 14.8 16.3 14.2 13.9 2. Commercial Driver’s License 13.3 13.4 13.5 13.1 3. Medical assistant/secretary 5.3 6.1 4.9 5.1 4. Administrative assistant 5.3 4.0* 5.4 6.4 5. Certified Nurse Aide 4.9 3.3** 5.2 6.0 6. Medical coding/billing 4.6 4.5 4.2 5.1 7. Computer programmer 4.0 3.9 3.8 4.4 8. Licensed practical nurse or registered nurse 3.8 4.3 3.6 3.5 9. Database administration 3.5 3.9 3.5 3.2 10. Web design 3.3 3.2 3.5 3.2 11. Accounting/

bookkeeping 3.0 2.4** 3.8 2.8* 12. Health technician/technologist 2.4 3.0** 1.7 2.5 13. Heating, ventilation, and air conditioning 2.1 1.7 2.5 2.0 14. Dental assistant 2.0 2.4*** 2.0 1.7 15. Paralegal 1.7 3.0 1.3 0.9 16. Teaching 1.4 1.8 1.3 1.2 17. Business/project administration 1.4 1.2 1.5 1.4 18. Computer installation, repair, and support 1.3 1.4 1.1 1.4 19. Computer-aided design and drafting 1.2 1.6 1.1 0.9 20. Data entry/processing 1.1 1.0 1.3 1.0

ITA Vouchers with Occupation Information 4,759 1,525 1,505 1,729 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). */**/*** = Difference relative to Approach 2 is statistically significant at the .10/.05/.01 confidence level.

We were surprised to find, however, that mandatory counseling did seem to affect the type

of training provider selected (Table 10). Approach 3 customers were significantly more likely to

choose training at a community or technical college and somewhat less likely to choose training

at private schools or four-year universities than Approach 1 or 2 customers.

29

Table 10. Training Providers and Average Duration of Training Program Selected by ITA Recipients

Overall

Approach 1

Approach 2

Approach 3

Type of Training Provider

Private, For-Profit (Proprietary) School 62.7% 63.5% 63.4% 61.3% Community or Public Technical College 29.0 26.2 28.0 32.2*** Nonprofit Organizations 1.9 2.0 2.2 1.5 State or Private (Four Year) University 6.1 7.6 6.3 4.5** Other 0.4 0.6* 0.1 0.6*

Average Duration of Training Program (Months) 7.5 8.2*** 7.0 7.3 Sample Size 4,756 1,517 1,520 1,719 Source: Study tracking system for the ITA experiment (data extract as of 5/17/04). Notes: Percentages may not sum to 100 due to rounding. Table excludes data on Charlotte, North Carolina, due to missing

training provider information.

Mandatory counseling did not, however, seem to affect the length of the program chosen.

Approach 2 and 3 customers chose programs of comparable lengths (Table 10). On average, the

programs chosen by Approach 2 and 3 ITA recipients were about 7 months long.

Approaches 1 and 2 differed in the intensity of counseling, the value of the ITA, and

whether the counselors could reject a customer’s choice. As counselors rarely rejected a

customer’s request, we suspect that their ability to do so had little impact on the selection of

training programs by Approach 1 customers. Approach 1 and 2 recipients of ITAs chose similar

types of training providers (Table 10). Hence, the additional counseling and the higher value

ITA did not seem to affect the choice of the type of provider.

Approach 1 customers signed up for longer programs on average—probably because they

were able to purchase more expensive programs (Table 10). Approach 1 customers chose

programs with an average duration of 8.2 months, about one month longer than the programs

chosen by Approach 2 and 3 customers. Interviews with program staff suggest that the

30

customers under Approach 1 were able to choose training programs that included additional

certifications than the training programs chosen by customers under the other approaches.

Anecdotal evidence suggests that the training providers learned of the higher value ITA under

Approach 1 and created these longer, more expensive, training programs for Approach 1

customers by bundling together other training programs.

G. DISCUSSION

This paper examined three different approaches to administering vouchers for training. The

approaches varied in the counseling requirements, restrictions placed on customers’ choices, and

the way limited training funds were allocated among customers. The lessons learned from this

study about the impacts of different approaches to administering ITAs are likely to be applicable

to other voucher programs.

The findings from this study suggest that it is difficult to implement a training voucher

approach that requires counselors to be highly directive in guiding customers’ training choices.

We found across all the sites that counselors tended to defer to customers preferences and rarely,

if ever, denied a training choice to a customer. Implementing Approach 1 successfully would

require a move away from the collaborative counseling approach used by most workforce agency

counselors. It would also require additional training and support of employment counselors to

more effectively steer the training choices made by customers.

We found that when they could, many customers chose to forgo additional counseling prior

to making their training choice. Interviews with program staff suggested that many customers

knew what program they wanted before they were found eligible for training. Some had already

approached a training provider and had been referred by the provider to the workforce agency for

funding.

31

Requirements to participate in counseling led to some people dropping out of the process

and not receiving an ITA. Contrary to the fears of some workforce agencies, customers who

chose not to receive counseling made similar training choices to those who were required to

receive counseling. The only difference we could detect between choices made by those who

were required to receive counseling and those who were not was that on average customers who

were required to receive counseling were more likely to choose private schools or four-year

colleges or universities for their training.

This paper examined two different ways of setting the amount of the ITA voucher—fixing

the cap on the award for everyone at the same value and asking counselors to customize the

funds available to each customer based on the expected return to their chosen training. We

found that sites had difficulty implementing a system in which the counselors customized the

amount based on the expected return and would deny a training program if it was unlikely to lead

to high earnings relative to its cost. Counselors were disinclined to deny customers a higher ITA

when customers wanted to pursue expensive training options. Hence, when the amount of the

ITA was customized, on average, counselors granted customers substantially more training

funds. This led to customers choosing more expensive programs, which tended to be longer. It

also meant that customers needed to finance less of the cost of the training program from non-

ITA funds.

We conclude that when counseling is voluntary, more people receive training and on

average they choose similar training programs than customers who are required to participate in

extensive counseling. Moreover, costs per ITA recipient are no higher than when counseling is

mandatory. However, it would be premature to recommend that workforce agencies drop the

counseling requirement associated with ITAs before addressing some remaining questions. Are

customers who receive counseling more likely to complete training? (It might be that the

32

customers who choose not to participate in training are those who are least likely to complete the

training.) And, how does the approach to disbursing training funds affect the customers’

subsequent employment and earnings? In future research, we plan to address these additional

questions using data from a survey of study participants 15 months after random assignment as

well as administrative data on earnings collected by the Unemployment Insurance

administration.

33

REFERENCES

Barnow, Burt S., and John W. Trutko. “Experiences with Training Vouchers Under the Job Training Partnership Act and Implications for Individual Training Accounts Under the Workforce Investment Act.” Washington, DC: U.S. Department of Labor, Employment and Training Administration, 1999.

Barnow, Burt S. “Vouchers for Federal Targeted Training Programs.” In Vouchers and the Provision of Public Services, C. Eugene Steuerle, Van Doorn Ooms, George Paterson, Robert D. Reischauer, editors. Washington, DC: Urban Institute Press, 2000.

Bureau of Labor Statistics. Occupational Outlook Handbook, http://www.bls.gov/oco, December 2004.

D’Amico, Ronald, and Jeffrey Salzman. “An Evaluation of the Individual Training Account/Eligible Training Provider Demonstration: Final Report.” Oakland, CA: Social Policy Research Associates, December 2004.

D’Amico, Ronald, Deborah Kogan, Suzanne Kreutzer, Andrew Wiegand, Alberta Baker, Gardner Carrick, and Carole McCarthy. “Interim Report: A Report on Early State and Local Progress Towards WIA Implementation.” Washington, DC: U.S. Department of Labor, Employment and Training Administration, February 2001.

Posner, Paul, Robert Yetvin, Mark Schneiderman, Christopher Spiro, and Andrea Barnett. “A Survey of Voucher Use: Variations and Common Elements.” In Vouchers and the Provision of Public Services, C. Eugene Steuerle, Van Doorn Ooms, George Paterson, Robert D. Reischauer, editors. Washington, DC: Urban Institute Press, 2000.

Public Policy Associates. “Dislocated Worker Program Report: Findings from the Career Management Account Demonstration.” Washington, DC: U.S. Department of Labor, Employment and Training Administration, 1999.

Steuerle, C. Eugene. “Common Issues for Voucher Programs.” In Vouchers and the Provision of Public Services, C. Eugene Steuerle, Van Doorn Ooms, George Paterson, Robert D. Reischauer, editors. Washington, DC: Urban Institute Press, 2000.

U.S. General Accounting Office. “Millions Spent to Train Students for Oversupplied Occupations.” GAO/HEHS-97-104. Washington, DC: GAO, June 1997.

34