Design and implementation of the Impact evaluation of ISKUR training programs June 15, 2011

This note describes the design and implementation of the impact evaluation of ISKUR’s vocational training programs. The study is a joint effort by ISKUR and the World Bank aimed at identifying ways to improve vocational training programs in the context of the rapid expansion of these programs in Turkey. In particular the study is designed to answer the following questions: (1) What is the average impact of ISKUR training on the labor market (as measured by the likelihood and quality of employment)? (2) Which trainees benefit the most from training (in terms of gender, age, level of education and skills, work experience etc.)? (3) What are mechanisms/processes through which training affects labor market outcomes (e.g. improved skills, reduced search costs etc.) (4)What provider characteristics make training most effective?

The impact evaluation study started in the spring of 2010 and is expected to be completed in the spring of 2012. The first phase has been completed, including (i) the selection of provinces (23), training courses (130) and participants (5,700) for the evaluation; and (ii) the collection of baseline data from about 5,300 evaluation participants before the start of the training courses. Data was collected between September 2010 and January 2011 as training courses were rolled out. Data on training providers of the trainings selected for the evaluation is being collected in June 2011. The final evaluation of ISKUR vocational training programs will be mainly based on the follow up survey of evaluation participants in early 2012.

The evaluation relies on an experimental approach based on the excess demand for ISKUR’s vocational training programs. Training providers interview all eligible and interested candidates. Participation in the training programs is then randomly awarded among the best candidates into treatment (i.e. those receiving training) and control (those not receiving the training) groups. Provided the randomization is successful (i.e. treatment and control groups are statistically equivalent at the outset), the evaluation strategy is simply to compare the labor market outcomes of treatment and control groups.

1

The note describes in detail the experimental approach used and the most important steps in the implementation of the impact evaluation. Among others, it discusses the implementation of the experimental design, the data collection activities and the main changes implemented in the ISKUR monitoring and information system. The ultimate goal of this note is to provide hands-on information on how to conduct impact evaluations of similar programs in Turkey and elsewhere, particularly in middle income countries (MIC), as this is the first rigorous impact evaluation of a large-scale publicly provided training program in MIC, and many in MIC face a similar excess demand for vocational training. This note accompanies another note analyzing the profile, job-search behavior and expectations of ISKUR vocational trainees (based on the baseline survey).

The rest of this note is organized as follows. Section 2 describes the evaluation design. Section 3 describes the data collection needs, main counterparts involved and the different steps in the evaluation. In particular, it describes the sampling and the course selection. Section 4 presents a detailed description of the sample including, its geographical coverage, the power calculations and regional targets for the participants and the set of vocational courses. It also discusses when and how the interviews take place. Section 5 discusses how the treatment and control groups were selected and section 6 discusses some of the most important changes in the ISKUR’s Monitoring and Information System.

1. Evaluation Methodology: Identifying the Treatment and Control Groups

The traditional assessment of the effectiveness of the ISKUR trainings, which is based on whether the trainees are placed into jobs, is likely to lead to biased estimates. Most ISKUR provincial offices still use the placement rates among trainees as a good proxy for the effectiveness of the trainings provided. However, this is often difficult to implement due to the lack of data. ISKUR’s Management Information System (MIS) does not keep up to up-to-date information on all the trainees nor does it collect information on labor market outcomes after trainings. Furthermore, to isolate the effects of the trainings from the overall economic conditions in the labor market, a control group is needed to establish a counterfactual (i.e. what would have happened in the absence of training). It is important to carefully construct this control group. ISKUR trainees are likely not identical to the average unemployed in Turkey. They meet very specific skills and experience requirements that are usually set ex-ante by the

2

training providers. All trainees also need to go through an interview with the training providers and local ISKUR staff before being selected for training.1

This evaluation study is based on a sample of the best eligible applicants to the ISKUR trainings. The sample frame includes a wide set of provinces where there is a large excess demand for the trainings. In particular, the sampling frame, includes all the registered unemployed individuals who are eligible to take up training (according to ISKUR basic criteria such as being registered to ISKUR) and are interested in taking up the training, i.e., they have they applied for them. Latter, the training provider’s interview this group of eligible and interested individuals and select the set of best candidates.

The evaluation study is based on an experimental design, comparing the labor market outcomes of treatment and control individuals before and after the trainings. The participation into the trainings is randomly awarded among the sample of the best applicants. The effects of the trainings in improving the labor market outcomes are quantified by comparing the performance in the labor market, before and after the trainings take place, for the two groups: treatment and control. Treatment and control groups together account for the evaluation sample.2 The random assignment to treatment and control groups ensures that both groups are as similar as possible (both in socio-economic background and in motivation). This is also the fairest way possible to allocate a small number of slots among a larger list of applicants. To ensure that the effects of the trainings will be identified separately for youth and women, the randomization is stratified by these two characteristics.

2. Main “Steps” and Counterparts in the EvaluationThe evaluation involves a close collaboration among the evaluation team, World Bank Ankara office, ISKUR central and provincial offices, the survey firm, and to smaller extent, some municipalities. The field coordinator and researchers are part of the evaluation team. The baseline and follow up surveys are implemented by an external agency (private firm). The firm is responsible for hiring and training of enumerators. Some guidelines are provided to the agency

1 World Bank (2011) reports the results for the baseline survey where the ISKUR beneficiaries are contrasted against the sample of average unemployed and inactive individuals. Results show that the ISKUR applicants are more likely to be women, younger and more educated. They also have less labor market experience.

2 There are other trainees that take up the trainings but they are not part of the evaluation. 3

and it presents a close monitoring of data quality is ensured with the supervision of the field coordinator. The success of the evaluation lies in the strong coordination across all these counterparts.

Figure 1 summarizes the main steps in the implementation of the evaluation. These include: the design of the sample and the province and course selection; the evaluation pilot and field coordination; the main pre-data collection activities; the individual randomization, the baseline survey and the delivery of the training programs; and, finally, the data analysis and planning of the follow up activities. Figure also shows that, throughout the entire implementation process, there is a continuous improvement of the ISKUR’s monitoring and information system. The figure reviews selected activities in each of these steps.

4

Figure 1: Process of the Evaluation

Note: Some of the activities (for e.g. maintaining treatment and control groups) cover the entire duration of the evaluation and some are conducted simultaneously (for e.g. preparing baseline survey and improvement of ISKUR's IT System).

5

Pre-Data Collection Activities

Randomization, Baseline Survey & Delivery Training

Analysis and Follow up Activities

Evaluation Pilot and Coordination

Sample Design and Province/Course Selection

Analyze data;

Update contact information for the evaluation sample;

Finalize and implement the follow-up survey;

Analyze results;

Recommend permanent changes in the MIS to ensure high-quality data.

Conduct a pilot in Ankara;

Finalize the baseline survey and adjust the ISKUR MIS system;

Communication and training of local ISKUR staff and course providers on the evaluation protocols;

Public announcements for the final set of evaluation courses;

Finalize baseline survey and the provider’s questionnaire with counterparts;

Selection of the survey firm.

Conduct the interviews throughout the country;

Randomly select the treatment and control groups;

Schedule additional interviews when no oversubscription is attained;

Collect baseline data;

Delivery of the training programs.

Conduct power calculations; Define regional targets for the number of participants in the evaluation;

Select provinces included in the evaluation;

Work together with ISKUR regional offices on the main procedures;

Select the vocational courses included in the evaluation.

Improvement of ISKUR Monitoring and Information System◦ For a stronger dissemination of the training courses to the target population◦ For the development of an interface for the training providers (allowing applicants registration and update information)◦ For the development and implementation of a randomization routine◦ To publish periodic reports informing on the tracking of the treatment and control groups

Box 1: A Workshop on Impact Evaluation: Building capacity and support and working together on the design of the impact evaluation

3. The Sample: Provinces, Courses and Power Calculations3.1. The provinces

The evaluation sample covers 23 provinces, which together are representative of the labor market conditions all over Turkey. Figure 2 reports the final set of 23 provinces selected to be included in the sample. Prior to the start of the study, ISKUR identified the set of provinces with at least two oversubscribed courses during the period 2009/2010.3 Oversubscription in the training courses was defined as having at least twice the number of applicants than training slots. The sample was stratified by the unemployment rate. A sample of 20 provinces was randomly selected, of which 10 provinces had “high” unemployment rates and ten provinces had a “low” rate. 4 Selection is proportional to percentage of individuals trained in 2009. Three additional provinces (Antalya,

3 Table A2 in the annex lists the total of 39 provinces out of the 81 provinces in Turkey that met these criteria. It also shows the strata they belonged to.

4 The provinces were classified into high and low unemployment provinces depending on whether the unemployment rate was above or below 10%.

6

Before the beginning of the evaluation, the central ISKUR office and the evaluation team organized a large workshop to discuss the design and the major implementation arrangements. The workshop took place in April 2010 and gathered ISKUR staff from all the 23 participating provinces, reaching a total of 85 participants. The main objective of the workshop was to discuss the design of the evaluation and follow up with the main counterparts on the most important implementation arrangements. It also aimed to gather consensus and support in the most relevant, and possibly controversial, steps.

The first part of the workshop focused on the design of the evaluation and some of the technical details. Main objective and expected outputs were described, the evaluation team was presented and the structure of the impact evaluation design was also discussed. A wide range of issues was discussed, ranging from the selection of provinces, regional target numbers, the importance of achieving oversubscription and the randomization. The randomization process and selection of treatment and control groups were the focus of the main part of the discussion.

Gaziantep, Diyarbakir) were proposed by ISKUR because of their importance in representing varying labor market conditions across Turkey.

Figure 2: Provinces in the Evaluation Sample

3.2. Power Calculations and Regional Targets for the Participants

The power calculations determined the optimal sample size to be able to detect a difference in job placement rates across treatment and control group. The outcome indicator used in the power calculations was the employment rate. Calculations also took into account observed past attrition from the trainings (or the treatment). This was based on the analysis of previous drop-out rates from training courses. Possible attrition rates also considered the possible non-response rates to the face-to face survey.5 The sample size estimates accounted for an imperfect take-up of the training programs a power of 90% and an “effect size” of 6%-7%.6

5 The evaluation team is interested in determining the minimum sample size possible allowing the identification of the effects of training on job placement.

6 We consider job placement rates of 20% and 30%. These numbers are defined using and analyzing the Social Security data. The placement rate is low because only the formal sector jobs were considered.

7

But the available information in the MIS may not lead to realistic assumptions on the share number of applicants that may drop-out of the trainings. The regional ISKUR offices and course providers use the information from the MIS to reach out to the registered unemployed and invite them to attend the training programs. When a candidate rejects to participate in the trainings or drops-out of the trainings, some of the individuals in a waitlist are invited to participate.7 Usually, the MIS records are updated based on the paper records that are obtained after these slots are filled with the waitlisted candidates. Therefore, the “real” share of drop-outs is not easily retrieved from the ISKUR records.

And, indeed, the pilot study showed that the initial assumption on the drop-out rates of the trainees from the sample was heavily underestimated. The Ankara pilot and the analysis of retrospective data, led to more realistic assumptions for (1) the share of trainees dropping out of treatment (also known as the uptake of the treatment group) and for (2) the share of control individuals getting into the training programs (also known as the uptake of control group). More realistic assumptions for the program take up and for the contamination of the control until the end of the evaluation period led to revisions in sample size. Table 1 reports the optimal sample size under different assumptions for the effect size; take up of treatment and control, base job placement rate.

Table 1: Optimal Sample Size8

Effect size Treatment take-up

Control take-up

Base Job Placement

Rate

Optimal Sample size

6% 85% 15% 20% 53906% 85% 20% 20% 62687% 80% 15% 20% 46457% 80% 20% 20% 54687% 85% 15% 30% 49737% 80% 15% 30% 5745

For reasonable assumptions, the optimal sample size is close to 5,700 individuals; the geographical composition replicates the importance of each province in the national provision of the trainings. Considering a 80%-85% uptake of treatment and 15%-20% uptake of control, sample attrition of 20% 7 They may drop either because they change their minds and decide to take another course, find a job. Others still cannot be reached. 8 The table assumes a power of 90%, an Alpha of 5%, and attrition rate of 20%.

8

and a base job placement rate of 20%-30% the optimal sample size is approximately 5,700 individuals. 50% of the individuals are in the treatment and 50% in the control group. The sample is disaggregated in a representative way across the country so that the number of evaluation participants in each province depends on the share of vocational trainees in that province in 2009. There is a minimum number of trainees at province level of 100 to benefit from the economies of scale in the implementation and data collection. Table 2 reports the target shares of each province in the overall sample and the number of trainees in 2009. Figure 3 reports the actual number of individuals successfully interviewed at the baseline survey.

Table 2: Sample size by Provinces

No Province

Number of trainees in 2009

Targets for the evaluation participants

1 ANKARA 3,344 3082 ANTALYA 1,771 1633 BAYBURT 249 1004 DENIZLI 1,420 1315 DIYARBAKIR 450 1006 DUZCE 702 1007 ELAZIG 1,204 1118 ERZURUM 836 1009 ESKISEHIR 2,450 22610 GAZIANTEP 751 10011 HATAY 2,889 26612 ISPARTA 1,152 10613 ISTANBUL 19,208 1,77114 IZMIR 1,971 18215 KAYSERI 3,067 28316 KIRIKKALE 1,329 12317 KOCAELI 7,519 69318 MANISA 3,547 32719 MUS 1,938 17920 SAKARYA 1,208 11121 TEKIRDAG 1,709 15822 TRABZON 1,628 150

9

23 USAK 1,484 137TOTAL 61,826 5,925

Source: Author’s calculations.

Figure 3: Provincial Breakdown of the Evaluation Sample at Baseline

3.3. The selection of the vocational courses

The evaluation team worked closely with the regional ISKUR offices to determine the final set of courses to be included in the evaluation. The sample was finalized by the ISKUR provincial directorates in August 2010. The number of courses used in the evaluation sample is 130. The course capacities range from 12 to 100 trainees. However, most of the courses in our sample have a capacity between 20 and 25 trainees. Several criteria dictate the final selection of courses:

Type of vocation: Even though ISKUR provides a wide set of training programs, including general vocational training programs, job guaranteed

10

internships and entrepreneurship courses - the evaluation focuses only on general vocational training programs.9

Oversubscription: The evaluation focuses only on the training courses which are likely to be oversubscribed. This is the only way to guarantee the random selection into treatment and control among individuals in the evaluation sample. Therefore, only the most popular and more demanded courses are selected.

Course capacity: The evaluation prioritizes larger courses to simplify the implementation of the randomization and the data collection activities.

Course providers: The evaluation prioritizes diversity in the training providers for the same vocation. This will enable the analysis of the effectiveness of the trainings for providers of the same courses with different qualities. A combination of public and private service providers is also explicitly targeted.10

Timing of the courses: Evaluation sample covers courses that start between October and December 2010. Preference was given to groups of courses starting at approximately the same time but terminating before the end of March 2011.11 This geographical proximity promoted some savings in the baseline survey which is conducted face-to-face. The latter is imposed to facilitate the collection of follow-up data at approximately the same time after the trainings have been completed.

The evaluation team worked closely with all provincial offices to plan and disseminate the trainings well in advance. In particular, an official letter explaining in detail the criteria was sent by the ISKUR general directorate to all 9 These courses are not job guaranteed. Therefore they are suitable for our purpose of measuring the effect of courses on finding jobs.

10 Inevitably, the timing of the evaluation affected the types of the training providers that are part of our sample. For example, in some provinces, the courses provided by MoNE open only in summer.11 Before the final selection of courses, the 23 provincial offices are informed about the criteria and their feedback is requested. All over Turkey, the month of July was considered problematic to achieve a large oversubscription since new employment fields are opening in summertime. For example in provinces with large agricultural sector the timing of the agricultural activities and harvest are relevant. Agricultural activity typically increases during the summer when most of the unemployed find jobs and may not be interested in training programs. The harvest time (September-October) also likely takes up some applicants. Therefore, November-December seem a better timing for the oversubscription. The provinces of Antalya, Bayburt, Denizli, Diyarbakir, Gaziantep, Isparta, Izmir, Kayseri, Kirikkale, Kocaeli, Sakarya, Trabzon, Usak stated larger seasonal effects.

11

the participating provincial offices. There, the importance of oversubscription is explained in detail. Provincial offices are also encouraged and supported to start to plan and announce courses as soon as possible after these are selected. Moreover, because of the risks of low over subscription, the regional offices were incentivized to select a set of reserve courses. Each province provide at least one reserve course whose interview is scheduled to be later than the evaluation courses’ interviews.12 Some provinces anticipated more challenges in achieving oversubscription of the courses than others. For example, the provinces of Istanbul, Kayseri, Denizli and Gaziantep anticipated having more oversubscription problems than other provinces. The ISKUR headquarters contacted periodically the heads of provincial offices and supported them to advertise more strongly the courses to avoid the risk of not oversubscribing.

The final set of courses included the evaluation is quite similar to the most popular vocations in previous years. Figure 4 refers to the final composition of trainings in the sample. The ISKUR provincial offices are requested to submit a set of proposed courses during the summer 2009. After receiving the initial proposals from all 23 provinces, the provincial course capacities are contrasted with the provincial targets (reported in Table 2 above). Those provinces that did not meet the initial target numbers and/or did not present any reserve courses, are contacted. Table A3, Table A4 and Table A5 in the annex show the trainings which have the largest number of trainees in 2006, 2007 and 2008, respectively. In Table A6 in the annex, the trainings with the largest number of trainees between Jan-Nov 2009 are listed.

Figure 4: The Set of Vocations included in the Evaluation Sample

12 Some provincial offices faced unexpected budget constraints to open new courses. However, the ISKUR General Directorate backed up the evaluation in a timely manner avoiding any disruption.

12

24

38

69

6

5

3

7

5

4

33

23

2 2 2 2 1111

Computer /Computer programmingAccounting professionalist/Computerized Accounting BabysitterCashierForeign Trade and Customs ProfessionalFitter(natural gas)/PlumberOld and Sick People NurseWelder, Gas ArcRetailing and Merchandising/SalespersonCookWeaver, carpet

-ehramModelist/

StylistCoiffeur and Hair care/Skin care and beautyOperators (forklift/sewing machine)Medical SecretaryHuman resources ManagementManufacturer, furnitureApplied Basic Electronics/Electronic techniciansFinalcutWaiter, serviceMoulder, groutingMechanic, maintenance

3.4. Scheduling the interviewsThe interviews for the trainings were scheduled so that a member of the evaluation team could be physically present to monitor the quality of the data. Provinces were grouped according to their geographical proximity and the interviews for each group are set in a nearby date. Within each group, the interviews were scheduled to be as close as possible. In practice, however, this was difficult to implement. The team tried to be present in at least the first interviews of each group to identify and solve early on the main problems. All courses were scheduled to have a minimum of 12 days between the interview and the start of the courses. Although a larger period is better for the data collection purposes, it may raise attrition from the treatment. For example, the trainees may be discouraged from not hearing the result and may apply another course or be less motivated. As a result, the optimum number of days between the interviews dates and the announcements (trainee list) are 10 to 15 days. The trainee lists must be announced 2 weekdays before the starting dates of the courses. Special attention was given to scheduling of interviews to promote the oversubscription of the courses. Below we highlight some important steps taken by the team regarding:

13

Days between public course announcements and the interview dates. The average time between the course announcement and the interview dates is approximately one month. This period is considered sufficient to allow the dissemination of the information and therefore expect a large awareness in the target population.

Date of the course interviews and course starting dates. Within the same provinces, and to the extent possible, the ISKUR provincial offices did not conduct interviews for different courses in the same day. Moreover, if courses were very close in content, the team tried to maximize the numbers of days as much as possible. The same applied to the course start date. This likely increased the number of applicants for each interview thus improving the likelihood of oversubscription.

Scheduling of additional interview dates. For some courses, there was no oversubscription in the first round of interviews. The provincial offices scheduled additional interviews for the same course to achieve this. Team tried to balance two forces when deciding the number of days between the first and the follow up interviews. On the one hand, it had to be long enough to allow for more time for announcing. On the other hand, it had to be sufficiently close to keep interested successful candidates in the first round of interviews. For most courses, the additional interviews took place within one week of the first interview.

Even though, the oversubscription was not a problem in most provinces, box 2 illustrates that Istanbul presented multiple challenges.

Box 2: The Fieldwork in Istanbul

The field work in Istanbul presented several challenges. However, the major challenge probably related with the low initial oversubscription for most of the evaluation courses. Given its large size and geographical spread, Istanbul is a unique province. Labor markets are characterized by a large and dynamic informal sector, which creates a diverse set of job opportunities at decent earnings. Therefore, in Istanbul the ISKUR vocational trainings are possibly less attractive as there are large opportunity costs for the trainee’s time.

Since the evaluation team faced very low over subscription rates in Istanbul, several actions were taken to minimize this problem. First, evaluation team organized a longer and more detailed capacity building event with the training providers in Istanbul. In addition, several follow up meetings with the multiple ISKUR offices in Istanbul were conducted to involve them actively at all the states of the evaluation. Second, the ISKUR management (centrally and locally) became more pro-active in Istanbul following a letter to the ISKUR central and provincial offices stating the importance of Istanbul to the study

14

4. Defining the Evaluation Sample: 4.1. The most relevant groups in the evaluation

The training providers were requested to provide a larger than usual shortlist of the “best interviewed candidates” for this study. In their standard screening of candidates, almost all eligible applicants to the ISKUR courses are interviewed by the training providers; only a small number is selected to participate. Most of the applicants who meet basic requirements for courses are usually interviewed by training providers.13 The group of best candidates is selected and admitted into the program. A small number of applicants are wait-listed while the remaining are rejected. However, for this impact evaluation, the procedures were slightly adjusted. Training providers interview all candidates that are suitable and select a list that is longer than usual. The list of the “best candidates” is now at least twice the size of the training slots.

Figure 5: Selection of ISKUR Trainees

Training providers are requested to select a list of the best candidates that is at least 2.2 times the number of training slots. Take for example, the case

13 The course requirements are very diverse across the trainings. For example, applicants for a computer course may need a secondary school degree, while other vocations are offered only for female participants.

15

All Applicants

Eligible Candidates Non-eligible Candidates

All Successful Candidates Waitlisted 2

The Best Candidates

Treatment Group

Control Group

Waitlisted

of a course with a total of 50 seats, this implies that the training provider will select the best 120 applicants. Among these, 50 are randomly assigned to the treatment group, 50 to the control group and 20 to the waitlisted group.14 When oversubscription implies a ratio of applicants to training size, between 1.8 and 2.2, the size of the treatment and control groups together are smaller than the total course capacity. In our example above, assume that the training provider selects only a list of the 100 best applicants. In this scenario, the evaluation team instructs that only 40 applicants are randomly selected to be treatment and control. The remaining 10 training slots need to be filled with the applicants that are waitlisted (i.e., they are trainees who are left outside the evaluation study). When the course capacity is smaller than 20, the size of the list of the best applicants list is smaller than 1.8 times and the treatment group is smaller than 10 individuals, the course is not included in the evaluation study. The reason for the former was to minimize the dispersion in the number of courses considered.

After receiving the short list of the best candidates, ISKUR conducts an automatic randomization among the treatment and control groups. Treatment and control groups are randomly selected among the group of the best applicants (together with the waitlisted). The treatment group is the set of individuals who will be offered a slot in the training program. The control group is the set of individuals who will not be offered a slot in the training programs. The waitlisted group is the set of individuals who will not be offered a slot to participate but are also not rejected. The participants in the evaluation consist only of those individuals in the treatment and control groups. This is the only group who will be interviewed at baseline and follow up.

The implementation of the evaluation requires the selection and labeling in the ISKUR’s MIS of some additional groups beyond the treatment and control group, including a second group of waitlisted and the non-eligible trainees. The waitlisted are the set of randomly selected applicants among the group of the “best candidates” defined during the interview process.15 This group is approximately 20% of the list of all the successful candidates. The second waitlisted are the set of individuals who, although not among the “best” applicants, are considered good enough by the training providers to participate in the trainings. They are not randomly selected and account for approximately 10% of

14 In this case, the ratio of the number of best applicants to course capacity is 2.2 (=120/50).15 Figure 5 shows that this group is randomly selected together with the set of treatment and control groups.

16

the number of training slots. When an individual in the treatment group drops from the training (up to a 10th of the training period), the training provider can replace the trainee first with a replacement from the waitlist and, upon exhaustion of the first waitlist, they may access the second waitlist. Finally, the trainees not in the evaluation group are relevant only when the oversubscription is not large enough and course capacity is small. In this case, a randomly selected group among the most successful candidates will take up the trainings but will not be part of the evaluation sample.16 Finally, the group of individuals who were interviewed but are considered not suitable to take up the courses by the providers are also labeled accordingly by the MIS system.

All these groups are appropriately labeled in the ISKUR’s MIS. In sum, in the ISKUR MIS, each applicant is classified into one of five categories - treatment, control, waitlisted, waitlisted 2, trainee not in the evaluation and not suitable candidate17.

16 In this case, the sum of treatment and control groups are together smaller than the total course capacity. The final size depends on the size of the list of best candidates and on the course capacity.17 The baseline and follow up data will only be collected for the set of individuals in the evaluation sample. This includes only those that are either in the treatment or in the control groups.

17

Box 3: The Pilot: Testing Main Design and Implementation Features of the Evaluation

In September 2010, and before the trainings start nationwide, a small pilot was designed to test the quality of the baseline questionnaire and the main features in the evaluation. The pilot study focused on two vocational trainings: secretary and graphic designer. The pilot highlighted important facts with implications for the scale up. Indeed, following the pilot there were also several adjustments in the monitoring and information system.

First, the pilot highlighted a large number of dropouts. This implied that the providers have to select a large enough list of the “best candidates”. Table 3 below shows the applicants’ dropout rate from the pilot until the start of the course, and up to 10% of the training period:

Table 3: Learning from the Pilot: The Applicants’ Drop-out Rate in Course X

Treatment Control Waitliste

dWaitlisted

2 TotalNumber of applicants 50 50 10 13 123Dropouts after the announcement of the trainees and until the start of the course

8 0 3 1 12

Dropouts between the start of the course and up to 10%

1 0 0 0 1

Total dropouts: 11 0 3 1 15(% applicants) 0.22 0 0.30 0.08 0.12

Table shows that for the graphic designer course, 22% of the initially selected trainees never attended or stopped attending the trainings early on. The numbers are even higher for the secretary course, and the data is provided in Annex A1. The course providers are allowed to fill these slots with individuals from the waitlist up to 10% of the course period. Furthermore, the table in the annex for the secretary course also shows that some of individuals in the control group ended up being invited to the course. These could not be replaced due to the small size of the initial waitlisted group and did not allow a full replacement. Therefore, a second waitlist was created – waitlist 2 – based on a small set of second best applicants.

Finally, the pilot also illustrated a number of other important points. First, the non-response to the survey was close to 10%. Second, the interface for the electronic registration had to be improved and an alternative plan

18

4.2. The short and medium term take up of the trainings by the non-trainees

Different strategies were discussed with ISKUR to discourage individuals in the control groups from taking other trainings. Maintaining most of the control group without participating in any training during the evaluation period (or approximately 18 months) is a very important dimension of the identification design. Even though most courses are only offered once a year, applicants can apply to other training courses. The ISKUR MIS tracks some of the registered unemployed overtime. In particular, it identifies when these individuals that are not suitable or waitlisted became trainees some months following the completion of the courses they initially applied for. Table 2 reports this information for six months and one year, respectively.18 Based on this information it was decided to discourage the participation of the control group in trainings for 6 months.

Table 4: The short and medium term take up of the trainings by the non-trainees

Initial Status Individuals whobecame trainees Total

Percentage of individuals who became trainees

Panel A: 6 months after application Waitlisted candidates 93 703 13.2Not suitable candidates 306 2,692 11.3Panel B: 12 months after the application Waitlisted candidates 109 703 15.5Not suitable candidates 354 2,692 13.1

Source: Author’s based on the ISKUR MIS for the 23 provinces in the evaluation (2009).

Note: Panel A reports the number of “waitlisted” and “not suitable” candidates who became trainees 6 months after applying. Panel B reports the number of “waitlisted” and “not suitable” candidates who became trainees 12 months after the application. “Individuals who became trainees” are the applicants that become trainees after not being selected in the initial interviews (when they were labeled waitlisted and not suitable in the system).

The evaluation team is always informed if individuals included in the control group apply to other training programs. Although some exceptions may be granted and applicants in the control group can become trainees, these are rare events and monitoring is strict. The evaluation team and ISKUR defined a 18 Data refers only to the 23 provinces that are part of the evaluation sample.

19

clear set of rules to guarantee the high quality of the evaluation. In particular, the evaluation team and ISKUR committed not to accept any individual in the control group into the training course without the prior approval of evaluation team. Until the final completion of evaluation, ISKUR provincial offices will also send reports regarding the applications of control group members.

The ISKUR provincial offices in smaller provinces considered the program take up among the control group, and especially among the most skilled, a greater concern. ISKUR provincial offices in smaller provinces have more difficulties in maintaining the control group members since the individuals know each other. In addition, courses with higher pre-requisites for the applicant’s skills – such as computerized accounting, network specialist, airplane maintenance – are also more likely to have controls contaminated. Since the skills are higher, individuals are more likely to apply several times for other courses.

5. Improving the ISKUR IT monitoring system This section discusses the improvements in the ISKUR MIS with the objective of improving the dissemination of the training courses, improving the training Providers Accounts to facilitate the collection of information, the randomization code and the procedures that minimized the take up of the trainings from applicants in the control group. Finally, we describe some of the periodical reporting issued by ISKUR through their MIS. They automatically generated the lists of the most successful candidates among all the applicants.

5.1. Dissemination of the training programs at the local level

The announcement of the courses relied heavily on the information available through the ISKUR MIS. Individuals may become aware of the courses through the announcements that sent to a group of randomly selected individuals among the registered unemployed, and after defining other desirable characteristics depending on the content of the vocations.19 In addition, individuals may also apply to a course through the internet or at the local ISKUR office. All these individuals are labeled in the ISKUR M&E system as "invited" and the IT system will automatically send emails or cell-phone messages to the group. Some provincial labor offices also mail letters.19 The most common restrictions include minimum and maximum age restrictions and a minimum level of education. Other constraints include the date of registration with ISKUR, location and size of the population.

20

The content of the mails, emails and cell-phone messages is enriched to provide more specific information about the content and location of the courses. For example, the ISKUR’s web address is added to the content of the cell-phone messages. Also included, some information on the number of the course and the date and address of the interview. In practice, there are large trade-offs between a more intensive outreach and the saturation of the individual when the number of the messages increases.20 In addition, information about course duration (days of week, hours a day - during the week and weekend separately, total number of days, total number of months), the details on course provider (title, address, phone numbers, the address of their webpage), the rules (attendance, signing a commitment letter, restriction on taking ISKUR courses in 6, 12 and 24 months, being unemployed, and etc), the minimum criterions to apply for this course, the rights (stipend, insurance, unemployment payment – the trainees are allowed to receive it while they are attending to a course) and employment consultancy were added to the contents of e-mails and webpage.

5.2. Improving the training provider’s accounts in MIS

The training provider’s account in the MIS plays an important role in the evaluation. All the training providers must be registered with ISKUR to offer a training course. The results of the interviews, including the list of the best candidates, are recorded in the MIS in this section. The ISKUR staff has access to this information confirming the applicants' current employment status and whether an applicant took a course from ISKUR in 24 months. Updated contact information can also be collected. The successful applicants are informed and invited to courses before their names are registered into the system. Before the evaluation, the trainees were classified as such in the system only after some candidates drop out of the courses.21 This change allows ISKUR to make a better assessment of the over subscription and drop-out rates. 22 20 ISKUR staff was concerned with the potential increase in costs – originating from sending two instead of one text message – without necessarily improving the outreach. Some of the unemployed do not read the second message. The emails and the webpage of the training courses on ISKUR's system also improved. 21 There could be different reasons for applicants to drop out of the courses. For example, some may change their minds and decide to take another course, others may find a job.22 During the interviews, the providers can either “Search” an existing profile of candidates or “Register” new applicants. After entering the identity number of an applicant and clicking on "Search" button, course providers can see whether this person is registered to ISKUR or not. If an applicant did not register to ISKUR, he/she is not allowed to interview. Otherwise, he/she must enroll.

21

The new interface is a friendly tool that updates applicants’ and providers’ data into the system. The staff conducting the interviews can register the applicants and check whether they are currently working. Through this interface, the provider can also have access to any information ISKUR already has on this individual. In addition, the information can be filled or updated.23 Whenever the individuals are working, a warning message is prompted. Those working are not allowed to take up the ISKUR trainings. The system formally registers these individuals as “Working”. However, in practice, almost none of the currently employed applicants are excluded from the interview. This happens because applicants may quit if accepted into the trainings. In addition, there is no formal regulation on employment status of interviewees.

Although the system is very robust, in some provinces the electronic system failed and the training providers recorded the relevant data in paper format. Course providers can record the information from different interviews simultaneously. While recording the data, the provider can view the information for all the applicants, recorded up to that time in another interface. The order of this list depends on the time of recording. Each course provider can contact interviewees with the help of this list and the contact information. In case the system fails, due to for example a power outage, a hard copy of the results needs to be available. The training providers are responsible to upload this data electronically as soon as the system is back to normal.

The system is important tool to enforce some of the pre-requisites needed to take up the training courses. After registering or uploading the info on a new applicant, the system automatically checks some requirement for the course, including age, education level, gender and whether person took a course over the last 24 months. If providers fail to enter all the requested information, the interviewee will not be able to be recorded as a trainee.

23 The headings on this screen are: 1) ID Number; 2) Name; 3) Surname; 4) Education level; 5) Home address; 6) Phone number (home); 7) Cell phone no; 8) E-Mail; 9) Emergency contact Name - Surname;10) Emergency contact Phone Number (home); 11) Emergency contact Phone Number (work); 12) Emergency contact Phone Number (cell); 13) Employment status (Yes/No: Recorded automatically after clicking on “Search” button). Note that both the edited information and the original information are being kept on the system.

22

The information on whether each applicant is successful during an interview is recorded into the MIS which latter randomizes among the group of the most successful candidates. The training provider account, keeps the information on whether each applicant was successfully passed the interview. The system also identifies the group of individuals who are not among the best applicants but are considered good enough to be part of the waitlisted-2 group. Providers are requested to enter the outcome of the interview into the system on the same day, or at most one day after the interview. This is important to ensure a timely randomization. It is also important for evaluation purposes because the baseline data need to be collected prior to the disclosure of the results and to the start of the courses. ISKUR provincial offices are requested to complete controls on the applicant’s employment status and share the final randomized list of applicants at most 2 days after the interviews.24

5.3. The randomization code

The randomization was implemented through an automated code built into the MIS. The code checks the validity of the main parameters and produces different results depending on the strength of the oversubscription. The randomization is conducted by computer after the provider’s finalize the short list of the most successful candidates. The computer code randomly divides the group of the most successful interviewees into treatment, control and the waitlisted. As explained in section 4, the randomization is different depending on the relative size of the list of the most successful candidates to the number of training slots. Let p be the size of the successful list relatively to the course capacity. Then p= m/c, where m is the number of most successful candidates and c is the course capacity. Depending on the value of c and on whether (1) p<1.8 or (2) p>=1.8 & p<2.2 or (3) p>=2.2, different treatment and control groups are constructed. In addition, the randomization code also checks the validity of the threshold on the oversubscription.25 Finally, the randomization is stratified by gender and age.

5.4. Minimizing the take up of the trainings by the control group

24 This list is denoted “Questionnaire Report” in the MIS.

25 If the list of successful applicants is not enough to constitute treatment and control groups, the code gives an error and lead the user to reach and inform the evaluation team.

23

The system is refined to prevent the take up of the trainings from those in the control group. On the one hand, applicants selected into the control (and also in the treatment) group no longer receive invitations for additional ISKUR courses. This is expected to minimize the likelihood of re-application, especially for controls. On the other hand, the IT system prompt an error messages if any of the applicants in control group is labeled in the future as “trainee”.26 There are exceptions that may be overruled by the evaluation team, though. In the IT system, the labels of treatment and control that are initially allocated immediately after the randomization remain constant for the remaining of the evaluation.

5.5. The lists of the most successful applicants

A formal report with detailed information on all the individuals in the evaluation sample is produced by the system and shared with all counterparts. Following the randomization, the system produces a list with all the individuals in the evaluation sample. The report is accessible to the ISKUR staff involved in the study, to the course providers and to the survey firm.27 The report gives priority to the control group members so that they are interviewed earlier. In particular, the control group is labeled as “1st group to be interviewed”, treatment group is labeled as “2nd group to be interviewed” and finally waitlisted will be labeled as “3rd group – Reserve”. The survey firm receives this report after it is revised by the IE field coordinator.

The report with the information on the waitlisted applicants is accessible to the training providers only after the beginning of the course. The report 26 System gives an error saying that “This person is not allowed to be a trainee. Please contact with ISKUR General Directorate”. Only one staff from ISKUR General Directorate can give permission for an individual in the control group to take a training program.

27 The report is denoted “Questionnaire Report” and it contains the following information: a) ID Number; b) Name; c) Surname (Recorded by the course provider); d) Surname (on the system records); e) Gender (on the system records); f) Date of birth (on the system records); g) Education level (Recorded by the course provider); h) Education level (on the system records); i) Home address (Recorded by the course provider); j) Home address (on the system records); k) Phone number (home) (Recorded by the course provider); l) Phone number (home) (on the system records); m) Phone number (cell) (Recorded by the course provider); n) Phone number (cell) (on the system records); o) E-Mail (Recorded by the course provider); p) E-Mail (on the system records); q) Emergency contact Name-Surname (Recorded by the course provider); r) Emergency contact phone number (home/work) (Recorded by the course provider); s) Emergency contact phone number (cell) (Recorded by the course provider); t) Questionnaire status.

24

produced through the system also includes contact information of the trainees not suitable and on the waitlisted-2. This information is available on the electronic format and in paper. Course provider can view this list two days before the beginning of the course through their accounts. The system prevents to have access to this information earlier than this period. 28

6. Important Activities for Training Providers, ISKUR Provincial Offices, and the Evaluation Team

This section summarizes the main activities and responsibilities of the ISKUR provincial directorates and for the course providers. The evaluation involves the close collaboration between the ISKUR central and provincial offices, the training providers, the World Bank and the survey firm. They differ substantially depending on the status of the evaluation and thus they are divided into: (a) activities conducted before the interviews, (b) activities conducted during the interviews, and (c) activities conducted following the interviews. Some responsibilities are present throughout the whole evaluation. For example, the ISKUR provincial offices cannot inform the participants that they are part of an evaluation study. In addition, throughout the whole evaluation period, ISKUR is responsible for supporting and ensuring highest quality possible for the activities led by the training providers.

Before the interviews

ISKUR Provincial offices and training providers collaborate closely to disseminate the information on the training programs. Several communication channels were used, including announcements through the web pages of the training providers and ISKUR, the network of training providers and previous trainees. Cell-phone messages, billboards, colorful and attractive posters or flyers, advertisements on newspapers, local magazines and TV were some of the other channels used. Often, this was conducted in close cooperation with all the municipalities. The contact information on the potential trainees in the MIS systems was used to send the announcements (including letters, SMSs). ISKUR also motivated the course providers to advertise aggressively and prepare a new protocol including the requirements and duties related with the evaluation. Finally, ISKUR senior staff received training and they trained their junior staff before the

28 The course provider's interface with new screens and reports was piloted across the 23 provinces in August 2010.

25

interviews so that at least one ISKUR staff can attend each interview. This was important to ensure high quality.

The evaluation team is responsible for the overall coordination of the study, including drafting the evaluation letters and for the follow up visits to all provincial offices. The evaluation team prepared two important letters before the evaluation started. The letters informed on the selection criteria for the training courses selected to be part of the evaluation; and on the implementation of the evaluation. The ISKUR general directorate distributed these letters to provincial offices. Letters give precise directions, state deadlines and responsibilities. The evaluation team has followed up with all the provincial offices before the first interviews took place.29

During the interviews

On the interview day, and before the start of the interview, training providers need to provide detailed information about the courses to interviewees, including: (1) pre-requisites for the course, (2) benefits received (e.g., stipend, unemployment wage, insurance) and conditions linked with being an ISKUR trainee (e.g., registered with ISKUR, unemployed), (3) the commitment letter (to be signed by interviewees),30 (4) exclusion from taking up other courses supported by ISKUR after being a trainee ( e.g., prohibited to take another ISKUR course within 24 months), (5) date and form of the disclosure of the results (6) course start and end dates, (7) Course duration (e.g., days, hours), (8) a brief definition of the vocation and the possible job opportunities, and (9) information about the survey study and the possibility of being invited to answer the “face-to-face” survey. Training providers also need to ensure that they correctly assess the willingness of the interviewees in taking and completing the trainings. This can be done with the help of a set of standardized questions on the type of commuting and overlap of trainings with the school and other activities.31

29 The meetings were conducted in two parts. In the first part, ISKUR staff is informed about the implementation of the evaluation and their role during the process. In second part, ISKUR and course providers were trained on how to use new interface in the MIS. The deadlines and the responsibilities of each party were also communicated in this session.30 The copies of the commitment letter (which includes rules and rights) or any material related with the training can be presented in waiting rooms. 31 “Why did you choose this course?” “How far away do you live from course location/how long did it take for you to get here?” “What is your primary means of transport?” “Do you

26

Informing the applicants about the main purpose of the evaluation study improves the willingness to take-up the face-to-face survey. Some of the surveys were conducted in the interview place. Applicants were informed that this study is being led by ISKUR and that their support is a key ingredient for its success. It is disclosed that the ultimate objective of the study is to improve the services provided by ISKUR. This clarification was important to gain the participants’ trust and support. It is clearly stated that participating in the face to face survey does not affect in any way the outcome of the interview.

After the interviews

The ISKUR provincial offices need to confirm the employment status of the most successful applicants. The final short list is sent to the evaluation team. Those identified to be working need to be removed from the list and replaced by candidates in the waitlist.

Whenever there is not enough oversubscription, the ISKUR provincial offices are responsible to schedule additional interviews. This is done in close collaboration with the training providers. They are also responsible to schedule additional courses – and invite new potential candidates – when the course in the evaluation sample presents problems to the identification strategy.

Until the completion of the study, ISKUR provincial offices need to report to the evaluation team on the number of re-applications from those in the control group. The evaluation team and the ISKUR general directorate need to be informed whenever there are several attempts from any given candidate. Only the evaluation team and ISKUR General directorate can overrule this restriction.

Provincial offices need to cooperate closely with the survey firm. Some baseline face-to-face interviews were conducted in the provincial offices when the space was available. ISKUR can also clarify the objectives of the study and inform on the follow up contacts by the survey firm.

think you will be able to get here every day?” “Do you have permission to from your relatives to attend the course?” “Will this course conflict with exams/school?” “Do you have a job that starts soon or any other commitment that might prevent you from taking this course if accepted?”

27

7. Data Sources

The impact evaluation requires data collection in at least two points in time, before and after the trainings take place. The baseline survey was collected face-to-face, before the training courses start. A follow-up survey is planned to be also face-to-face and will be collected more than 18 months after the trainings started. (possibly early in 2012). Between the baseline and the follow-up, the evaluation team is planning a short exercise to track some of the individuals in the sample. This is important to minimize the attrition from the sample, which is composed by relatively young and mobile individuals.

The baseline data collection started in September 2010 in Istanbul and was finalized in January/ February 2011. This data will capture detailed socio-economic information of each applicant as well their labor market trajectories. Some psychological treats of applicants - relevant to measure their efforts and persistency in searching for jobs - together with household characteristics and perceptions on training quality was also collected. An analysis of this data is done in World Bank (2011).

Box 4: Looking into the Content of the Baseline Survey

28

The baseline survey collected detailed information on the training applicants. First, it collected current and past labor market outcomes of applicants. The proposed set of indicators will inform on the individual current employment status (employed, unemployed, searching), labor earnings as well as on other indicators of job quality if employed (including number hours working, formality of the employment contract, social security coverage).

Second, it gathered basic information on the characteristics of the potential trainees. Third, it also identifies some of the channels (or intermediate outcomes) through which ISKUR trainings might influence workers’ employment and earnings outcomes. Trainings may increase individual productivity (or wages) through an increase in individual skills (either cognitive or non-cognitive skills), through reduced costs of searching for a job, through greater effort (optimism/enthusiasm) or networking. A special module with Raven’s test has also been included to capture an individual’s cognitive skills.

Forth, since the ISKUR trainings seem to be very useful for trainees we also propose to better understand why individuals are not capable of investing in these trainings on their own. For this we propose the collection of some information on individual assets and credit constraints, on the individual degree of risk aversion - to human capital and other investments, on individual’s behavioral responses to multi-year choices and on the accuracy of individual expectations on the importance of the trainings.

Finally, the evaluation will be looking at changes over time in the main

The baseline data collection was done through a face to face survey collecting comprehensive data on treatment and control individuals. Special attention was given to (i) pre-program levels of the main labor market outcomes of interest; (ii) pre-program measures of key control variables, including characteristics which are expected to result in heterogeneous reactions to the treatment; and (iii) tracking data. A list of 6,072 evaluation participants was shared with the survey firm. A total of 5,318 interviews were completed. Table A7 in the annex reports the number of trainees in 2009, the number of courses in the evaluation sample, and the number of registries shared with the survey firm for each of the provinces in Turkey.

29

8. AnnexA1: The number of dropouts in pilot - Secretary Course

Pilot Course 2 : Secretary Course

Treatment

Control

Waitlisted

Waitlisted2 Total

Number of applicants 25 25 3 NA 53

Number of dropouts after the announcement of the trainee list till the beginning of the course

8 5 3 NA 16

Number of dropouts after the beginning of the course till the 1/10th of the course

0 2 0 NA 2

TOTAL number of dropouts: 8 7 3 NA 18Percentages of dropouts: 0.32 0.28 1.00 NA 0.34

30

A2: Selection of provinces according to the unemployment rate and the number of oversubscribed courses in 2009

Code Province

Unemployment Rate

in 2009Strata Selected

Provinces

1 ADANA 20.5 2 02 AGRI 6.5 1 03 ANKARA 11.8 2 14 ANTALYA 9.7 1 15 AYDIN 12.4 2 06 BALIKESIR 7.9 1 07 BAYBURT 4.7 1 18 BINGOL 14.8 2 09 CANAKKALE 7 1 010 CORUM 7.5 1 011 DENIZLI 9.4 1 112 DIYARBAKIR 15.7 2 113 DUZCE 10.2 2 114 ELAZIG 15.5 2 115 ERZINCAN 6.1 1 016 ERZURUM 6.2 1 117 ESKISEHIR 9.9 1 118 GAZIANTEP 16.8 2 119 HATAY 17.7 2 120 ISPARTA 8.1 1 121 ISTANBUL 11.2 2 122 IZMIR 11.8 2 123 KASTAMON

U 5.3 1 024 KAYSERI 11.1 2 125 KILIS 10.9 2 026 KIRIKKALE 11.1 2 127 KOCAELI 11.6 2 128 KUTAHYA 6.5 1 029 MANISA 9.8 1 130 MUS 13.2 2 131 NEVSEHIR 9 1 032 RIZE 5.2 1 033 SAKARYA 8.5 1 134 SAMSUN 7.8 1 035 TEKIRDAG 9.1 1 136 TOKAT 5.9 1 037 TRABZON 6.1 1 138 USAK 9.2 1 139 YOZGAT 12.2 2 0

31

Note: Strata equals one if unemployment rate < %10, = 2 if unemployment rate > %10.

A3: ISKUR vocational training courses with the largest number of trainees in 2006

Order Name of the Vocation

Number of Courses

Number Of Trainees

Male Female Total

1 Cable Network Assembly Operator 148 1884 1085 29692 Computer 125 1434 444 18783 Production project for the disabled 1 720 720 1440

4Spot welding and Machine-tool assembly 45 802 0 802

5 Laborer, manufacturing/readymade 15 221 226 4476 Fitter, natural gas 19 326 3 3297 Welder 16 227 37 2648 Manufacturer, furniture 14 209 20 2299 Sewing Machine Operator 13 95 108 20310 Cook 12 188 2 190

TOTAL (Overall) 622 8603 382312426

A4: ISKUR vocational training courses with the largest number of trainees in 2007

Order Name of the Vocation

Number of Courses

Number Of Trainees

Male Female Total

1 Computer operator 117 1,346 271 1,61

72 Computing software operator 70 741 249 9903 Computerized Accountant 45 473 398 8714 Sewing-machine operator (fabric) 28 161 336 4975 Electric arc welder (with machines) 19 309 2 3116 Furniture manufacturer 16 241 1 2427 Security guard 11 195 28 2238 Electronic computing operator 13 191 23 2149 Sales person 10 94 84 17810 Sewers by hand and with sewing 8 76 97 173

32

machine

TOTAL 1,200 16,463 6,371 22,8

34

A5: ISKUR vocational training courses with the largest number of trainees in 2008

Order Name of the Vocation

Number of Courses

Number Of Trainees

Male Female Total

1 Computer operator 183 1,831 474 2,305

2 Computing software operator 114 1,119 446 1,565

3 Computerized Accountant 69 601 548 1,1494 Manuel Worker 45 777 296 1,0735 Sewing-machine operator (fabric) 49 152 859 1,0116 Sales person 28 266 266 5327 Office clerk (general) 1 250 250 5008 Security guard 26 371 72 4439 Gas Arc Welder 24 392 1 393

10Sewers by hand and with sewing machine 18 77 272 349

TOTAL 1,806 20,586

11,341

31,927

A6: ISKUR vocational training courses with the largest number of trainees between Jan-Nov 2009

Order Name of the VocationTrainees

Number Percentage

1 Computerized accountant 8,658 6.52 Security officer 7,707 5.73 Computer manager 7,552 5.6

33

4 Babysitter 4,125 3.05 Sewing machine operator (fabric) 3,311 2.46 Security officer (armed) 3,255 2.47 Cook 3,243 2.48 Caregiver (elder and sick) 3,026 2.29 Slopworker/ Confection worker 2,854 2.110 Computer repairer (maintenance and repair ) 2,828 2.1TOTAL 133,284 100

A7: Sample Trainees, Courses in the Evaluation and Surveyed by the Firm

No Province

Number of

trainees in 2009

Target interviews

Number of courses in the

evaluation sample

Contacts sent to the survey firm

1 ANKARA 3,344 308 9 3002 ANTALYA 1,771 163 3 1503 BAYBURT 249 100 3 1504 DENIZLI 1,420 131 4 152

5DIYARBAKIR 450 100 2 100

6 DUZCE 702 100 2 527 ELAZIG 1,204 111 2 1008 ERZURUM 836 100 2 1009 ESKISEHIR 2,450 226 7 34010 GAZIANTEP 751 100 3 12011 HATAY 2,889 266 8 50012 ISPARTA 1,152 106 3 11013 ISTANBUL 19,208 1,771 29 1,31414 IZMIR 1,971 182 5 19015 KAYSERI 3,067 283 5 28216 KIRIKKALE 1,329 123 4 12017 KOCAELI 7,519 693 15 60418 MANISA 3,547 327 6 29019 MUS 1,938 179 1 20020 SAKARYA 1,208 111 4 11821 TEKIRDAG 1,709 158 5 15022 TRABZON 1,628 150 4 18023 USAK 1,484 137 4 150

34

TOTAL 61,826 5,925 130 6,072

35