Transforming the Culture of Evaluation in Career and Employment Services

Saskatchewan Abilities CouncilSaskatchewan Education, Employment and Labour

New Brunswick Post-Secondary Training, Education and Labour

Canadian Career Development Foundation (CCDF)Goss-Gilroy Inc.

Canadian Research Working Group on Evidence Based Practice in Career Development (CRWG)

Common Indicators:

CCDF: Lynne Bezanson & Céline Renald

Dave Redekopp, Life-Role Development Group

Guylaine Michaud, Université de Sherbrooke

Ken Organ, Goss-Gilroy

Provincial Partners

◦ Managers and Practitioners◦ Clients: all who agree to participate; no criteria except not

one-off’s

Research Team

U Name It StructureInput Process Outcome

Indicators of client change 1. Learning outcomes

• Knowledge and skills linked to intervention

2. Personal attribute outcomes• Changes in self-

esteem, self efficacy, well-being, self confidence, self-awareness)

3. Impact outcomes • Impact of #1 & #2 on

client’s life, e.g., employment status, training status, quality, fit, standard of livinga

Activities to promote positive change

Generic interventions• Working alliance,

client engagementSpecific interventions and progress in:

Job ReadinessCareer Decision MakingSkill EnhancementWork SearchWork MaintenanceLife Circumstances

The support you provide and progress in personal attributes:

self-esteem, self efficacy, well-being, self confidence, self-awareness)

Resources available

• Staff: Number of staff, level of training, type of training

• Funding: Budget• Agency mandate• Community

resources• Access to

Employment Opportunities and Training

• Client Employment History

• Client Life Circumstances

What common indicators are applicable across different client contexts, different client groups, different agencies, and different interventions? 

What statements can be made about service effectiveness by tracking common indicators of inputs, processes and outcomes? If numbers permit, it will be possible to aggregate data to begin to address the ultimate question, “What kinds of interventions in what contexts produce what kinds of outcomes?”

Research Questions

5

Input ◦ LABOUR MARKET REALITIES: Catchment Area

Profile Local labour market opportunities – what is the quality and

range of work opportunities? Educational opportunities – easily accessible or not? Community supports/resources – adequate? accessible?

◦ CLIENT REALITIES: Client Employment Potential (demographic info) Work experience Level of education Language Health factors Other

6

Input ◦ PRACTITIONER PROFILE

Age Gender Experience Education/Training

◦ SERVICES OFFERED BY YOUR OFFICE Provided by managers

Process Build & Maintain Working Alliance with Client

Conduct Employability Needs Assessment◦ 5 Employability Dimensions and Additional Life

Circumstances

Develop and Update Action Plan

Interventions (throughout 6 weeks)

Track Client Progress (as appropriate throughout 6 weeks)

PRE-EMPLOYABILITY/JOB READINESS: Has the intention to seek employment and/or enter training but faces challenges, vulnerabilities and/or skill deficits that need to be addressed before the client can be successful in achieving employment/training goals

CAREER DECISION MAKING: Not yet clear about work/learning goals; needs to understand personal skills, interest, values and personality and more about different potential occupations in order to develop a vision for a preferred future

  SKILLS ENHANCEMENT: Has career goal, but needs to explore skills and qualifications

needed for the work they want and how to acquire them  WORK SEARCH: Has career goal and necessary skills/qualifications for desired work; needs

skills, strategies and tools necessary to conduct a successful job search  EMPLOYMENT MAINTENANCE: Has difficulty keeping work or may need to work on the

skills/attitudes required for success at work  OTHER: Many circumstances beyond those covered by the employability dimensions may

impact on the client’s employability outcomes. Some of these life circumstances may necessitate referral outside of career and employment services. Life circumstance issues may need to be addressed and resolved before progress toward employment can realistically be made.

Employability Dimensions

Client Progress Indicators Employability Needs

◦ 5 Employability Dimensions◦ Additional Life Circumstances

Personal Attributes Needs/Working Alliance◦ Self-management, self-esteem, well-being, self-efficacy,

self-awareness◦ Working alliance◦ Client engagement

Labour Market Indicators◦ Tailored to client status

Not at all

Not much

A little

Quite a lot

A lot

Identify and clarify future direction (e.g.,training, education, employment or change in life circumstances goal)

         

Identify personal strengths/resources that support future direction (e.g.,training, education, employment or change in life circumstances goal)

         

Resolve specific challenges/vulnerabilities that may impact on future direction (e.g., mortgage, public transit, day care etc.)

         

Access community resources/supports that address specific personal challenges/vulnerabilities and assist in helping move goal forward (e.g., mental health services; addictions counselling; public housing etc.)

         

Develop necessary personal supports needed to move toward future direction (e.g., family, childcare, transportation)

         

Acquire life/employment skills and attitudes that support future direction

         

Acquire attitudes which support future direction           Develop and follow a plan of action to move future

direction forward         

Keep appointments and sustain efforts           Other: (please specify)________________________________          

Indicators of challenges in client life circumstances are in evidence:

Not at all

Not much

A little

Quite a lot

A lot

Improved housing is needed           Improved transportation is needed           Improved capacity to work and/or study is

needed         

Increase in sense of responsibility for own choices and behaviours is needed

         

Increase in ability to set short and long term goals is needed

         

Increase in understanding expectations and demands of employers is needed

         

Reduction in destructive behavior is needed

         

Following medical and medication protocols is needed

         

Increased openness to change is needed           Increased access to constructive and

positive support systems is needed         

Improved relationships with family and friends are needed

         

Increased trust in other people is needed           Other: _________________________________          

Life Circumstances

Client needs assistance in the following areas: Not at all

Not much

A little

Quite a lot

A lot

Ability to Self-Manage: More info

         

Developing stronger Self-Esteem: More info

         

Improving sense of Well-Being: i.e.: More info

         

Developing stronger Self-Efficacy: i.e.: More info

         

Increasing Self-Awareness: More info

         

Other (Please specify): ________________________________________

  

         

Personal Attributes Example

more info: Self-Esteem: 

 

“a person’s overall emotional evaluation of his or her own worth, an attitude to self and a judgement of oneself. Self-esteem encompasses beliefs (e.g., I am competent; I am worthy) and emotions (pride, shame). Self-esteem is the positive or negative evaluation of the self” 

Demonstrates an optimistic outlook Sees self as competent and able Has confidence in ability to interact with others Uses positive self-talk Acts with little hesitation Raises few objections Asks questions Articulates skills assertively Makes eye contact Is hopeful Takes reasonable risks

Self-Esteem Attribute

Outcome Indicators Employability Needs

◦ 5 Employability Dimensions◦ Additional Life Circumstances

Personal Attributes Needs/Working Alliance◦ Self-management, self-esteem, well-being, self-efficacy,

self-awareness◦ Working alliance◦ Client engagement

Labour Market Indicators◦ Tailored to client status

quality and “fit” of employment or education/training, salary, location

Input & Outcome Connections◦ More educated clients do better than less

educated clients◦ Success is positively related to labour market

attachment ◦ “No Barrier” clients do better than “Single

Barrier” clients who do better than “Multiple Barrier” clients

◦ Success is positively related to entry self-efficacy◦ Success is positively related to entry self-esteem

What We Hope/Expect to Say

Outcomes and Processes◦ Learning Outcomes

Client skill/knowledge increases with any intervention◦ Personal Attribute Outcomes

Client attributes improve with any intervention Client self-esteem improves with any intervention Client self-efficacy improves with any intervention

◦ Labour Market Outcomes Employment/training increases with any intervention Quality/fit increases with any intervention Working alliance is positively related to employment/

training outcomes

What We Hope/Expect to Say (cont’d)

Relationships between Outcomes◦ Learning outcomes are positively related to:

personal attribute outcomes labour market outcomes

◦ Personal attribute outcomes are positively related to labour market outcomes

Personal Attribute Outcomes & Processes◦ Working alliance is positively related to self-

efficacy changes◦ Working alliance is positively related to self-

esteem changes

What We Hope/Expect to Say (cont’d)

Learning Outcomes & Processes◦ Client skill/knowledge increases are positively

correlated with working alliance Measurement Relationships

◦ Practitioner and client assessments of working alliance are positively related

◦ Practitioner assessment of personal attributes is positively related to client self-esteem scores

◦ Practitioner assessment of personal attributes is positively related to client self-efficacy scores

What We Hope/Expect to Say (cont’d)

U Name It StructureInput Process Outcome

Indicators of client change 1. Learning outcomes

• Knowledge and skills linked to intervention

2. Personal attribute outcomes• Changes in self-

esteem, self efficacy, well-being, self confidence, self-awareness)

3. Impact outcomes • Impact of #1 & #2 on

client’s life, e.g., employment status, training status, quality, fit, standard of livinga

Activities to promote positive change

Generic interventions• Working alliance,

client engagementSpecific interventions and progress in:

Job ReadinessCareer Decision MakingSkill EnhancementWork SearchWork MaintenanceLife Circumstances

The support you provide and progress in personal attributes:

self-esteem, self efficacy, well-being, self confidence, self-awareness)

Resources available

• Staff: Number of staff, level of training, type of training

• Funding: Budget• Agency mandate• Community

resources• Access to

Employment Opportunities and Training

• Client Employment History

• Client Life Circumstances

Where “U-Name-It” is headed… More evidence for the field to justify the power

and efficacy of career interventions

More integrity around outcomes

A culture of evaluation built into career and employment service practice

Over time, the ability to answer:◦ “What kinds of interventions in what contexts

produce what kinds of outcomes”

20

Research Reports will be completed by May 31, 2013

Follow the CRWG website at:

www.crwg-gdrc.ca

Thank You

To follow the study: