A NATIONAL A NATIONAL CERTIFICATION EXAM CERTIFICATION EXAM

FOR CHILD AND YOUTH FOR CHILD AND YOUTH CARE WORKERS: CARE WORKERS:

PRELIMARY RESULTSPRELIMARY RESULTS

Dale Curry, Kent State UniversityDale Curry, Kent State University

Basil Qaqish, University of North Basil Qaqish, University of North Carolina-GreensboroCarolina-Greensboro

Organization of CompetenciesOrganization of Competencies

DomainsDomains

Sub-DomainsSub-Domains

CategoriesCategories

CompetenciesCompetencies

The NACP Competencies

DomainsDomains

I. I. ProfessionalismProfessionalism

II. II. Cultural & Human DiversityCultural & Human Diversity

III. III. Applied Human DevelopmentApplied Human Development

IV.IV. Relationship & CommunicationRelationship & Communication

V. V. Developmental Practice Methods Developmental Practice Methods

Example

from NACP Competencies Document

I. PROFESSIONALISM 6. Advocacy – MEDIUM WEIGHT a. demonstrate knowledge and skills in use of advocacy - SUPERVISOR ASSESSMENT b. access information on the rights of children, youth and families - PORTFOLIO c. describe the rights of children youth and families in relevant setting/s and systems – EXAMINATION d. advocate for the rights of children, youth, and families in relevant setting/s and systems - PORTFOLIO e. describe safeguards for protection from abuse including institutional abuse - EXAMINATION f. advocate for safeguards for protection from abuse including institutional abuse - SUPERVISOR ASSESSMENT g. ensure that children, youth and family views are heard and considered during the decision making processes which directly affect them -EXAMINATION

Situational Judgment ItemsSituational Judgment Items

Can be used to place competencies in a Can be used to place competencies in a context of relationships, events, and context of relationships, events, and settingssettings

Can help establish face validity of test if Can help establish face validity of test if they are based on real case studies they are based on real case studies collected from the fieldcollected from the field

Moves beyond the simple recitation of Moves beyond the simple recitation of learned knowledgelearned knowledge

Situational Judgment ItemsSituational Judgment Items

Case studies of real incidents were Case studies of real incidents were collected from child and youth care collected from child and youth care practitioners in a variety of practice practitioners in a variety of practice settingssettings

Used as the basis for the exam questions.Used as the basis for the exam questions.

Example ofSituational Judgment Item

Example of Situational Judgment Item

Competencies assessed by examination:

I.6.c. Describe the rights of children youth and families in relevant setting/s and systems

I.6.e. Describe safeguards for protection from abuse including institutional abuse

Item 8: Read this case study of an incident and answer the questions at the end:

Lisa (a girl in care) became angry during individual counseling session with Social Service Manager (MH) and cursed at the staff member. Lisa was escorted to “off dorm” time out due to disrupting milieu. While in time out, Lisa continued to use profanity toward staff members, urinated on the floor, and said “maybe I should pimp again and make some money…

After giving her several expectations and after Lisa refused to calm down, staff member (JT) began to physically escort Lisa to seclusion when Lisa became physically assaultive, hitting staff member. Staff members JT and SR placed Lisa in a therapeutic hold and eventually removed her to seclusion room. By 8:15 P.M., Lisa regained physical control of herself and agreed to a plan to maintain the safety of Lisa and the staff members. Lisa cleaned seclusion room and time out room.

Which of the following statements are true? [Circle True or False for each statement]

T/F Lisa’s rights were properly protected in

this situation.

T/F The physical restraint, as explained in

this case study was justified and proper.

T/F An alternative intervention such as

having the staff member (JT) switch off

with another staff member could have

been more effective.

Pilot TestingPilot Testing

STUDY GOALSSTUDY GOALS

Administer exam and get feedback Administer exam and get feedback regarding face validity & ways to improveregarding face validity & ways to improve

Conduct item analysisConduct item analysis

Examine relationship between test scores Examine relationship between test scores and supervisory assessment of worker and supervisory assessment of worker performance (concurrent validity)performance (concurrent validity)

Explore differential performance resultsExplore differential performance results

Determine a cut score (pass/fail)Determine a cut score (pass/fail)

METHODMETHOD

Assessment Work Group developed 100 Assessment Work Group developed 100 item situational judgment exam.item situational judgment exam.

Exam was administered to 775 child and Exam was administered to 775 child and youth care workers in 29 sites.youth care workers in 29 sites.

Sites located in six states & two provinces.Sites located in six states & two provinces.

METHODMETHOD

Scores were statistically compared with Scores were statistically compared with supervisor’s rating of competence.supervisor’s rating of competence.

Supervisor ratings of worker competence Supervisor ratings of worker competence (six items) administered.(six items) administered.

Examinee feedback on exam and face Examinee feedback on exam and face validity solicited through questionnairevalidity solicited through questionnaire

PROCEDURESPROCEDURES

Item analysisItem analysis

Correlation of exam total score with Correlation of exam total score with supervisor composite ratings supervisor composite ratings

Modified Angoff procedure to determine Modified Angoff procedure to determine cut point (pass/fail)cut point (pass/fail)

ITEM ANALYSISITEM ANALYSIS

Reliability analysisReliability analysis

Difficulty analysisDifficulty analysis

Discrimination analysisDiscrimination analysis

Distracter analysisDistracter analysis

Differential Item Analysis (DIF)Differential Item Analysis (DIF)

RELIABILITY ANALYSISRELIABILITY ANALYSIS

Excellent internal reliability (Chronbach’s Excellent internal reliability (Chronbach’s alpha=.90)alpha=.90)

18 items identified for possible deletion or 18 items identified for possible deletion or revisionrevision

DIFFICULTY ANALYSISDIFFICULTY ANALYSIS

Difficulty scores obtained for each item Difficulty scores obtained for each item (acceptable difficulty score range (acceptable difficulty score range from .10-.80)from .10-.80)

Also examined top and bottom 20%Also examined top and bottom 20%

Most items were of moderate difficultyMost items were of moderate difficulty

DISCRIMINATION ANALYSISDISCRIMINATION ANALYSIS

Point Bi-serial correlation between the Point Bi-serial correlation between the item and total test scoreitem and total test score

Also compared top 20% with bottom 20%Also compared top 20% with bottom 20%

.15 minimum discrimination score.15 minimum discrimination score

DISTRACTER ANALYSISDISTRACTER ANALYSIS

0.00

0.20

0.40

0.60

0.80

1.00

upper2nd3rd4thlower

Correct answer=1

0.00

0.20

0.40

0.60

0.80

1.00

1234other

Figure 1

DIFFERENTIAL ITEM DIFFERENTIAL ITEM FUNCTIONING (DIF)FUNCTIONING (DIF)

Exploring differences for race and gender Exploring differences for race and gender for those with similar abilities on the testfor those with similar abilities on the test

Use of computer program SIBTEST and Use of computer program SIBTEST and Mantel Haenszel procedureMantel Haenszel procedure

7 items identified by either SIBTEST or 7 items identified by either SIBTEST or Mantel Haenszel for race and 1 item for Mantel Haenszel for race and 1 item for gender (must give these items more gender (must give these items more scrutiny)scrutiny)

FACE VALIDITY AND OTHER FACE VALIDITY AND OTHER FEEDBACK FROM EXAMINEESFEEDBACK FROM EXAMINEES

90% indicate that the items in the exam 90% indicate that the items in the exam accurately assess important aspects of child and accurately assess important aspects of child and youth care workyouth care work

90% indicate that the case studies provide 90% indicate that the case studies provide realistic samples of child and youth care workrealistic samples of child and youth care work

80% indicate that the exam content is similar to 80% indicate that the exam content is similar to their job dutiestheir job duties

59% indicate that their performance on the exam 59% indicate that their performance on the exam is an excellent indicator of their job performance is an excellent indicator of their job performance (34% neither agreed or disagreed)(34% neither agreed or disagreed)

OTHER FEEDBACKOTHER FEEDBACK

Test is consistent with real situations-Test is consistent with real situations-overall a good and fair testoverall a good and fair test

Motivating and thought provokingMotivating and thought provoking

Test too long – perhaps give a break or do Test too long – perhaps give a break or do it in two partsit in two parts

Not familiar with some of the terminologyNot familiar with some of the terminology

Some of the case studies were hard to Some of the case studies were hard to followfollow

CANADIAN RESULTSCANADIAN RESULTS

Mean total exam score - Mean total exam score - 75.775.7 (U.S. 59.3) (U.S. 59.3)

Mean competency rating (composite of 6 Mean competency rating (composite of 6 items, 5 point scale) – items, 5 point scale) – 28.4 28.4 (U.S. 26.2)(U.S. 26.2)

Mean face validity rating (composite of 4 Mean face validity rating (composite of 4 items, 5 point scale) – items, 5 point scale) – 15.415.4 (no significant (no significant difference with U.S. examinees)difference with U.S. examinees)

COMMENTS CANADIAN COMMENTS CANADIAN EXAMINEESEXAMINEES

Exam was too long, a break was neededExam was too long, a break was neededWell-thought questions with very realistic, Well-thought questions with very realistic, engaging scenarios but a mix in style of engaging scenarios but a mix in style of question type (e.g. true/false) may have question type (e.g. true/false) may have been a welcomebeen a welcomeThe exam did not reflect the cultural The exam did not reflect the cultural aspects of British Columbiaaspects of British ColumbiaKnew about N.A. Code but only familiar Knew about N.A. Code but only familiar with Ontario Code of Ethicswith Ontario Code of Ethics

POST PILOT ACTIVITIESPOST PILOT ACTIVITIES

Explored items for possible deletion to Explored items for possible deletion to improve reliability (18 items)improve reliability (18 items)Examined content of items identified by Examined content of items identified by DIF for gender/racial biasDIF for gender/racial biasExamined examinee feedback pertaining Examined examinee feedback pertaining to each item and regarding the test in to each item and regarding the test in generalgeneralCompleted process of setting cut-off Completed process of setting cut-off (passing scores) for exam(passing scores) for exam

What remains to be done?What remains to be done?

Continue to collect and analyze data to Continue to collect and analyze data to improve the exam and accumulate improve the exam and accumulate evidence of the exam’s validityevidence of the exam’s validity

Continue to develop and research new Continue to develop and research new exam itemsexam items

Continue to explore reliability and validity Continue to explore reliability and validity of exam in future as it is implementedof exam in future as it is implemented

More InformationMore Information

North American Certification ProjectNorth American Certification Project

www.acycp.orgwww.acycp.org