S PRACTICE OF LICENSED PSYCHOLOGISTS I UNITED …cdn.ymaws.com › › resource › ...STUDY OF THE PRACTICE OF LICENSED PSYCHOLOGISTS IN THE UNITED STATES AND CANADA prepared for

STUDY OF THE PRACTICE OF LICENSED PSYCHOLOGISTS IN THE UNITED STATES AND CANADA

prepared for

THE ASSOCIATION OF STATE AND PROVINCIAL PSYCHOLOGY BOARDS P.O. Box 241245

Montgomery, AL 36124

prepared by

Sandra Greenberg, Ph.D. Carla M. Caro, M.A. I. Leon Smith, Ph.D.

PROFESSIONAL EXAMINATION SERVICE Department of Research and Development

475 Riverside Drive New York, NY 10115-0089

July 2010

Copyright © 2010 by the Association of State and Provincial Psychology Boards P.O. Box 241245, Montgomery, AL 36124

ASSOCIATION OF STATE AND PROVINCIAL PSYCHOLOGY BOARDS

Board of Directors, 2008

Alex M. Siegel, J.D., Ph.D., President Kenneth G. Roy, Ed.D., Past President Emil Rodolfa, Ph.D., President Elect

Martha N. Storie, B.S., Secretary-Treasurer Joseph S. Rallo, Ph.D., Member-at-Large

Jack B. Schaffer, Ph.D., ABPP, Member-at-Large Carol Webb, Ph.D., ABPP, Member-at-Large


Emil Rodolfa, Ph.D., President Alex M. Siegel, J.D., Ph.D., Past President

Jack B. Schaffer, Ph.D., ABPP, President Elect Martha N. Storie, B.S., Secretary-Treasurer

Jacqueline B. Horn, PhD., ABPP, Member-at-Large Joseph S. Rallo, Ph.D., Member-at-Large

Carol Webb, Ph.D., ABPP, Member-at-Large


Jack B. Schaffer, Ph.D., ABPP, President Emil Rodolfa, Ph.D., Past President

Joseph S. Rallo, Ph.D., President Elect Martha N. Storie, B.S., Secretary-Treasurer

Carol Webb, Ph.D., ABPP, Member-at-Large Jacqueline B. Horn, Ph.D., Member-at-Large Fred Millán, Ph.D., ABPP, Member-at-Large

Practice Analysis Advisory Committee

Emil Rodolfa, Ph.D., Chair Greg Gormanous, Ph.D.

Joan Grusec, Ph.D. Catherine Yarrow, Ph.D.

Practice Analysis Task Force

Consuelo Arbona, Ph.D. Nancy Gourash Bliwise, Ph.D.

Darcy Cox, Psy.D., R. Psych., ABPP-CN Wil Counts, R.Ph., Ph.D.

Dennis Doverspike, Ph.D., ABPP Kelly Ducheny, Psy.D.

John Hunsley, Ph.D. Mary Pat McAndrews, Ph.D., C.Psych.

Morgan Sammons, Ph.D., ABPP Margaret Smith-Zoeller, Psy.D.

Lois Tetrick, Ph.D. Sheila Woody, Ph.D.

PES Final Report An Update Study of the Practice of Licensed Psychologists i

Acknowledgements On behalf of Professional Examination Service (PES), we are pleased to have conducted this major research study for the Association of State and Provincial Psychology Boards (ASPPB). This report summarizes the practice of licensed/registered psychologists in the U.S. and Canada and explores the evolving nature of competence across the professional lifespan of the practitioner and provides the groundwork for its assessment. Information included in this report came from the following sources: relevant literature in psychology as well as other professions regarding the assessment of competence; information from focus panels and independent reviews by subject-matter experts; a pilot survey and a validation survey of 5000 licensed/registered psychologists; and discussions with both regulators and educators. A content-based approach was used to systematically delineate the content areas and knowledge base required for entry into independent practice. Those results represent a contemporary description of the Scientific Knowledge underlying the practice of psychology and provide the basis for developing defensible test specification for the Examination for Professional Practice in Psychology (EPPP). A process-based approach was used to systematically delineate competencies associated with five additional areas of competence: Evidence-Based Decision Making/Critical Reasoning; Professionalism/Ethics; Assessment; and Intervention/Supervision/Consultation. Those results are a key requirement for assessing evolving levels of competence, including validated behavioral exemplars of competencies required at entry into independent practice. A project of this magnitude depends on the hard work and commitment of many professionals, and we are pleased to acknowledge their contributions to the final product. This study required a substantial investment of ASPPB’s financial resources and personnel. PES endorses ASPPB’s ongoing commitment to the continued development of an exemplary examination program as reflected in the EPPP as well as its forward looking approach to the assessment of competency. We wish to recognize the enduring contributions of the ASPPB Practice Analysis Advisory Committee for the wisdom and direction it provided. Its four members—Emil Rodolfa, Ph.D., Chair; Greg Gormanous, Ph.D.; Joan Grusec, Ph.D.; and Catherine Yarrow, Ph.D.—and the ASPPB Executive Officer, Stephen T. DeMers, Ed.D., worked with us and supported our efforts throughout the conduct of the study. The 12 members of the ASPPB Practice Analysis Task Force worked tirelessly through four face-to-face meetings; and numerous e-mail-based assignments and telephone conference calls in order to refine the content areas and knowledge statements; and draft, review and refine the competency clusters, competency statements, and behavioral exemplars. They approached each task with wisdom and wit, always willing to provide their own perspectives and listen to the views of others in order to articulate a clear, concise, and contemporary description of practice. We are indebted to Consuelo Arbona, Ph.D.; Nancy Gourash Bliwise, Ph.D.; Darcy Cox, Psy.D., R. Psych., ABPP-CN; Wil Counts, R.Ph., Ph.D.; Dennis Doverspike, Ph.D., ABPP; Kelly Ducheny, Psy.D.; John Hunsley, Ph.D.; Mary Pat McAndrews, Ph.D., C.Psych.; Morgan

PES Final Report An Update Study of the Practice of Licensed Psychologists ii

Sammons, Ph.D., ABPP; Margaret Smith-Zoeller, Psy.D.; Lois Tetrick, Ph.D.; and Sheila Woody, Ph.D. for making our work so easy. We conclude by stating that except where specifically noted, the views expressed in this report are those of PES and do not necessarily reflect the view of ASPPB or of those experts who provided advice on the conduct of this investigation. We look forward to the interpretation and application of the results by ASPPB as it strives to maintain the highest level of excellence in the credentialing of psychologists in the U.S. and Canada. Sandra Greenberg, Ph.D., Vice President for Research and Development, PES Carla M. Caro, M.A., Research Director, PES I. Leon Smith, Ph.D., President and CEO, PES

PES Final Report An Update Study of the Practice of Licensed Psychologists iii

Table of Contents Acknowledgements .......................................................................................................................... i Table of Contents ........................................................................................................................... iii Table of Appendices ...................................................................................................................... iv Table of Tables ............................................................................................................................... v Table of Exhibits ........................................................................................................................... vii Executive Summary ........................................................................................................................ 1 Introduction ..................................................................................................................................... 6 Conduct of the Update Practice Analysis Study of Licensed Psychologists .................................. 7

Select the Subject-Matter Experts to Participate in the PATF .................................................... 8 Develop the Draft and Final Delineations of Content Areas and Knowledge Statements; and Competency Clusters and Competencies, and Behavioral Exemplars .............................. 8 Develop, Review, and Approve the Form and Content of the Survey of the Practice of Psychology ............................................................................................................................. 13 Pilot Test of Online Survey Instrument ..................................................................................... 17 Develop the Sampling Plan for the Conduct of the Survey ....................................................... 17 Implement the Practice Analysis Survey ................................................................................... 19 Perform the Data Analyses, Develop Preliminary Test Specifications for the EPPP, and Discuss Implications of Findings Regarding the Assessment of Professional Competence .. 19 Complementary Data Collection Related to Competency Assessment ..................................... 20 Develop Final Recommendations Regarding the Test Specifications Underlying the EPPP and the Final Report ............................................................................................................... 21

Summary of Results for U.S and Canadian Respondents in Target Sample ................................ 22 Return Rate ................................................................................................................................ 22 Respondent Demographic and Professional Characteristics ..................................................... 25 Comparisons Between Target Sample and Samples Identified in Previous ASPPB Practice Analysis Studies and by Other Research Initiatives ............................................................... 49 Results Related to Content- and Competency-based Delineations ........................................... 54

Content-based Delineation ..................................................................................................... 54 Content areas ....................................................................................................................... 54 Knowledge in the practice of psychology ........................................................................... 62 Implications of the Ratings Related to the Content-based Delineation .............................. 74 Qualitative Results Related to Content Areas and Knowledge .......................................... 75

Content-Based Profile of Practice and Test Specifications for the EPPP .............................. 76 Profile of Practice ............................................................................................................... 76 Development of Test Specifications ................................................................................... 83

Competencies ......................................................................................................................... 91 Competency Clusters .......................................................................................................... 91 Competency Statements ...................................................................................................... 94 Competency Exemplars .................................................................................................... 105

Implications of Findings Regarding the Assessment of Professional Competence .................... 123 Results of PATF and PAAC Surveys ...................................................................................... 125 Results of Focus Panels and Interviews .................................................................................. 128

Conclusions ................................................................................................................................. 131 References and Background Literature ....................................................................................... 133

PES Final Report An Update Study of the Practice of Licensed Psychologists iv

Table of Appendices

Appendix 1 Practice Analysis Advisory Committee and their Professional Affiliations and ASPPB Personnel Monitoring Study Appendix 2 Members of the Practice Analysis Task Force and their Professional Affiliations Appendix 3 Final Report of Independent Review, Round 1 Appendix 4 Final Report of Focus Panels, Round 1 Appendix 5 Final Report of Independent Review, Round 2 Appendix 6 Materials Related to the Conduct of the Pilot Survey Appendix 7 Screen Shots of Large-Scale Validation Survey Appendix 8 Materials Related to the Conduct of the Large-Scale Validation Survey Appendix 9 Final Report of Focus Panels, Round 2 Appendix 10 Areas of Formal Post-doctoral Level Training Appendix 11 Knowledge Acquisition, Criticality, and Usage ratings of U.S. and Canadian Respondents Appendix 12 Knowledge Acquisition, Criticality, and Usage ratings of Recently and Less-recently Licensed/Registered Respondents Appendix 13 Verbatim Responses to Open-ended Question Appendix 14 Competency Frequency, Criticality, and Importance Ratings of U.S. and Canadian Respondents Appendix 15 Competency Frequency, Criticality, and Importance Ratings of Recently and Less-recently Licensed/Registered Respondents

PES Final Report An Update Study of the Practice of Licensed Psychologists v

Table of Tables Table 1 Response Rate ................................................................................................................ 22 Table 2 Responses by Survey Version ........................................................................................ 22 Table 3 Responses to Survey Version by Country ..................................................................... 23 Table 4 Respondents by Jurisdiction .......................................................................................... 23 Table 5 Respondents by Country ................................................................................................ 26 Table 6 Respondents by Years of Licensure/Registration .......................................................... 26 Table 7 Years of Licensure/Registration by Country ................................................................. 26 Table 8 Year Initially Fully Licensed/Registered to Practice Psychology by Country .............. 27 Table 9 Highest Level of Education in Psychology Beyond a Bachelor's Degree by Country 27 Table 10 Institution Where Awarded Highest Degree in Psychology by Country ..................... 28 Table 11 Highest Level of Education in Psychology Beyond a Bachelor's Degree by Years of Licensure/Registration ............................................................................................. 28 Table 12 Institution Where Awarded Highest Degree in Psychology by Years of

Licensure/Registration .................................................................................................. 29 Table 13 Major Area of Training by Country .............................................................................. 30 Table 14 Current Major Area of Practice by Country ................................................................. 31 Table 15 Participated in Formal Post-Doctoral Specialization or Re-specialization Program by Country .................................................................................................................... 32 Table 16 Major Area of Training by Years of Licensure/Registration ........................................ 32 Table 17 Current Major Area of Practice by Years of Licensure/Registration ........................... 33 Table 18 Participated in Formal Post-Doctoral Specialization or Re-specialization Program by Years of Licensure/Registration .............................................................................. 34 Table 19 Certified by ABPP by Country ..................................................................................... 35 Table 20 Certified by ABPP by Years of Experience .................................................................. 35 Table 21 Area(s) where certified by Country .............................................................................. 35 Table 22 Area(s) where certified by Years of Experience ........................................................... 36 Table 23 Employment Setting by Country .................................................................................. 37 Table 24 Health Service Provider by Country ............................................................................. 38 Table 25 Employment Setting by Years of Licensure/Registration ............................................ 39 Table 26 Health Service Provider by Years of Experience ......................................................... 40 Table 27 Primary Orientation by Country ................................................................................... 41 Table 28 Secondary Orientation by Country ............................................................................... 41 Table 29 Primary Orientation by Years of Licensure/Registration ............................................. 42 Table 30 Secondary Orientation by Years of Experience ............................................................ 42 Table 31 Area(s) in Which Respondents Have Expertise by Country ......................................... 43 Table 32 Experiences as a Psychologist Within the Past Three Years by Country ..................... 45 Table 33 Experiences as a Psychologist Within the Past Three Years by Years of Experience 46 Table 34 Sex by Country ............................................................................................................. 47 Table 35 Sex by Years of Experience .......................................................................................... 47 Table 36 Racial/Ethnic Background by Country ......................................................................... 47 Table 37 Racial/Ethnic Background by Years of Experience ..................................................... 48 Table 38 Disability (as defined by the Americans with Disabilities Act) by Country ................ 48 Table 39 Type of Disability by Country ...................................................................................... 48 Table 40 Disability by Years of Experience ................................................................................ 49

PES Final Report An Update Study of the Practice of Licensed Psychologists vi

Table 41 Institution Where Awarded Highest Degree in Psychology by Study Year and Country ......................................................................................................................... 50 Table 42 Major Area of Training by Study Year and Country .................................................... 51 Table 43 Current Major Area of Practice by Study Year and Country ....................................... 52 Table 44 Primary Theoretical Orientation by Study Year and Country ...................................... 53 Table 45 Sex by Study Year and Country ................................................................................... 53 Table 46 Racial/Ethnic Background by Study Year and Country ............................................... 54 Table 47 Knowledge Content Areas Frequency, Criticality, and Importance Ratings for Total Sample ................................................................................................................. 56 Table 48 Knowledge Content Areas— Frequency Ratings by Country and Years of Experience as Licensed/Registered Psychologist ......................................................... 59 Table 49 Knowledge Content Areas Criticality Ratings by Country and Years of Experience as Licensed/Registered Psychologist ......................................................... 60 Table 50 Knowledge Content Areas Importance Ratings by Country and Years of Experience as Licensed/Registered Psychologist ......................................................... 61 Table 51 Frequency Distribution for Acquisition, Criticality, and Usage Ratings, and Mean and Standard Deviation for Criticality Ratings for Total Sample of Respondents ...... 64 Table 52 Mean of Means for Criticality of Knowledge Statements ............................................ 74 Table 53 Empirically Derived Test Specifications for Content Areas ........................................ 78 Table 54 Knowledge Statements—Mean Weightings ................................................................. 78 Table 55 Recommended Test Specifications for Content Areas ................................................. 86 Table 56 Final Knowledge Statements and Ranks for EPPP ....................................................... 87 Table 57 Competency Clusters Frequency, Criticality, and Importance Ratings for Total Sample .......................................................................................................................... 93 Table 58 Mean of Means for Competency Statements on the Frequency, Criticality, and

Importance Rating Scales ............................................................................................. 94 Table 59 Frequency Distribution, Number, Mean, and Standard Deviation for Frequency

Ratings for Competency Statements ............................................................................ 95 Table 60 Frequency Distribution, Number, Mean, and Standard Deviation for Criticality

Ratings for Competency Statements ............................................................................ 98 Table 61 Frequency Distribution, Number, Mean, and Standard Deviation for Importance

Ratings for Competency Statements .......................................................................... 101 Table 62 Support for Acquisition of Behavioral Exemplars ...................................................... 107 Table 63 Methods for Enhancing the EPPP by Competency Cluster ........................................ 126 Table 64 Methods for Assessing Competency by Cluster (excluding Scientific Knowledge) .. 127

PES Final Report An Update Study of the Practice of Licensed Psychologists vii

Table of Exhibits

Exhibit 1 Recommended Test Specifications for Content Areas ................................................... 3 Exhibit 2 Content- and Competency-Based Delineation of the Practice of Psychology ............. 12 Exhibit 3 Validation Survey Content Outline .............................................................................. 14 Exhibit 4 Knowledge Statements Not Meeting Criterion for the Criticality Rating Scale .......... 62 Exhibit 5 Weighting of Knowledge Statements ........................................................................... 77 Exhibit 6 16 Modified Knowledge Statements ............................................................................ 84

PES Final Report An Update Study of the Practice of Licensed Psychologists 1

Executive Summary Study of the Practice of Licensed Psychologists in the United States and Canada

Sandra Greenberg, Ph.D.

Carla M. Caro, M.A. I. Leon Smith, Ph.D.

Professional Examination Service

The Association of State and Provincial Psychology Boards sponsored a study of the practice of licensed psychologists in the United States and Canada. The underlying conceptual charge was:

• The identification and validation of underlying professional competencies (including but not limited to those related to professional knowledge)

• The identification of assessment methods to best measure underlying professional competencies (including but not limited to those related to professional knowledge)

• Revised test specifications for the EPPP updating the knowledge base and integrating additional relevant competencies

Professional Examination Service implemented the study under the direction of a Practice Analysis Advisory Committee and in conjunction with a Practice Analysis Task Force. The study consisted of two partially concurrent, partially sequential explorations. As in previous practice analyses, the first exploration examined the knowledge required for psychology practice with the goal of updating the EPPP test specifications, and retained a content-based organizational structure including eight content areas comprised of knowledge statements. The second exploration examined the competencies underlying the practice of psychology. Accordingly, a competency-based framework was developed and validated, including the delineation of six competency clusters, associated competencies, and behavioral exemplars typifying the development of competence. A survey was developed and sent to approximately 5000 licensed psychologists in the United States and Canada in order to validate and update all elements in the comprehensive framework, including the content areas and knowledge statements, and the competency clusters, competencies, and exemplars. The return rate was 26%. Analysis of completed surveys produced information about (a) the demographic and professional background of licensed psychologists, (b) the critical knowledge licensed psychologists use, (c) comments about changes occurring in the profession, (d) the competencies required in professional practice, and (e) the validation of specific competencies and behavioral exemplars. Results related to the first exploration were used to review and refine the test specifications to ensure that the knowledge assessed in the EPPP is required for the performance of critical behaviors and serves the public protection function of regulation. Results related to the second exploration were used to develop and validate a conceptual framework for the assessment of competence at various stages in professional development. Various types of assessments were identified as useful for the assessment of competence. Preliminary discussions focused both on alternate question types that might be integrated into the EPPP, and the development of complementary assessment that might


be integrated into the assessment of licensed/registered either before, during, or after initial licensure/registration.

Key Findings and Conclusions • Regardless of country, respondents were more likely to have been trained in the major areas

of clinical, counseling, and educational psychology than they were to be currently practicing in those major areas; and were more likely to be currently practicing in the major areas of clinical neuropsychology, forensic psychology, geropsychology, health psychology, and rehabilitation psychology than to have initially been trained in those major areas.

• In the U.S., more than one-half of the respondents indicated cognitive/behavioral psychology (58%), and 9% and 11% indicated interpersonal psychology and psychodynamic psychology as their primary orientation, respectively. No more than 6% of the U.S. respondents indicated any of the other four specifically-delineated theoretical orientations as primary. In Canada, about two-thirds of the respondents indicated cognitive/behavioral psychology (66%), and 8% indicated interpersonal psychology as their primary orientation. No more than 4% of the Canadian respondents indicated any of the other specifically-delineated orientations as primary.

• Regardless of country, respondents were most likely to describe themselves as being experts in clinical psychology and in assessment/diagnosis/evaluation, and somewhat less likely to indicate clinical child psychology; counseling psychology; and treatment, intervention, and prevention. With very few exceptions, respondents indicated that they had expertise in one or more of each of 57 specifically-delineated areas of expertise.

• Nearly one-third of the U.S. respondents have participated in formal post-doctoral specialization and/or respecialization and 17% of Canadian respondents have done so as well. Recently licensed respondents are more likely to have trained in clinical, clinical child, and school psychology, and less likely to have trained in community, counseling, developmental, educational, experimental, industrial/organizational, and social psychology than less recently licensed/registered respondents.

• Regardless of country, the majority of respondents are female (62% and 70%, respectively),

and the recently licensed/registered respondents are more likely to be female than less-recently licensed/registered respondents (72% and 58%).

• The eight content areas and 77 associated knowledge statements were validated as an

organizing vehicle for the development of the EPPP. Exhibit 1 documents the recommended test specifications for the EPPP.


Exhibit 1 Recommended Test Specifications for Content Areas

% of exam

Biological Bases of Behavior –– knowledge of (a) biological and neural bases of behavior, (b) psychopharmacology, and (c) methodologies supporting this body of knowledge

12%

Cognitive-Affective Bases of Behavior— knowledge of (a) cognition, (b) theories and empirical bases of learning, memory, motivation, affect, emotion, and executive function, and (c) factors that influence cognitive performance and/or emotional experience and their interaction

13%

Social and Cultural Bases of Behavior — knowledge of (a) interpersonal, intrapersonal, intergroup, and intragroup processes and dynamics, (b) theories of personality, and (c) diversity issues

12%

Growth and Lifespan Development — knowledge of (a) development across the full life span, (b) atypical patterns of development, and (c) the protective and risk factors that influence developmental trajectories of individuals

12%

Assessment and Diagnosis — knowledge of (a) psychometrics, (b) assessment models and instruments, (c) assessment methods for initial status of and change by individuals, couples, families, groups, and organizations/systems, and (d) diagnostic classification systems and their limitations

14%

Treatment, Intervention, Prevention, and Supervision — knowledge of (a) individual, couple, family, group, organizational, or community interventions for specific problems/disorders in diverse populations, (b) intervention and prevention theories, (c) best practices and practice guidelines, (d) consultation and supervision models, and (e) evidence supporting efficacy and effectiveness of interventions

14%

Research Methods and Statistics — knowledge of (a) research design, methodology, and program evaluation, (b) instrument selection and validation, (c) statistical models, assumptions, and procedures, and (d) dissemination methods

8%

Ethical/Legal/Professional Issues — knowledge of (a) codes of ethics, (b) professional standards for practice, (c) legal mandates and restrictions, (d) guidelines for ethical decision-making, and (e) professional training and supervision

15%

• The competency-based model, including six competency clusters (Scientific Knowledge,

Evidence-Based Decision Making/Critical Reasoning, Interpersonal and Multicultural Competence, Professionalism/Ethics, Assessment, and Intervention/Supervision/Consultation) was validated via the large-scale survey of practice.

• The competency clusters each represent competencies which are performed frequently-to-

very frequently, are moderately -to-highly critical to optimizing outcomes for patient/client/public, and are moderately-to-very important to the practice of the respondents.


• The 37 competencies were generally validated and are performed frequently-to-very frequently, are moderately-to-highly critical to optimizing outcomes for patient/client/public, and are moderately-to-very important to the practice of the respondents.

• A detailed review of the results for the 277 behavioral exemplars associated with the competencies indicates a general level of support for the developmental unfolding of the validated competencies as operationalized by the exemplars.

• Methodologies for assessing competence were evaluated.

• The feasibility of enhancing the EPPP with alternate item types was explored as one way of augmenting the current licensure/registration process.

• The potential for developing new assessments to complement the EPPP was preliminarily

discussed as was the use of such assessments at various points of time pre- and post-licensure/registration.

• Discussions amongst the members of the PATF and the PAAC confirmed the utility of the

EPPP as an effective tool for the assessment of the Scientific Knowledge base underlying the practice of psychology. Recommendations for a revised set of test specifications for the EPPP were approved. Future directions in regard to alternate item types that might be built into this computer-delivered examination were discussed.

• Discussions amongst the members of the PATF and the PAAC as well as key stakeholders

involved in the assessment of competency in students, interns, and practica participants, and jurisdictional regulation confirmed interest in the assessment of competency via complementary assessments that might be integrated into pre- and post-licensure/registration activities and/or licensure/registration requirements.

• A substantial investment in time and resources will be required should ASPPB choose to

support the development of entry-level competency assessments for licensed/registered psychologists. The widespread dissemination of this technical report, including information about the development of validated competencies to be acquired pre- and post licensure/registration, should be a priority and might easily be accomplished via the ASPPB’s website.

• ASPPB needs to consider the leadership role it might play in the future development and

support of competency assessment. To that end, a meeting of key stakeholders including other organizations within the profession, may be desirable in order to sort through the issues and develop a game plan for moving the competency assessment agenda forward across the professional development continuum. Among numerous resources available for the discussion are the following:

o The results of the two ASPPB Foundation grant-supported assessment initiatives focusing on the development of competency assessment pre-licensure, as well as numerous other research-based assessment development initiatives occurring within the professional education and training community;


o The recently published work of the Competency Assessment Workgroup, including the portfolio of assessment tools known as the Competency Assessment Toolkit for Professional Psychology (Toolkit); and, now,

o The results of the ASPPB-sponsored practice analysis study in the U.S. and Canada, including not only the validation of the Scientific Knowledge competency cluster, but the validation of five other competency clusters, each including behavioral exemplars representing the evolution of the validated competencies which develop across key milestones.


Introduction The Association of State and Provincial Psychology Boards (ASPPB) sponsored an update study of the practice of licensed psychologists in the United States and Canada to review and refine the test specifications for the Examination for Professional Practice in Psychology (EPPP). This study utilized a content-based approach to refine the delineation of the content areas and knowledge required in practice and to identify the competencies performed by psychologists. Additionally, the study was designed to build on competency frameworks being explored by other key stakeholder groups in the profession. The results are expected to provide ASPPB with the empirical support necessary for considering changes in the examination program for licensure, including augmentation of the EPPP and/or the development of complementary assessments to the EPPP. The specific purposes of the update study were to:

• refine the knowledge required for professional practice, • validate the test specifications underlying the construction of the EPPP, • develop a comprehensive competency-based model of professional practice, • develop testable behavioral exemplars that operationalize competency at pre-determined

developmental anchor points of professional life, including at entry to practice, and • evaluate methodologies for assessing competence and explore the feasibility of

augmenting the licensure/registration process by enhancing the EPPP with alternate item types, and/or developing new competency-based assessments to complement the EPPP.

The overall process of practice analysis is described in the 1999 revision of the Standards for Educational and Psychological Testing (American Educational Research Association, the National Council on Measurement in Education, and the American Psychological Association). This document emphasizes the concept of content validity and the need to conduct a practice analysis to assure that the professional knowledge/skills/abilities assessed in credentialing initiatives are in fact limited to those required for competent performance and serve a public protection function. Practice analysis, then, becomes the primary basis by which a professional association or credentialing agency establishes and defends the content validity of its credentialing initiatives.


Conduct of the Update Practice Analysis Study of Licensed Psychologists This section contains a description of the procedures Professional Examination Service (PES) implemented to conduct an update practice analysis study of psychology on behalf of the ASPPB. PES conducted the update practice analysis study in 2007 – 2009 in conjunction with a four-member Practice Analysis Advisory Committee (PAAC) and a 12-member Practice Analysis Task Force (PATF). The members of the PAAC represent key decision makers in ASPPB; in part, they are responsible for guiding aspects of the licensure/registration program, including examination development and validation efforts. The members of the PATF represent academics, training programs, and practitioners in different major areas of practice, including new and emerging specialties, and with expertise in competency assessment. The responsibilities of the PAAC were to (a) identify key parameters and select subject-matter experts to serve on the PATF; (b) facilitate the conduct of focus panels and complementary data collection initiatives; (c) review and approve the form and content of the survey, data collection procedures, data analyses, and results; (d) review the draft final report for presentation to ASPPB’s Board of Directors for final approval; and (e) develop recommendations regarding the test specifications for the EPPP and other assessment initiatives. Appendix 1 lists the members of the PAAC and their professional work affiliations. The Chair of the PAAC served as the liaison to the PATF and had the responsibility of maintaining communications between the PATF and the PAAC. The Chair of the PAAC also had the responsibility of maintaining communications between the ASPPB Board of Directors and the PAAC. One member of the PAAC had the responsibility of maintaining communications between the Exam Committee of ASPPB and the PAAC. In addition to the Chair, one member of the PAAC attended each of the first three Task Force meetings; and all PAAC members attended the final Task Force meeting. The PAAC developed a conceptual charge for the study that included the following goals:

• The identification and validation of underlying professional competencies (including but not limited to those related to professional knowledge).

• The identification of assessment methods to best measure underlying professional competencies (including but not limited to those related to professional knowledge).

• Revision of test specifications for the EPPP updating the knowledge base and integrating additional relevant competencies.

The responsibilities of the PATF were to (a) develop the draft update and final delineations of practice including both content areas and knowledge statements as well as competency clusters, competencies, and behavioral exemplars; (b) select subject-matter experts to participate in complementary data collections procedures; (c) discuss and review the form and content of a Web-based survey of practice; and (d) review and discuss the form and content of the data analyses and results. The PATF met four times and conducted additional business by teleconference and e-mail between 2008 and 2009.


Select the Subject-Matter Experts to Participate in the PATF At a 1-day meeting in November 2007, members of the PAAC developed key parameters for identifying potential members of the PATF. These parameters included expertise in competency assessment and the training of psychology students and psychologists; familiarity with the work of the APA Assessment of Competency Benchmarks Workgroup (2006); experience on the EPPP exam committee; representation from varied professional practice environments, including private practice, consultants, and academics; expertise in a variety of subject-matter areas, including new and emerging areas of practice; and representation of a range of demographic characteristics. Key individuals who members of the PAAC wished to recruit were also identified and discussed. Subsequent to the meeting, calls went out to a variety of organizations and individuals, requesting assistance with outreach effort seeking nominees for the PATF who met the criteria. These organizations included the ASPPB Board of Directors, the Association of Canadian Psychology Regulatory Organizations (ACPRO), the Canadian Council of Professional Psychology Programs (CCPPP), the Council of Chairs of Training Councils (CCTC), and various other professional associations representing psychologists, such as the Association of Black Psychologists and the Asian-American Psychological Association. An online nomination form was used to collect professional and demographic information on the nominees, and nominees were requested to submit their current curriculum vitae. Nominees were also asked if they would be willing to fill other study roles as independent reviewers or pilot test participants if they were not selected to serve on the PATF. In January 2008, PES conducted a series of telephone conference calls with the PAAC wherein they discussed the intended composition of the PATF, reviewed the qualifications and areas of expertise of each nominee, and voted for a final slate of participants. A listing of the PATF members, their professional affiliations, and jurisdictional location may be found in Appendix 2. In advance of the first PATF meeting, Task Force members were sent a packet of information describing the study goals, including a description of the EPPP content areas and knowledge, as well as the study’s additional focus on competency assessment. Task Force members were also provided background literature on competency assessment, including concurrent APA work on competency benchmarks; and articles on competency assessment including studies of competency conducted for other professions, to read in advance of the first PATF meeting. (Note: A list of the articles provided to the members of the Task Force prior to the meeting and throughout the study may be found within the references of this report.)

Develop the Draft and Final Delineations of Content Areas and Knowledge Statements; and Competency Clusters and Competencies, and Behavioral Exemplars

In 1995, process- and content-based approaches were implemented by PES to study the practice of licensed psychologists in the United States and Canada (Greenberg, Smith, & Muenzen, 1996). In 2002 to 2003, the process- and content-based descriptions of practice were updated. (Greenberg & Jesuitus, 2003). The process- and content-based approaches used in those studies provided a structure for describing contemporary practice—that is, for identifying what


psychologists do—and facilitated the development of examination items in a practice-related framework. The process-based approach comprised the delineation of roles and associated responsibilities performed by psychologists. Roles represented major categories of activities, and responsibilities represented specific activities psychologists performed within each role. Unique sets of responsibilities were associated with each role. The content-based delineation comprised the content areas and the knowledge statements required to perform the entire set of responsibilities. Content areas represented categories of knowledge used by psychologists in practice, and knowledge statements related to the content areas; they described an organized body of information needed to perform responsibilities. The current study consisted of two partially concurrent, partially sequential explorations. As in previous practice analyses, the first exploration examined the knowledge required for psychology practice with the goal of updating the EPPP test specifications, and retained a content-based organizational structure including content areas comprised of knowledge statements. In addition, the study built on current initiatives in credentialing, in general, and psychology, in specific, by incorporating related and ongoing work currently being supported by key stakeholder groups in the profession relative to competency assessment. For example, the APA Task Force on the Assessment of Competence in Professional Psychology (2006) has stated:

. . . it is timely for professional psychology to embrace not only a culture of competence, but also a culture of the assessment of competence (Roberts, Borden, & Christiansen, 2005) that builds upon a long history of the assessment of competence both within and outside of psychology. . . (T)he assessment of competence fosters learning, evaluates progress, assists in determining the effectiveness of the curriculum and training program, advances the field, and protects the public (Kaslow, 2004).

To that end, the second exploration examined the competencies underlying the practice of psychology. Accordingly, the PATF developed a competency-based framework included the delineation of competency clusters, associated competencies, and behavioral exemplars typifying the development of competence. A range of complementary data collection efforts, including independent reviews, in-person and web-based focus panels, and supplementary surveys, supported the development and validation of the delineations of content areas and knowledge; and competency clusters, competencies, and behavioral exemplars. The results of this study, including a large-scale survey of practice and other complementary data collection efforts, provided empirical support for updating the EPPP test specifications, as well as beginning to identify ways in which the existing computer-based examination could be augmented with the addition of enhanced question types. The parallel focus of the study on competencies underlying professional practice laid the initial groundwork for the identification of essential competencies and the further exploration of the development of complementary assessments to best measure professional competencies. Accordingly, the study included an initial exploration of the feasibility of incorporating such competency assessments into both training programs (pre-licensure) and the licensure/registration processes.


The following sequence of activities provided the basis for the development and validation of the delineations: PATF Meeting 1 The PATF first met in April 2008 for an initial review and revision of the existing content-based delineation including content areas and knowledge statements, and to begin consideration of how to build a model of psychology practice that includes professional competence. The Task Force was introduced to the study’s parallel exploration of the development of complementary competency assessment methodologies that might be implemented in conjunction with the EPPP, and discussed the potential impact of such assessments to graduate training and professional development. To this end, the PATF referred to the ongoing work of the APA Competency Benchmarks Workgroup, and examined related literature and models (for example, the “Cube” model, Rodolfa, Bent & Eisman, et al., 2005), and reviewed methods of assessing competency in a number of different professions. The Task Force proposed an initial set of revisions to the content-based delineation, including the eight content areas, definitions for each area, and knowledge statements. They also began to develop a competency-based model of practice consisting of six clusters of competence. The first cluster encompassed the entire knowledge content currently tested by the EPPP and was designated as the Scientific Knowledge Base. Five additional proposed clusters consisted of three foundational clusters (Evidence-Based Decision Making/Critical Reasoning, Interpersonal and Cultural Competence, and Professionalism/Ethics) and two functional clusters (Assessment and Intervention/Consultation). The Task Force also identified four levels of professional development at which specific behaviors demonstrating competency could be demonstrated. These levels were designed to complement the work of the APA Assessment of Competency Benchmarks Workgroup, and included:

Pre-licensure (prior to independent practice) Level 1: Entry-level supervised practice (during training) Level 2: Advanced supervised practice (internship level) Post-licensure (independent practice) Level 3: Entry-level independent practice (just licensed) Level 4: Advanced independent practice

Following the meeting, PATF members completed an e-mail-based critical review of their work products. PES then convened a series of conference calls with Task Force subcommittees to reconcile the suggestions about both the updated delineation of content areas and knowledge statements, and the newly developed model of competency clusters and competencies. The products of those conference telephone calls were then disseminated for review by independent subject matter experts (SMEs). Independent Review, Round 1 In July of 2008, the first round of complementary data collection was undertaken to supplement the work of the PATF. This consisted of a review of the content-based delineation and competency model by independent SMEs, and was designed to ensure the completeness and


clarity of both components. The SMEs were identified through the Task Force nomination process and were invited to participate by Task Force members. Two separate panels were created to critically review either the content-based delineation (the Scientific Knowledge Base) or the draft competency model, comprising the remaining five competency clusters. Extensive suggestions were submitted by the reviewers and were collated by PES. Then, by way of a PES-facilitated series of subcommittee teleconferences, the Task Force reconciled the suggestions and produced the updated drafts of the content-based delineation and the competency model. A complete report of the Independent Review, Round 1 process and results may be found in Appendix 3. Focus Panels, Round 1 Two rounds of focus panels were conducted as part of this study. The first round consisted of three focus panels held in conjunction with the American Psychological Association Annual Meeting held in Boston, MA in August 2008, and a single panel conducted at the Annual Meeting of the Ontario College of Psychologists in Toronto, ON in September 2008. The goal of these focus panels facilitated by PES was two-fold: to critically examine the proposed competency-based model of psychology practice, including the clusters and competencies, and to begin developing behavioral exemplars of the competencies at the four professional levels identified by the Task Force. A total of 23 SMEs participated in these panels. A complete report of the focus panel process and results, including the protocol used and a demographic description of the participants, may be found in Appendix 4. PATF Meeting 2 In November, 2008 the PATF convened its second 3-day meeting. At that meeting, Task Force members reviewed the results of the independent review and focus panels. They incorporated the findings of these complementary data collection efforts as they continued to refine the content- and competency-based models. Task Force members participated in small group work to revise the content areas and knowledge statements and the competency clusters and competencies. After de-briefing, additional small group work involved the development of behavioral exemplars demonstrating each competency at the four levels of professional development previously identified. Finally, Task Force members were asked to nominate colleagues to participate in the second round of independent review, and in particular, to identify SMEs with expertise in the areas of biological psychology and industrial/organizational psychology— areas which were identified as needing augmentation. As in the first meeting, PATF members engaged in a post-meeting e-mail-based review of the iteration of the model produced at the meeting, and participated subcommittee teleconferences to reconcile their edits, suggestions, and comments submitted during the review process. The products of those conference telephone calls were then prepared for review by independent SMEs. Independent Review, Round 2 The revised content- and competency-based model was sent to a second panel of independent experts for outside review and critique. Extensive suggestions were submitted by the reviewers and were collated by PES. Subsequently, PES facilitated a series of teleconferences with Task Force members organized into subcommittees, reconciled the suggestions, and produced the next draft of the model. A complete report of the Round 2 Independent Review process and results may be found in Appendix 5.


Exhibit 2 presents an overview of the work product developed through these processes. The content-based delineation consists of eight content areas comprised of 78 knowledge statements. The competency based delineation consists of a model containing six competency clusters. The first of these is the Scientific Knowledge Base which is described within the content-based delineation. The model posits three foundational, cross-cutting competency clusters (Evidence-Based Decision Making/Critical Reasoning, Cultural and Interpersonal Competence, and Professionalism/Ethics) comprised of 19 competencies, and two functional competency clusters (Assessment and Intervention/Supervision/Consultation) comprised of 18 competencies. A total of 277 behavioral exemplars that demonstrate competence at four levels of professional development were developed in association with the 37 competencies.

Exhibit 2 Content- and Competency-Based Delineation of the Practice of Psychology

Content Areas in the Scientific Knowledge Base Number of Knowledge Statements

Biological Bases of Behavior 6

Cognitive-Affective Bases of Behavior 7

Social and Multicultural Bases of Behavior 14

Growth and Lifespan Development 9

Assessment and Diagnosis 12

Treatment, Intervention, and Prevention 12

Research Methods and Statistics 8

Ethical/Legal/Professional Issues 10

Total 78

Competency Clusters Number of Competencies

Number of Behavioral Exemplars

Scientific Knowledge Base n/a n/a

Evidence-Based Decision Making/Critical Reasoning 5 34

Cultural and Interpersonal Competence 4 39 Professionalism/Ethics 10 74 Assessment 10 63 Intervention/Supervision/ Consultation 8 67 Total 37 277


Develop, Review, and Approve the Form and Content of the Survey of the Practice of Psychology

The study’s exploration of three aspects of practice (the knowledge base needed for practice, the professional competencies of psychologists, and the behavioral exemplars by which these competencies are demonstrated) necessitated developing different versions of the survey due to the time required to evaluate each aspect. Additionally, since the content- and competency-based sections of the survey were designed to elicit ratings on three different scales, these versions were further subdivided, with each version asking respondents to rate the knowledge or competency on two of the three possible ratings scales. Finally, although the version containing the behavioral exemplars included only one rating scale, it contained a large number of items to be rated; accordingly it was divided into two subsets of the exemplars. Exhibit 3 presents the numbers of ratable items and the rating scales employed for each of the six resulting survey versions. In addition to these scales, all versions of the survey included a section containing 19 questions about the respondents’ professional and demographic background, and a qualitative question regarding expected changes in the profession. Descriptions of the specific rating scales follow the survey content outline.


Exhibit 3 Validation Survey Content Outline

Version

A B C D E F

Versions A & B

Section 1: Knowledge Statements – 78 statements • Acquisition–4 point scale ! ! • Usage–3 point scale ! • Criticality–4 point scale !

# ratings – subtotal 156 156 Section 2: Knowledge Content Areas – 8 content areas • Frequency–4 point scale ! ! • Criticality–4 point scale ! • Importance–4-point scale !

# ratings – subtotal 16 16 Knowledge missing from survey 1 1

Total ratings: Versions A & B 173 173

Versions C & D

Section 1: Competencies – 37 competencies • Frequency–4 point scale ! ! • Criticality–4 point scale ! • Importance–4 point scale !

# ratings – subtotal 74 74 Section 2: Competency Clusters – 6 clusters • Frequency–4 point scale ! ! • Criticality–4 point scale ! ! • Importance–4 point scale ! !

# ratings – subtotal 18 18 Total ratings: Versions C & D 92 92

VERSIONS E & F

Behavioral Exemplars – 277 exemplars • Acquisition–4 point scale

o Subset of exemplars (Clusters 02. 03, 06) ! o Subset of exemplars (Clusters 04, 05) !

Total ratings Versions: E & F 140 137

ALL VERSIONS

• Predicted changes in the profession in next 5 years • Professional and Demographic Background 19 19 19 19 19 19


VERSIONS A & B––Knowledge Statements and Content Areas Section 1. Knowledge Statements (78 statements) Respondents rated each statement on two of the three scales (Acquisition plus either Criticality or Usage).

Acquisition–At what point should the knowledge be acquired by psychologists? (4 point scale: Not necessary at any point; Primarily before initial licensure/registration; Primarily after initial licensure/registration, but before specialization; Primarily after specialization) Criticality–How critical is possessing the knowledge to protecting the patient/client/public from harm? (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Usage–What level best represents your use of this knowledge in your practice? (3 point scale: I do not have the knowledge; I recognize/recall the knowledge; I apply/interpret/integrate the knowledge)

Section 2. Knowledge Areas (8 content areas with definitions) Respondents rated each content area on two of the three scales (Frequency plus either Criticality or Importance).

Frequency–How frequently did you call upon knowledge from the content area in your practice during the past year? (4 point scale: Never or very rarely; Infrequently; Frequently; Very frequently) Criticality–How critical is the knowledge in this content area to protecting the patient/client/public from harm? (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Importance–How important is the knowledge in the content area to your practice as a psychologist during the past year? (4 point scale: Not important; Minimally important; Moderately important; Very important)

All respondents completing the two Knowledge versions were asked to describe any knowledge they call upon in practice that was not included in the survey. VERSIONS C & D––Competencies and Competency Clusters Section 1: Competencies (37 competencies) Respondents rated each competency on two of the three scales (Frequency plus either Criticality or Importance).


Frequency–How frequently did you perform the competency in your practice during the past year? (4 point scale: Never or very rarely; Infrequently; Frequently; Very frequently) Criticality–How critical is the competency to optimizing outcomes for the patient/client/public (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Importance–How important was performance of the competency to your practice as a psychologist during the past year? (4 point scale: Not important; Minimally important; Moderately important; Very important)

Section 2: Competency Clusters (6 clusters with definitions) Respondents rated all 6 clusters on all 3 scales.

Frequency–How frequently did you perform the competencies in this cluster in your practice during the past year? (4 point scale: Never or very rarely; Infrequently; Frequently; Very frequently) Criticality–How critical are the competencies in this cluster to optimizing outcomes for the patient/client/public? (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Importance–How important was performance of the competencies in this cluster to your practice during the past year? (4 point scale: Not important; Minimally important; Moderately important; Very important)

VERSION E—Behavioral Exemplars in Competency Clusters 02, 03, 06 (140 exemplars) VERSION F––Behavioral Exemplars in Competency Clusters 04, 05 (137 exemplars)

Section 1. Behavioral Exemplars Respondents rate a subset of the 277 exemplars on the following scale:

Acquisition–At what point should a psychologist be able to demonstrate this behavior? (4 point scale: During entry-level supervised practice (practicum); During advanced-level supervised practice (internship); At entry to practice through 3 years of independent practice; After 3 years of independent practice)

ALL VERSIONS––Demographic and Professional Information and Open-Ended Questionnaire Respondents answered 19 questions regarding their background, education, and practice. Respondents were also asked to comment upon what they believe to be the long-range changes occurring in the practice of psychology.


Pilot Test of Online Survey Instrument A draft Web-based Survey of the Practice of Psychology – Pilot Version, (referred to as the Pilot Survey) was developed to study the practice of licensed psychologists in the United States and Canada to explore the professional competencies and behaviors of psychologists and validate the content areas and knowledge required for practice. In Spring 2009, PES implemented the test of the Pilot Survey to identify question defects such as ambiguous items and to identify respondents’ cognitive difficulties as they formed answers to Pilot Survey questions. Members of the PATF were invited to nominate licensed psychologists to participate in the online pilot test. The members were asked to identify individuals in a variety of practice areas and employment settings, and to personally invite the nominees to participate in the pilot test. As had been the case with each of the previously described complementary data collection methods, the final pool of participants also included some psychologists nominated for, but not appointed to, the PATF. Finally, the PAAC and the PATF were invited to participate in the pilot test. Participants in the online pilot test were requested to complete the Pilot Survey and to critique each section to identify any unclear elements. As a check on the length of time needed to complete the Pilot Survey, participants were asked to record how long it took them to complete the survey. Appendix 6 contains a copy of the instructions provided to the participants in the mail-based pilot test of the Pilot Survey and a list of the participants. Based on the results of the Pilot Test and its own review, the PAAC approved the form and content of the final version of the online survey materials. See Appendix 7 for screen captures of the Survey of the Practice of Psychology (referred to as the Survey)—used in the large-scale validation of the practice analysis of licensed psychologists in the United States and Canada.

Develop the Sampling Plan for the Conduct of the Survey

PES prepared a sampling plan for the conduct of a large-scale validation survey of the practice of licensed psychologists in the United States and Canada. During the Spring of 2009, the PAAC reviewed and approved the general strategy, and PES implemented the plan to develop the sample. The sampling plan for dissemination of the survey was designed to (a) generate a sample of licensed psychologists from all ASPPB member jurisdictions in the United States and Canada, and (b) ensure the representation of licensed psychologists at or near the entry level of the profession. PES implemented the plan based on the following specific information elements:

• The target sample size was approximately 5,000 licensed, practicing psychologists.

• There was no comprehensive database of all licensed psychologists in either the United States or Canada. An e-mail letter outlining the study goals and purposes and signed by the ASPPB president was sent to the jurisdictional administrators of each of the 64 ASPPB member jurisdictions with a request for a sample of psychologists.


• One hundred and twenty psychologists were requested for the sample from the top third

most populated jurisdictions in the U.S. and Canada. Eighty psychologists were requested for the sample from the middle third most populated jurisdictions, and 40 psychologists were requested to be sampled from the smallest third of the jurisdictions.

• The sample should ensure equal representation of recently licensed psychologists (i.e.,

licensed in 2006 or after) and less-recently licensed psychologists (i.e., licensed in 2005 or earlier). Where possible, the ASPPB-member boards were asked to sample equal numbers of recently and less-recently licensed psychologists, divided between those licensed or registered within the past three years and those licensed or registered more than three years ago.

The results of the request for the samples of licensed/registered psychologists were as follows:

• ASPPB-member jurisdictions in the United States and Canada varied considerably in their ability to select a specific sample of licensed practicing psychologists from among the population of all licensed psychologists in the jurisdiction. Of those providing information, most jurisdictions were able to sample psychologists or supply databases sorted by year of licensure and practice status (active/inactive).

• In 11 cases, the jurisdiction was able to provide a sample or database that included e-mail addresses. In nine cases, the jurisdiction was able to provide e-mail addresses for some members of the sample, and mailing addresses for other members of the sample. In 38 cases, the jurisdictions were able or willing to supply mailing addresses for the sample.

• Due to privacy concerns and regulations governing the dissemination of members’ contact information, one jurisdiction drew the sample and disseminated the invitation using its own e-mail database. Another jurisdiction sent out e-mails to members directing them to an online sign-up form on which they could register for the study.

• One jurisdiction declined to participate entirely.

• Three jurisdictions did not respond in any way to repeated attempts to contact them.

• In some cases, the jurisdictions drew the requested sample for PES. In other instances, the jurisdictions provided their entire databases and PES drew the sample.

Because some of the smaller jurisdictions were unable to supply the minimum number of names and because some jurisdictions either declined to participate or failed to respond to requests to participate, the final sample for the survey consisted of 4,732 licensed/registered psychologists.


Implement the Practice Analysis Survey PES prepared a plan for the conduct of the large-scale survey of the practice of licensed psychologists in the United States and Canada. During 2008 through the Spring of 2009, the PAAC reviewed and approved the elements of the plan. Following the completion of the Pilot Survey, the development and creation of the online Survey and all related materials, and the implementation of all procedures required in connection with assembling the sample, PES implemented the plan. Appendix 8 contains copies of all the materials used in the Survey invitation mailings. The delivery method of the survey invitation depended on whether PES had received mailing addresses or e-mails from jurisdictional administrators. The survey was disseminated as follows:

1. In June 2009, potential respondents receiving the survey via U.S. Mail or Canadian Postal Service delivery received a letter from the President of the ASPPB describing the nature and scope of the project and inviting their participation in the data collection efforts. The invitation contained a personalized, password-protected link to the online survey.

2. In June 2009, potential respondents for whom PES had e-mail addresses received an e-

mail invitation from the President of the ASPPB describing the nature and scope of the project and inviting their participation in the data collection efforts. The invitation contained an embedded password-protected link to the online survey.

3. Approximately two weeks later, each potential member of the sample who had not yet

completed the survey received a reminder letter or e-mail invitation with the same information and link.

4. Finally, approximately one week later, each non-respondent was sent a final reminder

invitation to the survey.

Perform the Data Analyses, Develop Preliminary Test Specifications for the EPPP, and Discuss Implications of Findings Regarding the Assessment of Professional Competence

In Summer 2009, PES prepared preliminary descriptive statistical analyses on the basis of the demographic and professional data supplied by the respondents. The results of the analyses were presented at a meeting of the PATF held in July 2009. At that meeting, and following a review of the demographic and professional characteristics of the respondents to the survey, the PATF recommended the implementation of additional outreach efforts to Canadian psychologists so as to increase their representation in the sample. Subsequent to the meeting, members of the PAAC initiated renewed outreach efforts to those jurisdictions not previously participating in the survey and implemented a second round of surveys to a limited number of Canadian jurisdictions. As a result of these efforts, an additional 33 Canadian respondents were included in the survey In addition, the PATF suggested that ASPPB engage in the conduct of two supplemental outreach efforts to specific populations of psychologists—industrial/organizational (I/O) psychologists and psychologists with expertise in the assessment of competency, particularly competency of entry-level psychologists. ASPPB authorized PES to conduct two additional data


collection initiatives with two newly developed samples. The survey of I/O psychologists was accomplished with the assistance of the Society for Industrial and Organizational Psychologists (SIOP), which provided e-mail addresses for its voting members. The survey of educators was accomplished with the assistance of the Association of Psychology Postdoctoral and Internship Programs (APPIC), which provided e-mail addresses for three cohorts: subscribers (Directors of academic programs); Internship Training Directors; and Post-Doctoral Training Directors. Members of the PAAC and the PATF were especially helpful in establishing these liaisons between PES and these organizations. The survey of I/O psychologists, the results of which will be presented in detail in a subsequent report, included 157 respondents. They responded to all six survey versions –– two rating knowledge content and statements, two rating clusters and competencies, and two rating behavioral exemplars. The survey of APPIC respondents, the results of which will also be presented in detail in a subsequent report, included 37 directors of academic programs, 15 post-doctoral training directors, and 105 internship directors. Because this supplementary sample was expected to have expertise in competency assessment and the acquisition of demonstrated professional competency, respondents completed either one of the two competency versions or one of the two behavioral exemplars versions of the survey.

Complementary Data Collection Related to Competency Assessment

Following the conduct of PATF Meeting 3 and prior to the fourth and final meeting of the Task Force, PES implemented two final complementary data collection initiatives (i.e., focus panels and surveys) related to the enhancement of the EPPP and the implementation of additional competency-based assessments to complement the EPPP. Focus Panels, Round 2 PES conducted a second round of focus panels and interviews to elicit information from targeted stakeholders regarding the feasibility of implementing competency assessments as part of the licensure/registration process. In one initiative, educators in the United States and Canada participated in a virtual (teleconference) focus panel or a telephone interview. In a separate initiative, key ASPPB stakeholders, including jurisdictional registrars and those involved in licensure or registration of psychologists, participated in a combined in-person/virtual focus panel during the 2009 ASPPB Annual Meeting, wherein members of the Advisory Committee moderated the panel and PES participated via teleconference. Participants were asked their views about how the validated competencies might be assessed either at or before for entry to practice, and the feasibility of implementing competency-based assessments into the licensure/registration of psychologists. A complete report of the results of the Round 2 Focus Panels may be found in Appendix 9. Survey regarding Enhanced EPPP and Alternative Assessment Methodologies Members of the Advisory Committee and Task Force were surveyed regarding possible enhancements to the existing computer-based EPPP, and about the most effective methodologies to assess competence.


PATF/PAAC Meeting 4 At the final combined meeting of the PATF and PAAC held in November 2009, the results of the augmented target sample survey (with additional Canadian respondents) and the two supplementary surveys were presented, as well as the results of the second round of focus panels and telephone interviews. The Task Force revised the method by which the Frequency and Criticality ratings for the knowledge statements were combined to develop weightings for these statements in the test specifications. They recommended that a subset of knowledge statements be edited to enhance their clarity. The results related to the expanded target sample data analyses are presented in the next section of this report, including the Task Force’s recommendations for revised test specifications for the EPPP. Detailed reports of the two supplementary surveys will be presented in the future. During the meeting, Task Force and Advisory Committee members also explored potential enhancements to the current EPPP by use of alternative items types, and reviewed complementary methodologies for assessing competence. These recommendations are presented later in this report.

Develop Final Recommendations Regarding the Test Specifications Underlying the EPPP and

the Final Report PES participated in a conference telephone call with the ASPPB Board of Directors to review the recommendations from PATF/PAAC Meeting 4 regarding the revision of the test specifications underlying the construction of the EPPP. Members of the Board approved the recommended test specifications, including percentage allocations for questions at the content area level, and guidelines for the selection of questions related to the revised and rank-ordered knowledge statements. In Spring 2010, PES submitted a draft final report to the members of the PAAC. Based on feedback from the PAAC, PES revised the report for review and approval by the ASPPB Board of Directors.


Summary of Results for U.S and Canadian Respondents in Target Sample This section presents information on the: (a) return rate for the respondents; (b) demographic and professional description of the samples; (c) comparison of the samples to the samples described in previous practice analysis studies and in other large-scale studies of the profession; (d) ratings of content areas, knowledge statements, competency clusters, competencies, and behavioral exemplars; (e) qualitative comments made by the respondents; (f) content-based profiles of practice; (g) hypothetical and final test specifications for content areas and knowledge statements; and (h) discussions related to the development of assessments of competence.

Return Rate Table 1 presents information regarding the return rate of completed surveys for U.S. and Canadian licensed psychologists, and for the total sample. The rates were based on the number of potential respondents eligible to be included in the sample rather than the number of invitation letters mailed to potential respondents. The number eligible was defined as the total number of potential survey respondents, minus addressees who were deceased or confirmed as no longer practicing, or whose materials were returned as undeliverable. Inspection of Table 1 indicates that the overall return rate for the Survey was about 26%. A higher percentage of Canadian licensed psychologists (53%) completed and returned the Survey than did licensed U.S. psychologists (24%).

Table 1 Response Rate

Country Sent Undeliverable Valid Responses Rate

United States 4375 230 4145 991 23.9%

Canada 367 12 355 189 53.2%

Total 4742 242 4500 1180 26.2%

Table 2 presents information on the total number and percent of respondents rating each of the six versions of the survey. About 200 respondents rated each of the versions of the survey.

Table 2 Responses by Survey Version

n %

Knowledge - Acquisition and Criticality 183 16%

Knowledge - Acquisition and Usage 203 17%

Competencies - Frequency and Criticality 196 17%

Competencies - Frequency and Importance 207 18%

Exemplars - Clusters 02, 03, 06 194 16%

Exemplars - Clusters 04, 05 197 17%

Total 1180 100%


Table 3 documents the number and percent of the total sample by survey version and country. As can be seen, respondents from the U.S. were slightly more likely to complete the survey versions containing knowledge statements while respondents from Canada were slightly more likely to complete the survey versions containing the exemplars of competencies.

Table 3 Responses to Survey Version by Country

United States Canada

n % n %

Knowledge - Acquisition and Criticality 161 16% 22 12%

Knowledge - Acquisition and Usage 173 17% 30 16%

Competencies - Frequency and Criticality 162 16% 34 18%

Competencies - Frequency and Importance 177 18% 30 16%

Exemplars - Clusters 02, 03, 06 159 16% 35 19%

Exemplars - Clusters 04, 05 159 16% 38 20%

Total 991 100% 189 100%

Table 4 presents information regarding respondents by U.S. and Canadian ASPPB-member jurisdictions, including the number of eligible respondents completing the survey from each jurisdiction and their percentage of the respondent pool from the US and Canada, respectively. All but two U.S. and one Canadian ASPPB-member jurisdictions were represented in the final sample of returns. The percentage of returns by jurisdiction varied from 0% to 100%.

Table 4 Respondents by Jurisdiction

n % of US

respondents

Alabama 24 2.4%

Alaska 11 1.1%

Arizona 27 2.7%

Arkansas 14 1.4%

California 32 3.2%

Colorado 17 1.7%

Connecticut 12 1.2%

Delaware 5 .5%

District of Columbia 12 1.2%

Florida 23 2.3%

Georgia 33 3.3%


Guam 0 0.0%

Hawaii 1 .1%

Idaho 9 .9%

Illinois 32 3.2%

Indiana 21 2.1%

Iowa 18 1.8%

Kansas 22 2.2%

Kentucky 24 2.4%

Louisiana 20 2.0%

Maine 6 .6%

Maryland 31 3.1%

Massachusetts 32 3.2%

Michigan 15 1.5%

Minnesota 25 2.5%

Mississippi 22 2.2%

Missouri 37 3.7%

Montana 6 .6%

Nebraska 18 1.8%

Nevada 17 1.7%

New Hampshire 5 .5%

New Jersey 16 1.6%

New Mexico 14 1.4%

New York 40 4.0%

North Carolina 24 2.4%

North Dakota 10 1.0%

Ohio 25 2.5%

Oklahoma 18 1.8%

Oregon 22 2.2%

Pennsylvania 38 3.8%

Puerto Rico 33 3.3%

Rhode Island 10 1.0%

South Carolina 12 1.2%

South Dakota 7 .7%

Tennessee 26 2.6%

Texas 30 3.0%

US Virgin Islands 0 0.0%

Utah 3 .3%

Vermont 5 .5%

Virginia 20 2.0%

Washington 27 2.7%


West Virginia 16 1.6%

Wisconsin 16 1.6%

Wyoming 8 .8%

Total 991 100.0%

n % of Canadian

respondents

Alberta 26 13.8%

British Columbia 38 20.1%

Newfoundland and Labrador 0 0.0%

Manitoba1 4 2.1%

New Brunswick 15 7.9%

Nova Scotia 16 8.5%

Ontario 55 29.1%

Prince Edward island 6 3.2%

Quebec2 1 .5%

Saskatchewan 28 14.8%

Total 189 100.0% 1 While no volunteers from this jurisdiction specifically signed up to participate on an online collector, four respondents sampled from other jurisdictions indicated Manitoba to be their primary employment location 2 While this jurisdiction declined to provide a sample for the survey, one respondent sampled from other jurisdictions indicated Quebec to be his or her primary employment location

Respondent Demographic and Professional Characteristics

This section presents information on the demographic and professional characteristics of U.S. and Canadian respondents, and recently and less-recently licensed/registered respondents, as derived from the responses to the section of the survey focused on Demographic and Professional Information. Selected comparisons to the demographic and professional information reported in the ASPPB’s 1995 and 2003 practice analysis studies are presented, as available, as are comparisons to other demographic and professional characterizations of the profession. Table 5 documents the number and percent of the total sample of respondents from the U.S. (84%) and Canada (16%), and Table 6 documents the number and percent of the total sample of recently (36%) and less-recently licensed/registered respondents (64%). The composition of the sample of respondents is consistent with the sampling plan which had been designed to represent respondents in general proportion to the population by country and to over-represent recently-licensed/registered psychologists (defined as those licensed/registered within the most recent three-year period, 2007–2009).


Table 5 Respondents by Country

n %

United States 991 84%

Canada 189 16%

Total 1180 100%

Table 6 Respondents by Years of Licensure/Registration

n %

1 – 3 424 36%

4+ 738 64%

Total 1162 100%

As documented in Table 7, respondents from the U.S. were more likely (39%) to be recently licensed/registered than respondents from Canada (23%).

Table 7 Years of Licensure/Registration by Country


n % n %

0 10 1% 3 2%

1 – 3 380 39% 44 23%

4+ 597 60% 141 75%

Total 987 100% 188 100%

While respondents in the U.S. were more likely to have been fully licensed/registered to practice psychology in the years 2007 – 2009, about 40% of both the U.S. and the Canadian sample were fully licensed/registered to practice in 1999 or earlier (see Table 8).


Table 8 Year Initially Fully Licensed/Registered to Practice Psychology by Country


n % n %

Before 1970 15 2% 1 1%

1970 – 1979 79 8% 14 7%

1980 – 1989 124 13% 19 10%

1990 – 1999 164 17% 42 22%

2000 – 2003 106 11% 24 13%

2004 50 5% 17 9%

2005 57 6% 19 10%

2006 116 12% 21 11%

2007 104 11% 18 10%

2008 129 13% 8 4%

2009 43 4% 5 3%

Total 987 100% 188 100%

Information about th

Documents

S PRACTICE OF LICENSED PSYCHOLOGISTS I UNITED …cdn.ymaws.com › › resource › ...STUDY OF THE PRACTICE OF LICENSED PSYCHOLOGISTS IN THE UNITED STATES AND CANADA prepared for