S PRACTICE OF LICENSED PSYCHOLOGISTS I …c.ymcdn.com/sites/ Final Report An Update Study of the Practice of Licensed Psychologists iv Table of Appendices Appendix 1 Practice Analysis

STUDY OF THE PRACTICE OF LICENSED PSYCHOLOGISTS IN THE UNITED STATES AND CANADA

prepared for

THE ASSOCIATION OF STATE AND PROVINCIAL PSYCHOLOGY BOARDS P.O. Box 241245

Montgomery, AL 36124

prepared by

Sandra Greenberg, Ph.D. Carla M. Caro, M.A. I. Leon Smith, Ph.D.

PROFESSIONAL EXAMINATION SERVICE Department of Research and Development

475 Riverside Drive New York, NY 10115-0089

July 2010

Copyright © 2010 by the Association of State and Provincial Psychology Boards P.O. Box 241245, Montgomery, AL 36124

ASSOCIATION OF STATE AND PROVINCIAL PSYCHOLOGY BOARDS

Board of Directors, 2008

Alex M. Siegel, J.D., Ph.D., President Kenneth G. Roy, Ed.D., Past President Emil Rodolfa, Ph.D., President Elect

Martha N. Storie, B.S., Secretary-Treasurer Joseph S. Rallo, Ph.D., Member-at-Large

Jack B. Schaffer, Ph.D., ABPP, Member-at-Large Carol Webb, Ph.D., ABPP, Member-at-Large


Emil Rodolfa, Ph.D., President Alex M. Siegel, J.D., Ph.D., Past President

Jack B. Schaffer, Ph.D., ABPP, President Elect Martha N. Storie, B.S., Secretary-Treasurer

Jacqueline B. Horn, PhD., ABPP, Member-at-Large Joseph S. Rallo, Ph.D., Member-at-Large

Carol Webb, Ph.D., ABPP, Member-at-Large


Jack B. Schaffer, Ph.D., ABPP, President Emil Rodolfa, Ph.D., Past President

Joseph S. Rallo, Ph.D., President Elect Martha N. Storie, B.S., Secretary-Treasurer

Carol Webb, Ph.D., ABPP, Member-at-Large Jacqueline B. Horn, Ph.D., Member-at-Large Fred Millán, Ph.D., ABPP, Member-at-Large

Practice Analysis Advisory Committee

Emil Rodolfa, Ph.D., Chair Greg Gormanous, Ph.D.

Joan Grusec, Ph.D. Catherine Yarrow, Ph.D.

Practice Analysis Task Force

Consuelo Arbona, Ph.D. Nancy Gourash Bliwise, Ph.D.

Darcy Cox, Psy.D., R. Psych., ABPP-CN Wil Counts, R.Ph., Ph.D.

Dennis Doverspike, Ph.D., ABPP Kelly Ducheny, Psy.D.

John Hunsley, Ph.D. Mary Pat McAndrews, Ph.D., C.Psych.

Morgan Sammons, Ph.D., ABPP Margaret Smith-Zoeller, Psy.D.

Lois Tetrick, Ph.D. Sheila Woody, Ph.D.

PES Final Report An Update Study of the Practice of Licensed Psychologists i

Acknowledgements On behalf of Professional Examination Service (PES), we are pleased to have conducted this major research study for the Association of State and Provincial Psychology Boards (ASPPB). This report summarizes the practice of licensed/registered psychologists in the U.S. and Canada and explores the evolving nature of competence across the professional lifespan of the practitioner and provides the groundwork for its assessment. Information included in this report came from the following sources: relevant literature in psychology as well as other professions regarding the assessment of competence; information from focus panels and independent reviews by subject-matter experts; a pilot survey and a validation survey of 5000 licensed/registered psychologists; and discussions with both regulators and educators. A content-based approach was used to systematically delineate the content areas and knowledge base required for entry into independent practice. Those results represent a contemporary description of the Scientific Knowledge underlying the practice of psychology and provide the basis for developing defensible test specification for the Examination for Professional Practice in Psychology (EPPP). A process-based approach was used to systematically delineate competencies associated with five additional areas of competence: Evidence-Based Decision Making/Critical Reasoning; Professionalism/Ethics; Assessment; and Intervention/Supervision/Consultation. Those results are a key requirement for assessing evolving levels of competence, including validated behavioral exemplars of competencies required at entry into independent practice. A project of this magnitude depends on the hard work and commitment of many professionals, and we are pleased to acknowledge their contributions to the final product. This study required a substantial investment of ASPPB’s financial resources and personnel. PES endorses ASPPB’s ongoing commitment to the continued development of an exemplary examination program as reflected in the EPPP as well as its forward looking approach to the assessment of competency. We wish to recognize the enduring contributions of the ASPPB Practice Analysis Advisory Committee for the wisdom and direction it provided. Its four members—Emil Rodolfa, Ph.D., Chair; Greg Gormanous, Ph.D.; Joan Grusec, Ph.D.; and Catherine Yarrow, Ph.D.—and the ASPPB Executive Officer, Stephen T. DeMers, Ed.D., worked with us and supported our efforts throughout the conduct of the study. The 12 members of the ASPPB Practice Analysis Task Force worked tirelessly through four face-to-face meetings; and numerous e-mail-based assignments and telephone conference calls in order to refine the content areas and knowledge statements; and draft, review and refine the competency clusters, competency statements, and behavioral exemplars. They approached each task with wisdom and wit, always willing to provide their own perspectives and listen to the views of others in order to articulate a clear, concise, and contemporary description of practice. We are indebted to Consuelo Arbona, Ph.D.; Nancy Gourash Bliwise, Ph.D.; Darcy Cox, Psy.D., R. Psych., ABPP-CN; Wil Counts, R.Ph., Ph.D.; Dennis Doverspike, Ph.D., ABPP; Kelly Ducheny, Psy.D.; John Hunsley, Ph.D.; Mary Pat McAndrews, Ph.D., C.Psych.; Morgan

PES Final Report An Update Study of the Practice of Licensed Psychologists ii

Sammons, Ph.D., ABPP; Margaret Smith-Zoeller, Psy.D.; Lois Tetrick, Ph.D.; and Sheila Woody, Ph.D. for making our work so easy. We conclude by stating that except where specifically noted, the views expressed in this report are those of PES and do not necessarily reflect the view of ASPPB or of those experts who provided advice on the conduct of this investigation. We look forward to the interpretation and application of the results by ASPPB as it strives to maintain the highest level of excellence in the credentialing of psychologists in the U.S. and Canada. Sandra Greenberg, Ph.D., Vice President for Research and Development, PES Carla M. Caro, M.A., Research Director, PES I. Leon Smith, Ph.D., President and CEO, PES

PES Final Report An Update Study of the Practice of Licensed Psychologists iii

Table of Contents Acknowledgements .......................................................................................................................... i Table of Contents ........................................................................................................................... iii Table of Appendices ...................................................................................................................... iv Table of Tables ............................................................................................................................... v Table of Exhibits ........................................................................................................................... vii Executive Summary ........................................................................................................................ 1 Introduction ..................................................................................................................................... 6 Conduct of the Update Practice Analysis Study of Licensed Psychologists .................................. 7

Select the Subject-Matter Experts to Participate in the PATF .................................................... 8 Develop the Draft and Final Delineations of Content Areas and Knowledge Statements; and Competency Clusters and Competencies, and Behavioral Exemplars .............................. 8 Develop, Review, and Approve the Form and Content of the Survey of the Practice of Psychology ............................................................................................................................. 13 Pilot Test of Online Survey Instrument ..................................................................................... 17 Develop the Sampling Plan for the Conduct of the Survey ....................................................... 17 Implement the Practice Analysis Survey ................................................................................... 19 Perform the Data Analyses, Develop Preliminary Test Specifications for the EPPP, and Discuss Implications of Findings Regarding the Assessment of Professional Competence .. 19 Complementary Data Collection Related to Competency Assessment ..................................... 20 Develop Final Recommendations Regarding the Test Specifications Underlying the EPPP and the Final Report ............................................................................................................... 21

Summary of Results for U.S and Canadian Respondents in Target Sample ................................ 22 Return Rate ................................................................................................................................ 22 Respondent Demographic and Professional Characteristics ..................................................... 25 Comparisons Between Target Sample and Samples Identified in Previous ASPPB Practice Analysis Studies and by Other Research Initiatives ............................................................... 49 Results Related to Content- and Competency-based Delineations ........................................... 54

Content-based Delineation ..................................................................................................... 54 Content areas ....................................................................................................................... 54 Knowledge in the practice of psychology ........................................................................... 62 Implications of the Ratings Related to the Content-based Delineation .............................. 74 Qualitative Results Related to Content Areas and Knowledge .......................................... 75

Content-Based Profile of Practice and Test Specifications for the EPPP .............................. 76 Profile of Practice ............................................................................................................... 76 Development of Test Specifications ................................................................................... 83

Competencies ......................................................................................................................... 91 Competency Clusters .......................................................................................................... 91 Competency Statements ...................................................................................................... 94 Competency Exemplars .................................................................................................... 105

Implications of Findings Regarding the Assessment of Professional Competence .................... 123 Results of PATF and PAAC Surveys ...................................................................................... 125 Results of Focus Panels and Interviews .................................................................................. 128

Conclusions ................................................................................................................................. 131 References and Background Literature ....................................................................................... 133

PES Final Report An Update Study of the Practice of Licensed Psychologists iv

Table of Appendices

Appendix 1 Practice Analysis Advisory Committee and their Professional Affiliations and ASPPB Personnel Monitoring Study Appendix 2 Members of the Practice Analysis Task Force and their Professional Affiliations Appendix 3 Final Report of Independent Review, Round 1 Appendix 4 Final Report of Focus Panels, Round 1 Appendix 5 Final Report of Independent Review, Round 2 Appendix 6 Materials Related to the Conduct of the Pilot Survey Appendix 7 Screen Shots of Large-Scale Validation Survey Appendix 8 Materials Related to the Conduct of the Large-Scale Validation Survey Appendix 9 Final Report of Focus Panels, Round 2 Appendix 10 Areas of Formal Post-doctoral Level Training Appendix 11 Knowledge Acquisition, Criticality, and Usage ratings of U.S. and Canadian Respondents Appendix 12 Knowledge Acquisition, Criticality, and Usage ratings of Recently and Less-recently Licensed/Registered Respondents Appendix 13 Verbatim Responses to Open-ended Question Appendix 14 Competency Frequency, Criticality, and Importance Ratings of U.S. and Canadian Respondents Appendix 15 Competency Frequency, Criticality, and Importance Ratings of Recently and Less-recently Licensed/Registered Respondents

PES Final Report An Update Study of the Practice of Licensed Psychologists v

Table of Tables Table 1 Response Rate ................................................................................................................ 22 Table 2 Responses by Survey Version ........................................................................................ 22 Table 3 Responses to Survey Version by Country ..................................................................... 23 Table 4 Respondents by Jurisdiction .......................................................................................... 23 Table 5 Respondents by Country ................................................................................................ 26 Table 6 Respondents by Years of Licensure/Registration .......................................................... 26 Table 7 Years of Licensure/Registration by Country ................................................................. 26 Table 8 Year Initially Fully Licensed/Registered to Practice Psychology by Country .............. 27 Table 9 Highest Level of Education in Psychology Beyond a Bachelor's Degree by Country 27 Table 10 Institution Where Awarded Highest Degree in Psychology by Country ..................... 28 Table 11 Highest Level of Education in Psychology Beyond a Bachelor's Degree by Years of Licensure/Registration ............................................................................................. 28 Table 12 Institution Where Awarded Highest Degree in Psychology by Years of

Licensure/Registration .................................................................................................. 29 Table 13 Major Area of Training by Country .............................................................................. 30 Table 14 Current Major Area of Practice by Country ................................................................. 31 Table 15 Participated in Formal Post-Doctoral Specialization or Re-specialization Program by Country .................................................................................................................... 32 Table 16 Major Area of Training by Years of Licensure/Registration ........................................ 32 Table 17 Current Major Area of Practice by Years of Licensure/Registration ........................... 33 Table 18 Participated in Formal Post-Doctoral Specialization or Re-specialization Program by Years of Licensure/Registration .............................................................................. 34 Table 19 Certified by ABPP by Country ..................................................................................... 35 Table 20 Certified by ABPP by Years of Experience .................................................................. 35 Table 21 Area(s) where certified by Country .............................................................................. 35 Table 22 Area(s) where certified by Years of Experience ........................................................... 36 Table 23 Employment Setting by Country .................................................................................. 37 Table 24 Health Service Provider by Country ............................................................................. 38 Table 25 Employment Setting by Years of Licensure/Registration ............................................ 39 Table 26 Health Service Provider by Years of Experience ......................................................... 40 Table 27 Primary Orientation by Country ................................................................................... 41 Table 28 Secondary Orientation by Country ............................................................................... 41 Table 29 Primary Orientation by Years of Licensure/Registration ............................................. 42 Table 30 Secondary Orientation by Years of Experience ............................................................ 42 Table 31 Area(s) in Which Respondents Have Expertise by Country ......................................... 43 Table 32 Experiences as a Psychologist Within the Past Three Years by Country ..................... 45 Table 33 Experiences as a Psychologist Within the Past Three Years by Years of Experience 46 Table 34 Sex by Country ............................................................................................................. 47 Table 35 Sex by Years of Experience .......................................................................................... 47 Table 36 Racial/Ethnic Background by Country ......................................................................... 47 Table 37 Racial/Ethnic Background by Years of Experience ..................................................... 48 Table 38 Disability (as defined by the Americans with Disabilities Act) by Country ................ 48 Table 39 Type of Disability by Country ...................................................................................... 48 Table 40 Disability by Years of Experience ................................................................................ 49

PES Final Report An Update Study of the Practice of Licensed Psychologists vi

Table 41 Institution Where Awarded Highest Degree in Psychology by Study Year and Country ......................................................................................................................... 50 Table 42 Major Area of Training by Study Year and Country .................................................... 51 Table 43 Current Major Area of Practice by Study Year and Country ....................................... 52 Table 44 Primary Theoretical Orientation by Study Year and Country ...................................... 53 Table 45 Sex by Study Year and Country ................................................................................... 53 Table 46 Racial/Ethnic Background by Study Year and Country ............................................... 54 Table 47 Knowledge Content Areas Frequency, Criticality, and Importance Ratings for Total Sample ................................................................................................................. 56 Table 48 Knowledge Content Areas— Frequency Ratings by Country and Years of Experience as Licensed/Registered Psychologist ......................................................... 59 Table 49 Knowledge Content Areas Criticality Ratings by Country and Years of Experience as Licensed/Registered Psychologist ......................................................... 60 Table 50 Knowledge Content Areas Importance Ratings by Country and Years of Experience as Licensed/Registered Psychologist ......................................................... 61 Table 51 Frequency Distribution for Acquisition, Criticality, and Usage Ratings, and Mean and Standard Deviation for Criticality Ratings for Total Sample of Respondents ...... 64 Table 52 Mean of Means for Criticality of Knowledge Statements ............................................ 74 Table 53 Empirically Derived Test Specifications for Content Areas ........................................ 78 Table 54 Knowledge Statements—Mean Weightings ................................................................. 78 Table 55 Recommended Test Specifications for Content Areas ................................................. 86 Table 56 Final Knowledge Statements and Ranks for EPPP ....................................................... 87 Table 57 Competency Clusters Frequency, Criticality, and Importance Ratings for Total Sample .......................................................................................................................... 93 Table 58 Mean of Means for Competency Statements on the Frequency, Criticality, and

Importance Rating Scales ............................................................................................. 94 Table 59 Frequency Distribution, Number, Mean, and Standard Deviation for Frequency

Ratings for Competency Statements ............................................................................ 95 Table 60 Frequency Distribution, Number, Mean, and Standard Deviation for Criticality

Ratings for Competency Statements ............................................................................ 98 Table 61 Frequency Distribution, Number, Mean, and Standard Deviation for Importance

Ratings for Competency Statements .......................................................................... 101 Table 62 Support for Acquisition of Behavioral Exemplars ...................................................... 107 Table 63 Methods for Enhancing the EPPP by Competency Cluster ........................................ 126 Table 64 Methods for Assessing Competency by Cluster (excluding Scientific Knowledge) .. 127

PES Final Report An Update Study of the Practice of Licensed Psychologists vii

Table of Exhibits

Exhibit 1 Recommended Test Specifications for Content Areas ................................................... 3 Exhibit 2 Content- and Competency-Based Delineation of the Practice of Psychology ............. 12 Exhibit 3 Validation Survey Content Outline .............................................................................. 14 Exhibit 4 Knowledge Statements Not Meeting Criterion for the Criticality Rating Scale .......... 62 Exhibit 5 Weighting of Knowledge Statements ........................................................................... 77 Exhibit 6 16 Modified Knowledge Statements ............................................................................ 84

PES Final Report An Update Study of the Practice of Licensed Psychologists 1

Executive Summary Study of the Practice of Licensed Psychologists in the United States and Canada

Sandra Greenberg, Ph.D.

Carla M. Caro, M.A. I. Leon Smith, Ph.D.

Professional Examination Service

The Association of State and Provincial Psychology Boards sponsored a study of the practice of licensed psychologists in the United States and Canada. The underlying conceptual charge was:

• The identification and validation of underlying professional competencies (including but not limited to those related to professional knowledge)

• The identification of assessment methods to best measure underlying professional competencies (including but not limited to those related to professional knowledge)

• Revised test specifications for the EPPP updating the knowledge base and integrating additional relevant competencies

Professional Examination Service implemented the study under the direction of a Practice Analysis Advisory Committee and in conjunction with a Practice Analysis Task Force. The study consisted of two partially concurrent, partially sequential explorations. As in previous practice analyses, the first exploration examined the knowledge required for psychology practice with the goal of updating the EPPP test specifications, and retained a content-based organizational structure including eight content areas comprised of knowledge statements. The second exploration examined the competencies underlying the practice of psychology. Accordingly, a competency-based framework was developed and validated, including the delineation of six competency clusters, associated competencies, and behavioral exemplars typifying the development of competence. A survey was developed and sent to approximately 5000 licensed psychologists in the United States and Canada in order to validate and update all elements in the comprehensive framework, including the content areas and knowledge statements, and the competency clusters, competencies, and exemplars. The return rate was 26%. Analysis of completed surveys produced information about (a) the demographic and professional background of licensed psychologists, (b) the critical knowledge licensed psychologists use, (c) comments about changes occurring in the profession, (d) the competencies required in professional practice, and (e) the validation of specific competencies and behavioral exemplars. Results related to the first exploration were used to review and refine the test specifications to ensure that the knowledge assessed in the EPPP is required for the performance of critical behaviors and serves the public protection function of regulation. Results related to the second exploration were used to develop and validate a conceptual framework for the assessment of competence at various stages in professional development. Various types of assessments were identified as useful for the assessment of competence. Preliminary discussions focused both on alternate question types that might be integrated into the EPPP, and the development of complementary assessment that might


be integrated into the assessment of licensed/registered either before, during, or after initial licensure/registration.

Key Findings and Conclusions • Regardless of country, respondents were more likely to have been trained in the major areas

of clinical, counseling, and educational psychology than they were to be currently practicing in those major areas; and were more likely to be currently practicing in the major areas of clinical neuropsychology, forensic psychology, geropsychology, health psychology, and rehabilitation psychology than to have initially been trained in those major areas.

• In the U.S., more than one-half of the respondents indicated cognitive/behavioral psychology (58%), and 9% and 11% indicated interpersonal psychology and psychodynamic psychology as their primary orientation, respectively. No more than 6% of the U.S. respondents indicated any of the other four specifically-delineated theoretical orientations as primary. In Canada, about two-thirds of the respondents indicated cognitive/behavioral psychology (66%), and 8% indicated interpersonal psychology as their primary orientation. No more than 4% of the Canadian respondents indicated any of the other specifically-delineated orientations as primary.

• Regardless of country, respondents were most likely to describe themselves as being experts in clinical psychology and in assessment/diagnosis/evaluation, and somewhat less likely to indicate clinical child psychology; counseling psychology; and treatment, intervention, and prevention. With very few exceptions, respondents indicated that they had expertise in one or more of each of 57 specifically-delineated areas of expertise.

• Nearly one-third of the U.S. respondents have participated in formal post-doctoral specialization and/or respecialization and 17% of Canadian respondents have done so as well. Recently licensed respondents are more likely to have trained in clinical, clinical child, and school psychology, and less likely to have trained in community, counseling, developmental, educational, experimental, industrial/organizational, and social psychology than less recently licensed/registered respondents.

• Regardless of country, the majority of respondents are female (62% and 70%, respectively),

and the recently licensed/registered respondents are more likely to be female than less-recently licensed/registered respondents (72% and 58%).

• The eight content areas and 77 associated knowledge statements were validated as an

organizing vehicle for the development of the EPPP. Exhibit 1 documents the recommended test specifications for the EPPP.


Exhibit 1 Recommended Test Specifications for Content Areas

% of exam

Biological Bases of Behavior –– knowledge of (a) biological and neural bases of behavior, (b) psychopharmacology, and (c) methodologies supporting this body of knowledge

12%

Cognitive-Affective Bases of Behavior— knowledge of (a) cognition, (b) theories and empirical bases of learning, memory, motivation, affect, emotion, and executive function, and (c) factors that influence cognitive performance and/or emotional experience and their interaction

13%

Social and Cultural Bases of Behavior — knowledge of (a) interpersonal, intrapersonal, intergroup, and intragroup processes and dynamics, (b) theories of personality, and (c) diversity issues

12%

Growth and Lifespan Development — knowledge of (a) development across the full life span, (b) atypical patterns of development, and (c) the protective and risk factors that influence developmental trajectories of individuals

12%

Assessment and Diagnosis — knowledge of (a) psychometrics, (b) assessment models and instruments, (c) assessment methods for initial status of and change by individuals, couples, families, groups, and organizations/systems, and (d) diagnostic classification systems and their limitations

14%

Treatment, Intervention, Prevention, and Supervision — knowledge of (a) individual, couple, family, group, organizational, or community interventions for specific problems/disorders in diverse populations, (b) intervention and prevention theories, (c) best practices and practice guidelines, (d) consultation and supervision models, and (e) evidence supporting efficacy and effectiveness of interventions

14%

Research Methods and Statistics — knowledge of (a) research design, methodology, and program evaluation, (b) instrument selection and validation, (c) statistical models, assumptions, and procedures, and (d) dissemination methods

8%

Ethical/Legal/Professional Issues — knowledge of (a) codes of ethics, (b) professional standards for practice, (c) legal mandates and restrictions, (d) guidelines for ethical decision-making, and (e) professional training and supervision

15%

• The competency-based model, including six competency clusters (Scientific Knowledge,

Evidence-Based Decision Making/Critical Reasoning, Interpersonal and Multicultural Competence, Professionalism/Ethics, Assessment, and Intervention/Supervision/Consultation) was validated via the large-scale survey of practice.

• The competency clusters each represent competencies which are performed frequently-to-

very frequently, are moderately -to-highly critical to optimizing outcomes for patient/client/public, and are moderately-to-very important to the practice of the respondents.


• The 37 competencies were generally validated and are performed frequently-to-very frequently, are moderately-to-highly critical to optimizing outcomes for patient/client/public, and are moderately-to-very important to the practice of the respondents.

• A detailed review of the results for the 277 behavioral exemplars associated with the competencies indicates a general level of support for the developmental unfolding of the validated competencies as operationalized by the exemplars.

• Methodologies for assessing competence were evaluated.

• The feasibility of enhancing the EPPP with alternate item types was explored as one way of augmenting the current licensure/registration process.

• The potential for developing new assessments to complement the EPPP was preliminarily

discussed as was the use of such assessments at various points of time pre- and post-licensure/registration.

• Discussions amongst the members of the PATF and the PAAC confirmed the utility of the

EPPP as an effective tool for the assessment of the Scientific Knowledge base underlying the practice of psychology. Recommendations for a revised set of test specifications for the EPPP were approved. Future directions in regard to alternate item types that might be built into this computer-delivered examination were discussed.

• Discussions amongst the members of the PATF and the PAAC as well as key stakeholders

involved in the assessment of competency in students, interns, and practica participants, and jurisdictional regulation confirmed interest in the assessment of competency via complementary assessments that might be integrated into pre- and post-licensure/registration activities and/or licensure/registration requirements.

• A substantial investment in time and resources will be required should ASPPB choose to

support the development of entry-level competency assessments for licensed/registered psychologists. The widespread dissemination of this technical report, including information about the development of validated competencies to be acquired pre- and post licensure/registration, should be a priority and might easily be accomplished via the ASPPB’s website.

• ASPPB needs to consider the leadership role it might play in the future development and

support of competency assessment. To that end, a meeting of key stakeholders including other organizations within the profession, may be desirable in order to sort through the issues and develop a game plan for moving the competency assessment agenda forward across the professional development continuum. Among numerous resources available for the discussion are the following:

o The results of the two ASPPB Foundation grant-supported assessment initiatives focusing on the development of competency assessment pre-licensure, as well as numerous other research-based assessment development initiatives occurring within the professional education and training community;


o The recently published work of the Competency Assessment Workgroup, including the portfolio of assessment tools known as the Competency Assessment Toolkit for Professional Psychology (Toolkit); and, now,

o The results of the ASPPB-sponsored practice analysis study in the U.S. and Canada, including not only the validation of the Scientific Knowledge competency cluster, but the validation of five other competency clusters, each including behavioral exemplars representing the evolution of the validated competencies which develop across key milestones.


Introduction The Association of State and Provincial Psychology Boards (ASPPB) sponsored an update study of the practice of licensed psychologists in the United States and Canada to review and refine the test specifications for the Examination for Professional Practice in Psychology (EPPP). This study utilized a content-based approach to refine the delineation of the content areas and knowledge required in practice and to identify the competencies performed by psychologists. Additionally, the study was designed to build on competency frameworks being explored by other key stakeholder groups in the profession. The results are expected to provide ASPPB with the empirical support necessary for considering changes in the examination program for licensure, including augmentation of the EPPP and/or the development of complementary assessments to the EPPP. The specific purposes of the update study were to:

• refine the knowledge required for professional practice, • validate the test specifications underlying the construction of the EPPP, • develop a comprehensive competency-based model of professional practice, • develop testable behavioral exemplars that operationalize competency at pre-determined

developmental anchor points of professional life, including at entry to practice, and • evaluate methodologies for assessing competence and explore the feasibility of

augmenting the licensure/registration process by enhancing the EPPP with alternate item types, and/or developing new competency-based assessments to complement the EPPP.

The overall process of practice analysis is described in the 1999 revision of the Standards for Educational and Psychological Testing (American Educational Research Association, the National Council on Measurement in Education, and the American Psychological Association). This document emphasizes the concept of content validity and the need to conduct a practice analysis to assure that the professional knowledge/skills/abilities assessed in credentialing initiatives are in fact limited to those required for competent performance and serve a public protection function. Practice analysis, then, becomes the primary basis by which a professional association or credentialing agency establishes and defends the content validity of its credentialing initiatives.


Conduct of the Update Practice Analysis Study of Licensed Psychologists This section contains a description of the procedures Professional Examination Service (PES) implemented to conduct an update practice analysis study of psychology on behalf of the ASPPB. PES conducted the update practice analysis study in 2007 – 2009 in conjunction with a four-member Practice Analysis Advisory Committee (PAAC) and a 12-member Practice Analysis Task Force (PATF). The members of the PAAC represent key decision makers in ASPPB; in part, they are responsible for guiding aspects of the licensure/registration program, including examination development and validation efforts. The members of the PATF represent academics, training programs, and practitioners in different major areas of practice, including new and emerging specialties, and with expertise in competency assessment. The responsibilities of the PAAC were to (a) identify key parameters and select subject-matter experts to serve on the PATF; (b) facilitate the conduct of focus panels and complementary data collection initiatives; (c) review and approve the form and content of the survey, data collection procedures, data analyses, and results; (d) review the draft final report for presentation to ASPPB’s Board of Directors for final approval; and (e) develop recommendations regarding the test specifications for the EPPP and other assessment initiatives. Appendix 1 lists the members of the PAAC and their professional work affiliations. The Chair of the PAAC served as the liaison to the PATF and had the responsibility of maintaining communications between the PATF and the PAAC. The Chair of the PAAC also had the responsibility of maintaining communications between the ASPPB Board of Directors and the PAAC. One member of the PAAC had the responsibility of maintaining communications between the Exam Committee of ASPPB and the PAAC. In addition to the Chair, one member of the PAAC attended each of the first three Task Force meetings; and all PAAC members attended the final Task Force meeting. The PAAC developed a conceptual charge for the study that included the following goals:

• The identification and validation of underlying professional competencies (including but not limited to those related to professional knowledge).

• The identification of assessment methods to best measure underlying professional competencies (including but not limited to those related to professional knowledge).

• Revision of test specifications for the EPPP updating the knowledge base and integrating additional relevant competencies.

The responsibilities of the PATF were to (a) develop the draft update and final delineations of practice including both content areas and knowledge statements as well as competency clusters, competencies, and behavioral exemplars; (b) select subject-matter experts to participate in complementary data collections procedures; (c) discuss and review the form and content of a Web-based survey of practice; and (d) review and discuss the form and content of the data analyses and results. The PATF met four times and conducted additional business by teleconference and e-mail between 2008 and 2009.


Select the Subject-Matter Experts to Participate in the PATF At a 1-day meeting in November 2007, members of the PAAC developed key parameters for identifying potential members of the PATF. These parameters included expertise in competency assessment and the training of psychology students and psychologists; familiarity with the work of the APA Assessment of Competency Benchmarks Workgroup (2006); experience on the EPPP exam committee; representation from varied professional practice environments, including private practice, consultants, and academics; expertise in a variety of subject-matter areas, including new and emerging areas of practice; and representation of a range of demographic characteristics. Key individuals who members of the PAAC wished to recruit were also identified and discussed. Subsequent to the meeting, calls went out to a variety of organizations and individuals, requesting assistance with outreach effort seeking nominees for the PATF who met the criteria. These organizations included the ASPPB Board of Directors, the Association of Canadian Psychology Regulatory Organizations (ACPRO), the Canadian Council of Professional Psychology Programs (CCPPP), the Council of Chairs of Training Councils (CCTC), and various other professional associations representing psychologists, such as the Association of Black Psychologists and the Asian-American Psychological Association. An online nomination form was used to collect professional and demographic information on the nominees, and nominees were requested to submit their current curriculum vitae. Nominees were also asked if they would be willing to fill other study roles as independent reviewers or pilot test participants if they were not selected to serve on the PATF. In January 2008, PES conducted a series of telephone conference calls with the PAAC wherein they discussed the intended composition of the PATF, reviewed the qualifications and areas of expertise of each nominee, and voted for a final slate of participants. A listing of the PATF members, their professional affiliations, and jurisdictional location may be found in Appendix 2. In advance of the first PATF meeting, Task Force members were sent a packet of information describing the study goals, including a description of the EPPP content areas and knowledge, as well as the study’s additional focus on competency assessment. Task Force members were also provided background literature on competency assessment, including concurrent APA work on competency benchmarks; and articles on competency assessment including studies of competency conducted for other professions, to read in advance of the first PATF meeting. (Note: A list of the articles provided to the members of the Task Force prior to the meeting and throughout the study may be found within the references of this report.)

Develop the Draft and Final Delineations of Content Areas and Knowledge Statements; and Competency Clusters and Competencies, and Behavioral Exemplars

In 1995, process- and content-based approaches were implemented by PES to study the practice of licensed psychologists in the United States and Canada (Greenberg, Smith, & Muenzen, 1996). In 2002 to 2003, the process- and content-based descriptions of practice were updated. (Greenberg & Jesuitus, 2003). The process- and content-based approaches used in those studies provided a structure for describing contemporary practice—that is, for identifying what


psychologists do—and facilitated the development of examination items in a practice-related framework. The process-based approach comprised the delineation of roles and associated responsibilities performed by psychologists. Roles represented major categories of activities, and responsibilities represented specific activities psychologists performed within each role. Unique sets of responsibilities were associated with each role. The content-based delineation comprised the content areas and the knowledge statements required to perform the entire set of responsibilities. Content areas represented categories of knowledge used by psychologists in practice, and knowledge statements related to the content areas; they described an organized body of information needed to perform responsibilities. The current study consisted of two partially concurrent, partially sequential explorations. As in previous practice analyses, the first exploration examined the knowledge required for psychology practice with the goal of updating the EPPP test specifications, and retained a content-based organizational structure including content areas comprised of knowledge statements. In addition, the study built on current initiatives in credentialing, in general, and psychology, in specific, by incorporating related and ongoing work currently being supported by key stakeholder groups in the profession relative to competency assessment. For example, the APA Task Force on the Assessment of Competence in Professional Psychology (2006) has stated:

. . . it is timely for professional psychology to embrace not only a culture of competence, but also a culture of the assessment of competence (Roberts, Borden, & Christiansen, 2005) that builds upon a long history of the assessment of competence both within and outside of psychology. . . (T)he assessment of competence fosters learning, evaluates progress, assists in determining the effectiveness of the curriculum and training program, advances the field, and protects the public (Kaslow, 2004).

To that end, the second exploration examined the competencies underlying the practice of psychology. Accordingly, the PATF developed a competency-based framework included the delineation of competency clusters, associated competencies, and behavioral exemplars typifying the development of competence. A range of complementary data collection efforts, including independent reviews, in-person and web-based focus panels, and supplementary surveys, supported the development and validation of the delineations of content areas and knowledge; and competency clusters, competencies, and behavioral exemplars. The results of this study, including a large-scale survey of practice and other complementary data collection efforts, provided empirical support for updating the EPPP test specifications, as well as beginning to identify ways in which the existing computer-based examination could be augmented with the addition of enhanced question types. The parallel focus of the study on competencies underlying professional practice laid the initial groundwork for the identification of essential competencies and the further exploration of the development of complementary assessments to best measure professional competencies. Accordingly, the study included an initial exploration of the feasibility of incorporating such competency assessments into both training programs (pre-licensure) and the licensure/registration processes.


The following sequence of activities provided the basis for the development and validation of the delineations: PATF Meeting 1 The PATF first met in April 2008 for an initial review and revision of the existing content-based delineation including content areas and knowledge statements, and to begin consideration of how to build a model of psychology practice that includes professional competence. The Task Force was introduced to the study’s parallel exploration of the development of complementary competency assessment methodologies that might be implemented in conjunction with the EPPP, and discussed the potential impact of such assessments to graduate training and professional development. To this end, the PATF referred to the ongoing work of the APA Competency Benchmarks Workgroup, and examined related literature and models (for example, the “Cube” model, Rodolfa, Bent & Eisman, et al., 2005), and reviewed methods of assessing competency in a number of different professions. The Task Force proposed an initial set of revisions to the content-based delineation, including the eight content areas, definitions for each area, and knowledge statements. They also began to develop a competency-based model of practice consisting of six clusters of competence. The first cluster encompassed the entire knowledge content currently tested by the EPPP and was designated as the Scientific Knowledge Base. Five additional proposed clusters consisted of three foundational clusters (Evidence-Based Decision Making/Critical Reasoning, Interpersonal and Cultural Competence, and Professionalism/Ethics) and two functional clusters (Assessment and Intervention/Consultation). The Task Force also identified four levels of professional development at which specific behaviors demonstrating competency could be demonstrated. These levels were designed to complement the work of the APA Assessment of Competency Benchmarks Workgroup, and included:

Pre-licensure (prior to independent practice) Level 1: Entry-level supervised practice (during training) Level 2: Advanced supervised practice (internship level) Post-licensure (independent practice) Level 3: Entry-level independent practice (just licensed) Level 4: Advanced independent practice

Following the meeting, PATF members completed an e-mail-based critical review of their work products. PES then convened a series of conference calls with Task Force subcommittees to reconcile the suggestions about both the updated delineation of content areas and knowledge statements, and the newly developed model of competency clusters and competencies. The products of those conference telephone calls were then disseminated for review by independent subject matter experts (SMEs). Independent Review, Round 1 In July of 2008, the first round of complementary data collection was undertaken to supplement the work of the PATF. This consisted of a review of the content-based delineation and competency model by independent SMEs, and was designed to ensure the completeness and


clarity of both components. The SMEs were identified through the Task Force nomination process and were invited to participate by Task Force members. Two separate panels were created to critically review either the content-based delineation (the Scientific Knowledge Base) or the draft competency model, comprising the remaining five competency clusters. Extensive suggestions were submitted by the reviewers and were collated by PES. Then, by way of a PES-facilitated series of subcommittee teleconferences, the Task Force reconciled the suggestions and produced the updated drafts of the content-based delineation and the competency model. A complete report of the Independent Review, Round 1 process and results may be found in Appendix 3. Focus Panels, Round 1 Two rounds of focus panels were conducted as part of this study. The first round consisted of three focus panels held in conjunction with the American Psychological Association Annual Meeting held in Boston, MA in August 2008, and a single panel conducted at the Annual Meeting of the Ontario College of Psychologists in Toronto, ON in September 2008. The goal of these focus panels facilitated by PES was two-fold: to critically examine the proposed competency-based model of psychology practice, including the clusters and competencies, and to begin developing behavioral exemplars of the competencies at the four professional levels identified by the Task Force. A total of 23 SMEs participated in these panels. A complete report of the focus panel process and results, including the protocol used and a demographic description of the participants, may be found in Appendix 4. PATF Meeting 2 In November, 2008 the PATF convened its second 3-day meeting. At that meeting, Task Force members reviewed the results of the independent review and focus panels. They incorporated the findings of these complementary data collection efforts as they continued to refine the content- and competency-based models. Task Force members participated in small group work to revise the content areas and knowledge statements and the competency clusters and competencies. After de-briefing, additional small group work involved the development of behavioral exemplars demonstrating each competency at the four levels of professional development previously identified. Finally, Task Force members were asked to nominate colleagues to participate in the second round of independent review, and in particular, to identify SMEs with expertise in the areas of biological psychology and industrial/organizational psychology— areas which were identified as needing augmentation. As in the first meeting, PATF members engaged in a post-meeting e-mail-based review of the iteration of the model produced at the meeting, and participated subcommittee teleconferences to reconcile their edits, suggestions, and comments submitted during the review process. The products of those conference telephone calls were then prepared for review by independent SMEs. Independent Review, Round 2 The revised content- and competency-based model was sent to a second panel of independent experts for outside review and critique. Extensive suggestions were submitted by the reviewers and were collated by PES. Subsequently, PES facilitated a series of teleconferences with Task Force members organized into subcommittees, reconciled the suggestions, and produced the next draft of the model. A complete report of the Round 2 Independent Review process and results may be found in Appendix 5.


Exhibit 2 presents an overview of the work product developed through these processes. The content-based delineation consists of eight content areas comprised of 78 knowledge statements. The competency based delineation consists of a model containing six competency clusters. The first of these is the Scientific Knowledge Base which is described within the content-based delineation. The model posits three foundational, cross-cutting competency clusters (Evidence-Based Decision Making/Critical Reasoning, Cultural and Interpersonal Competence, and Professionalism/Ethics) comprised of 19 competencies, and two functional competency clusters (Assessment and Intervention/Supervision/Consultation) comprised of 18 competencies. A total of 277 behavioral exemplars that demonstrate competence at four levels of professional development were developed in association with the 37 competencies.

Exhibit 2 Content- and Competency-Based Delineation of the Practice of Psychology

Content Areas in the Scientific Knowledge Base Number of Knowledge Statements

Biological Bases of Behavior 6

Cognitive-Affective Bases of Behavior 7

Social and Multicultural Bases of Behavior 14

Growth and Lifespan Development 9

Assessment and Diagnosis 12

Treatment, Intervention, and Prevention 12

Research Methods and Statistics 8

Ethical/Legal/Professional Issues 10

Total 78

Competency Clusters Number of Competencies

Number of Behavioral Exemplars

Scientific Knowledge Base n/a n/a

Evidence-Based Decision Making/Critical Reasoning 5 34

Cultural and Interpersonal Competence 4 39 Professionalism/Ethics 10 74 Assessment 10 63 Intervention/Supervision/ Consultation 8 67 Total 37 277


Develop, Review, and Approve the Form and Content of the Survey of the Practice of Psychology

The study’s exploration of three aspects of practice (the knowledge base needed for practice, the professional competencies of psychologists, and the behavioral exemplars by which these competencies are demonstrated) necessitated developing different versions of the survey due to the time required to evaluate each aspect. Additionally, since the content- and competency-based sections of the survey were designed to elicit ratings on three different scales, these versions were further subdivided, with each version asking respondents to rate the knowledge or competency on two of the three possible ratings scales. Finally, although the version containing the behavioral exemplars included only one rating scale, it contained a large number of items to be rated; accordingly it was divided into two subsets of the exemplars. Exhibit 3 presents the numbers of ratable items and the rating scales employed for each of the six resulting survey versions. In addition to these scales, all versions of the survey included a section containing 19 questions about the respondents’ professional and demographic background, and a qualitative question regarding expected changes in the profession. Descriptions of the specific rating scales follow the survey content outline.


Exhibit 3 Validation Survey Content Outline

Version

A B C D E F

Versions A & B

Section 1: Knowledge Statements – 78 statements • Acquisition–4 point scale ! ! • Usage–3 point scale ! • Criticality–4 point scale !

# ratings – subtotal 156 156 Section 2: Knowledge Content Areas – 8 content areas • Frequency–4 point scale ! ! • Criticality–4 point scale ! • Importance–4-point scale !

# ratings – subtotal 16 16 Knowledge missing from survey 1 1

Total ratings: Versions A & B 173 173

Versions C & D

Section 1: Competencies – 37 competencies • Frequency–4 point scale ! ! • Criticality–4 point scale ! • Importance–4 point scale !

# ratings – subtotal 74 74 Section 2: Competency Clusters – 6 clusters • Frequency–4 point scale ! ! • Criticality–4 point scale ! ! • Importance–4 point scale ! !

# ratings – subtotal 18 18 Total ratings: Versions C & D 92 92

VERSIONS E & F

Behavioral Exemplars – 277 exemplars • Acquisition–4 point scale

o Subset of exemplars (Clusters 02. 03, 06) ! o Subset of exemplars (Clusters 04, 05) !

Total ratings Versions: E & F 140 137

ALL VERSIONS

• Predicted changes in the profession in next 5 years • Professional and Demographic Background 19 19 19 19 19 19


VERSIONS A & B––Knowledge Statements and Content Areas Section 1. Knowledge Statements (78 statements) Respondents rated each statement on two of the three scales (Acquisition plus either Criticality or Usage).

Acquisition–At what point should the knowledge be acquired by psychologists? (4 point scale: Not necessary at any point; Primarily before initial licensure/registration; Primarily after initial licensure/registration, but before specialization; Primarily after specialization) Criticality–How critical is possessing the knowledge to protecting the patient/client/public from harm? (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Usage–What level best represents your use of this knowledge in your practice? (3 point scale: I do not have the knowledge; I recognize/recall the knowledge; I apply/interpret/integrate the knowledge)

Section 2. Knowledge Areas (8 content areas with definitions) Respondents rated each content area on two of the three scales (Frequency plus either Criticality or Importance).

Frequency–How frequently did you call upon knowledge from the content area in your practice during the past year? (4 point scale: Never or very rarely; Infrequently; Frequently; Very frequently) Criticality–How critical is the knowledge in this content area to protecting the patient/client/public from harm? (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Importance–How important is the knowledge in the content area to your practice as a psychologist during the past year? (4 point scale: Not important; Minimally important; Moderately important; Very important)

All respondents completing the two Knowledge versions were asked to describe any knowledge they call upon in practice that was not included in the survey. VERSIONS C & D––Competencies and Competency Clusters Section 1: Competencies (37 competencies) Respondents rated each competency on two of the three scales (Frequency plus either Criticality or Importance).


Frequency–How frequently did you perform the competency in your practice during the past year? (4 point scale: Never or very rarely; Infrequently; Frequently; Very frequently) Criticality–How critical is the competency to optimizing outcomes for the patient/client/public (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Importance–How important was performance of the competency to your practice as a psychologist during the past year? (4 point scale: Not important; Minimally important; Moderately important; Very important)

Section 2: Competency Clusters (6 clusters with definitions) Respondents rated all 6 clusters on all 3 scales.

Frequency–How frequently did you perform the competencies in this cluster in your practice during the past year? (4 point scale: Never or very rarely; Infrequently; Frequently; Very frequently) Criticality–How critical are the competencies in this cluster to optimizing outcomes for the patient/client/public? (4 point scale: Not critical; Minimally critical; Moderately critical; Highly critical) Importance–How important was performance of the competencies in this cluster to your practice during the past year? (4 point scale: Not important; Minimally important; Moderately important; Very important)

VERSION E—Behavioral Exemplars in Competency Clusters 02, 03, 06 (140 exemplars) VERSION F––Behavioral Exemplars in Competency Clusters 04, 05 (137 exemplars)

Section 1. Behavioral Exemplars Respondents rate a subset of the 277 exemplars on the following scale:

Acquisition–At what point should a psychologist be able to demonstrate this behavior? (4 point scale: During entry-level supervised practice (practicum); During advanced-level supervised practice (internship); At entry to practice through 3 years of independent practice; After 3 years of independent practice)

ALL VERSIONS––Demographic and Professional Information and Open-Ended Questionnaire Respondents answered 19 questions regarding their background, education, and practice. Respondents were also asked to comment upon what they believe to be the long-range changes occurring in the practice of psychology.


Pilot Test of Online Survey Instrument A draft Web-based Survey of the Practice of Psychology – Pilot Version, (referred to as the Pilot Survey) was developed to study the practice of licensed psychologists in the United States and Canada to explore the professional competencies and behaviors of psychologists and validate the content areas and knowledge required for practice. In Spring 2009, PES implemented the test of the Pilot Survey to identify question defects such as ambiguous items and to identify respondents’ cognitive difficulties as they formed answers to Pilot Survey questions. Members of the PATF were invited to nominate licensed psychologists to participate in the online pilot test. The members were asked to identify individuals in a variety of practice areas and employment settings, and to personally invite the nominees to participate in the pilot test. As had been the case with each of the previously described complementary data collection methods, the final pool of participants also included some psychologists nominated for, but not appointed to, the PATF. Finally, the PAAC and the PATF were invited to participate in the pilot test. Participants in the online pilot test were requested to complete the Pilot Survey and to critique each section to identify any unclear elements. As a check on the length of time needed to complete the Pilot Survey, participants were asked to record how long it took them to complete the survey. Appendix 6 contains a copy of the instructions provided to the participants in the mail-based pilot test of the Pilot Survey and a list of the participants. Based on the results of the Pilot Test and its own review, the PAAC approved the form and content of the final version of the online survey materials. See Appendix 7 for screen captures of the Survey of the Practice of Psychology (referred to as the Survey)—used in the large-scale validation of the practice analysis of licensed psychologists in the United States and Canada.

Develop the Sampling Plan for the Conduct of the Survey

PES prepared a sampling plan for the conduct of a large-scale validation survey of the practice of licensed psychologists in the United States and Canada. During the Spring of 2009, the PAAC reviewed and approved the general strategy, and PES implemented the plan to develop the sample. The sampling plan for dissemination of the survey was designed to (a) generate a sample of licensed psychologists from all ASPPB member jurisdictions in the United States and Canada, and (b) ensure the representation of licensed psychologists at or near the entry level of the profession. PES implemented the plan based on the following specific information elements:

• The target sample size was approximately 5,000 licensed, practicing psychologists.

• There was no comprehensive database of all licensed psychologists in either the United States or Canada. An e-mail letter outlining the study goals and purposes and signed by the ASPPB president was sent to the jurisdictional administrators of each of the 64 ASPPB member jurisdictions with a request for a sample of psychologists.


• One hundred and twenty psychologists were requested for the sample from the top third

most populated jurisdictions in the U.S. and Canada. Eighty psychologists were requested for the sample from the middle third most populated jurisdictions, and 40 psychologists were requested to be sampled from the smallest third of the jurisdictions.

• The sample should ensure equal representation of recently licensed psychologists (i.e.,

licensed in 2006 or after) and less-recently licensed psychologists (i.e., licensed in 2005 or earlier). Where possible, the ASPPB-member boards were asked to sample equal numbers of recently and less-recently licensed psychologists, divided between those licensed or registered within the past three years and those licensed or registered more than three years ago.

The results of the request for the samples of licensed/registered psychologists were as follows:

• ASPPB-member jurisdictions in the United States and Canada varied considerably in their ability to select a specific sample of licensed practicing psychologists from among the population of all licensed psychologists in the jurisdiction. Of those providing information, most jurisdictions were able to sample psychologists or supply databases sorted by year of licensure and practice status (active/inactive).

• In 11 cases, the jurisdiction was able to provide a sample or database that included e-mail addresses. In nine cases, the jurisdiction was able to provide e-mail addresses for some members of the sample, and mailing addresses for other members of the sample. In 38 cases, the jurisdictions were able or willing to supply mailing addresses for the sample.

• Due to privacy concerns and regulations governing the dissemination of members’ contact information, one jurisdiction drew the sample and disseminated the invitation using its own e-mail database. Another jurisdiction sent out e-mails to members directing them to an online sign-up form on which they could register for the study.

• One jurisdiction declined to participate entirely.

• Three jurisdictions did not respond in any way to repeated attempts to contact them.

• In some cases, the jurisdictions drew the requested sample for PES. In other instances, the jurisdictions provided their entire databases and PES drew the sample.

Because some of the smaller jurisdictions were unable to supply the minimum number of names and because some jurisdictions either declined to participate or failed to respond to requests to participate, the final sample for the survey consisted of 4,732 licensed/registered psychologists.


Implement the Practice Analysis Survey PES prepared a plan for the conduct of the large-scale survey of the practice of licensed psychologists in the United States and Canada. During 2008 through the Spring of 2009, the PAAC reviewed and approved the elements of the plan. Following the completion of the Pilot Survey, the development and creation of the online Survey and all related materials, and the implementation of all procedures required in connection with assembling the sample, PES implemented the plan. Appendix 8 contains copies of all the materials used in the Survey invitation mailings. The delivery method of the survey invitation depended on whether PES had received mailing addresses or e-mails from jurisdictional administrators. The survey was disseminated as follows:

1. In June 2009, potential respondents receiving the survey via U.S. Mail or Canadian Postal Service delivery received a letter from the President of the ASPPB describing the nature and scope of the project and inviting their participation in the data collection efforts. The invitation contained a personalized, password-protected link to the online survey.

2. In June 2009, potential respondents for whom PES had e-mail addresses received an e-

mail invitation from the President of the ASPPB describing the nature and scope of the project and inviting their participation in the data collection efforts. The invitation contained an embedded password-protected link to the online survey.

3. Approximately two weeks later, each potential member of the sample who had not yet

completed the survey received a reminder letter or e-mail invitation with the same information and link.

4. Finally, approximately one week later, each non-respondent was sent a final reminder

invitation to the survey.

Perform the Data Analyses, Develop Preliminary Test Specifications for the EPPP, and Discuss Implications of Findings Regarding the Assessment of Professional Competence

In Summer 2009, PES prepared preliminary descriptive statistical analyses on the basis of the demographic and professional data supplied by the respondents. The results of the analyses were presented at a meeting of the PATF held in July 2009. At that meeting, and following a review of the demographic and professional characteristics of the respondents to the survey, the PATF recommended the implementation of additional outreach efforts to Canadian psychologists so as to increase their representation in the sample. Subsequent to the meeting, members of the PAAC initiated renewed outreach efforts to those jurisdictions not previously participating in the survey and implemented a second round of surveys to a limited number of Canadian jurisdictions. As a result of these efforts, an additional 33 Canadian respondents were included in the survey In addition, the PATF suggested that ASPPB engage in the conduct of two supplemental outreach efforts to specific populations of psychologists—industrial/organizational (I/O) psychologists and psychologists with expertise in the assessment of competency, particularly competency of entry-level psychologists. ASPPB authorized PES to conduct two additional data


collection initiatives with two newly developed samples. The survey of I/O psychologists was accomplished with the assistance of the Society for Industrial and Organizational Psychologists (SIOP), which provided e-mail addresses for its voting members. The survey of educators was accomplished with the assistance of the Association of Psychology Postdoctoral and Internship Programs (APPIC), which provided e-mail addresses for three cohorts: subscribers (Directors of academic programs); Internship Training Directors; and Post-Doctoral Training Directors. Members of the PAAC and the PATF were especially helpful in establishing these liaisons between PES and these organizations. The survey of I/O psychologists, the results of which will be presented in detail in a subsequent report, included 157 respondents. They responded to all six survey versions –– two rating knowledge content and statements, two rating clusters and competencies, and two rating behavioral exemplars. The survey of APPIC respondents, the results of which will also be presented in detail in a subsequent report, included 37 directors of academic programs, 15 post-doctoral training directors, and 105 internship directors. Because this supplementary sample was expected to have expertise in competency assessment and the acquisition of demonstrated professional competency, respondents completed either one of the two competency versions or one of the two behavioral exemplars versions of the survey.

Complementary Data Collection Related to Competency Assessment

Following the conduct of PATF Meeting 3 and prior to the fourth and final meeting of the Task Force, PES implemented two final complementary data collection initiatives (i.e., focus panels and surveys) related to the enhancement of the EPPP and the implementation of additional competency-based assessments to complement the EPPP. Focus Panels, Round 2 PES conducted a second round of focus panels and interviews to elicit information from targeted stakeholders regarding the feasibility of implementing competency assessments as part of the licensure/registration process. In one initiative, educators in the United States and Canada participated in a virtual (teleconference) focus panel or a telephone interview. In a separate initiative, key ASPPB stakeholders, including jurisdictional registrars and those involved in licensure or registration of psychologists, participated in a combined in-person/virtual focus panel during the 2009 ASPPB Annual Meeting, wherein members of the Advisory Committee moderated the panel and PES participated via teleconference. Participants were asked their views about how the validated competencies might be assessed either at or before for entry to practice, and the feasibility of implementing competency-based assessments into the licensure/registration of psychologists. A complete report of the results of the Round 2 Focus Panels may be found in Appendix 9. Survey regarding Enhanced EPPP and Alternative Assessment Methodologies Members of the Advisory Committee and Task Force were surveyed regarding possible enhancements to the existing computer-based EPPP, and about the most effective methodologies to assess competence.


PATF/PAAC Meeting 4 At the final combined meeting of the PATF and PAAC held in November 2009, the results of the augmented target sample survey (with additional Canadian respondents) and the two supplementary surveys were presented, as well as the results of the second round of focus panels and telephone interviews. The Task Force revised the method by which the Frequency and Criticality ratings for the knowledge statements were combined to develop weightings for these statements in the test specifications. They recommended that a subset of knowledge statements be edited to enhance their clarity. The results related to the expanded target sample data analyses are presented in the next section of this report, including the Task Force’s recommendations for revised test specifications for the EPPP. Detailed reports of the two supplementary surveys will be presented in the future. During the meeting, Task Force and Advisory Committee members also explored potential enhancements to the current EPPP by use of alternative items types, and reviewed complementary methodologies for assessing competence. These recommendations are presented later in this report.

Develop Final Recommendations Regarding the Test Specifications Underlying the EPPP and

the Final Report PES participated in a conference telephone call with the ASPPB Board of Directors to review the recommendations from PATF/PAAC Meeting 4 regarding the revision of the test specifications underlying the construction of the EPPP. Members of the Board approved the recommended test specifications, including percentage allocations for questions at the content area level, and guidelines for the selection of questions related to the revised and rank-ordered knowledge statements. In Spring 2010, PES submitted a draft final report to the members of the PAAC. Based on feedback from the PAAC, PES revised the report for review and approval by the ASPPB Board of Directors.


Summary of Results for U.S and Canadian Respondents in Target Sample This section presents information on the: (a) return rate for the respondents; (b) demographic and professional description of the samples; (c) comparison of the samples to the samples described in previous practice analysis studies and in other large-scale studies of the profession; (d) ratings of content areas, knowledge statements, competency clusters, competencies, and behavioral exemplars; (e) qualitative comments made by the respondents; (f) content-based profiles of practice; (g) hypothetical and final test specifications for content areas and knowledge statements; and (h) discussions related to the development of assessments of competence.

Return Rate Table 1 presents information regarding the return rate of completed surveys for U.S. and Canadian licensed psychologists, and for the total sample. The rates were based on the number of potential respondents eligible to be included in the sample rather than the number of invitation letters mailed to potential respondents. The number eligible was defined as the total number of potential survey respondents, minus addressees who were deceased or confirmed as no longer practicing, or whose materials were returned as undeliverable. Inspection of Table 1 indicates that the overall return rate for the Survey was about 26%. A higher percentage of Canadian licensed psychologists (53%) completed and returned the Survey than did licensed U.S. psychologists (24%).

Table 1 Response Rate

Country Sent Undeliverable Valid Responses Rate

United States 4375 230 4145 991 23.9%

Canada 367 12 355 189 53.2%

Total 4742 242 4500 1180 26.2%

Table 2 presents information on the total number and percent of respondents rating each of the six versions of the survey. About 200 respondents rated each of the versions of the survey.

Table 2 Responses by Survey Version

n %

Knowledge - Acquisition and Criticality 183 16%

Knowledge - Acquisition and Usage 203 17%

Competencies - Frequency and Criticality 196 17%

Competencies - Frequency and Importance 207 18%

Exemplars - Clusters 02, 03, 06 194 16%

Exemplars - Clusters 04, 05 197 17%

Total 1180 100%


Table 3 documents the number and percent of the total sample by survey version and country. As can be seen, respondents from the U.S. were slightly more likely to complete the survey versions containing knowledge statements while respondents from Canada were slightly more likely to complete the survey versions containing the exemplars of competencies.

Table 3 Responses to Survey Version by Country

United States Canada

n % n %

Knowledge - Acquisition and Criticality 161 16% 22 12%

Knowledge - Acquisition and Usage 173 17% 30 16%

Competencies - Frequency and Criticality 162 16% 34 18%

Competencies - Frequency and Importance 177 18% 30 16%

Exemplars - Clusters 02, 03, 06 159 16% 35 19%

Exemplars - Clusters 04, 05 159 16% 38 20%

Total 991 100% 189 100%

Table 4 presents information regarding respondents by U.S. and Canadian ASPPB-member jurisdictions, including the number of eligible respondents completing the survey from each jurisdiction and their percentage of the respondent pool from the US and Canada, respectively. All but two U.S. and one Canadian ASPPB-member jurisdictions were represented in the final sample of returns. The percentage of returns by jurisdiction varied from 0% to 100%.

Table 4 Respondents by Jurisdiction

n % of US respondents

Alabama 24 2.4%

Alaska 11 1.1%

Arizona 27 2.7%

Arkansas 14 1.4%

California 32 3.2%

Colorado 17 1.7%

Connecticut 12 1.2%

Delaware 5 .5%

District of Columbia 12 1.2%

Florida 23 2.3%

Georgia 33 3.3%


Guam 0 0.0%

Hawaii 1 .1%

Idaho 9 .9%

Illinois 32 3.2%

Indiana 21 2.1%

Iowa 18 1.8%

Kansas 22 2.2%

Kentucky 24 2.4%

Louisiana 20 2.0%

Maine 6 .6%

Maryland 31 3.1%

Massachusetts 32 3.2%

Michigan 15 1.5%

Minnesota 25 2.5%

Mississippi 22 2.2%

Missouri 37 3.7%

Montana 6 .6%

Nebraska 18 1.8%

Nevada 17 1.7%

New Hampshire 5 .5%

New Jersey 16 1.6%

New Mexico 14 1.4%

New York 40 4.0%

North Carolina 24 2.4%

North Dakota 10 1.0%

Ohio 25 2.5%

Oklahoma 18 1.8%

Oregon 22 2.2%

Pennsylvania 38 3.8%

Puerto Rico 33 3.3%

Rhode Island 10 1.0%

South Carolina 12 1.2%

South Dakota 7 .7%

Tennessee 26 2.6%

Texas 30 3.0%

US Virgin Islands 0 0.0%

Utah 3 .3%

Vermont 5 .5%

Virginia 20 2.0%

Washington 27 2.7%


West Virginia 16 1.6%

Wisconsin 16 1.6%

Wyoming 8 .8%

Total 991 100.0%

n % of Canadian respondents

Alberta 26 13.8%

British Columbia 38 20.1%

Newfoundland and Labrador 0 0.0%

Manitoba1 4 2.1%

New Brunswick 15 7.9%

Nova Scotia 16 8.5%

Ontario 55 29.1%

Prince Edward island 6 3.2%

Quebec2 1 .5%

Saskatchewan 28 14.8%

Total 189 100.0% 1 While no volunteers from this jurisdiction specifically signed up to participate on an online collector, four respondents sampled from other jurisdictions indicated Manitoba to be their primary employment location 2 While this jurisdiction declined to provide a sample for the survey, one respondent sampled from other jurisdictions indicated Quebec to be his or her primary employment location

Respondent Demographic and Professional Characteristics

This section presents information on the demographic and professional characteristics of U.S. and Canadian respondents, and recently and less-recently licensed/registered respondents, as derived from the responses to the section of the survey focused on Demographic and Professional Information. Selected comparisons to the demographic and professional information reported in the ASPPB’s 1995 and 2003 practice analysis studies are presented, as available, as are comparisons to other demographic and professional characterizations of the profession. Table 5 documents the number and percent of the total sample of respondents from the U.S. (84%) and Canada (16%), and Table 6 documents the number and percent of the total sample of recently (36%) and less-recently licensed/registered respondents (64%). The composition of the sample of respondents is consistent with the sampling plan which had been designed to represent respondents in general proportion to the population by country and to over-represent recently-licensed/registered psychologists (defined as those licensed/registered within the most recent three-year period, 2007–2009).


Table 5 Respondents by Country

n %

United States 991 84%

Canada 189 16%

Total 1180 100%

Table 6 Respondents by Years of Licensure/Registration

n %

1 – 3 424 36%

4+ 738 64%

Total 1162 100%

As documented in Table 7, respondents from the U.S. were more likely (39%) to be recently licensed/registered than respondents from Canada (23%).

Table 7 Years of Licensure/Registration by Country


n % n %

0 10 1% 3 2%

1 – 3 380 39% 44 23%

4+ 597 60% 141 75%

Total 987 100% 188 100%

While respondents in the U.S. were more likely to have been fully licensed/registered to practice psychology in the years 2007 – 2009, about 40% of both the U.S. and the Canadian sample were fully licensed/registered to practice in 1999 or earlier (see Table 8).


Table 8 Year Initially Fully Licensed/Registered to Practice Psychology by Country


n % n %

Before 1970 15 2% 1 1%

1970 – 1979 79 8% 14 7%

1980 – 1989 124 13% 19 10%

1990 – 1999 164 17% 42 22%

2000 – 2003 106 11% 24 13%

2004 50 5% 17 9%

2005 57 6% 19 10%

2006 116 12% 21 11%

2007 104 11% 18 10%

2008 129 13% 8 4%

2009 43 4% 5 3%

Total 987 100% 188 100%

Information about the highest level of education in psychology beyond a Bachelor’s degree is presented in Table 9. Inspection of Table 9 indicates that respondents in the U.S. were more likely to have earned a doctorate than respondents in Canada. In the U.S., about 71% of the U.S. respondents had earned either a doctorate in philosophy (Ph.D.) or education (Ed.D.), and 27% had earned a doctorate in psychology (Psy.D.), whereas in Canada, 54% of the respondents had earned the doctorate in either philosophy or education, 7% had earned the doctorate in psychology, and 26% had earned a Master’s degree in psychology.

Table 9 Highest Level of Education in Psychology Beyond a Bachelor's Degree by Country


n % n %

Master‘s degree 6 1% 49 26%

Graduate work beyond Master‘s degree (at least 12 semester hours or equivalent) 5 1% 9 5%

Coursework for Doctorate completed 0 0% 8 4%

Psy.D. 264 27% 14 7%

Ed.D. 17 2% 2 1%

Ph.D. 675 69% 99 53%

Other degree 18 2% 7 4%

Total 985 100% 188 100%


A small number of respondents indicated earning an Other degree as their highest degree in psychology. A review of their specific responses indicated that the degrees were most likely to be post-doctoral certificates or diplomas. As documented in Table 10, both U.S. (84%) and Canadian (94%) respondents were most likely to be awarded their highest degree in psychology from a university (including an affiliated school of professional psychology).

Table 10 Institution Where Awarded Highest Degree

in Psychology by Country


n % n %

Free-standing school of professional psychology 157 16% 12 6%

University (including an affiliated school of professional psychology) 828 84% 175 94%

Total 985 100% 187 100%

Table 11 and Table 12 present information about the highest degree in psychology earned and the institution from which the degree was earned from the perspective of years of experience as a licensed/registered psychologist. As seen in Table 11, recently licensed/registered psychologists are less likely to have earned a Master’s degree or an Other degree and more likely to have earned a Doctorate in psychology than less-recently licensed psychologists. Consistent with the latter, recently licensed/registered psychologists are more likely to have graduated from a free-standing school of professional psychology than a less-recently licensed/registered psychologist (see Table 12).

Table 11 Highest Level of Education in Psychology Beyond a Bachelor's Degree

by Years of Licensure/Registration

1 - 3 4+

n % n %

Master‘s degree 13 3% 41 6%

Graduate work beyond Master‘s degree (at least 12 semester hours or equivalent) 1 0% 13 2%

Coursework for Doctorate completed 1 0% 7 1%

Psy.D. 149 35% 125 17%

Ed.D. 1 0% 18 2%

Ph.D. 257 61% 508 69%

Other degree 2 0% 23 3%

Total 424 100% 735 100%


Table 12

Institution Where Awarded Highest Degree in Psychology by Years of Licensure/Registration

1 - 3 4+

n % n %

Free-standing school of professional psychology 78 18% 88 12%

University (including an affiliated school of professional psychology) 346 82% 646 88%

Total 424 100% 734 100%

Table 13 and Table 14 document the major area of training in the graduate program from which U.S. and Canadian respondents received their highest degree in psychology, and their current major area of practice, respectively. Regardless of country, respondents were more likely to have been trained in the major areas of clinical, counseling, and educational psychology than they were to be currently practicing in those major areas; and were more likely to be currently practicing in the major areas of clinical neuropsychology, forensic psychology, geropsychology, health psychology, and rehabilitation psychology than to have initially been trained in those major areas.


Table 13 Major Area of Training by Country


n % n %

Clinical Psychology 562 57% 84 45%

Clinical Child Psychology 63 6% 14 7%

Clinical Neuropsychology 30 3% 8 4%

Cognitive Psychology 6 1% 1 1%

Community Psychology 6 1% 1 1%

Comparative Psychology 0 0% 2 1%

Counseling Psychology 170 17% 26 14%

Developmental Psychology 11 1% 5 3%

Educational Psychology 10 1% 18 10%

Environmental Psychology 0 0% 0 0%

Experimental Psychology 5 1% 1 1%

Forensic Psychology 2 0% 0 0%

General Psychology/Methods & Systems 2 0% 0 0%

Geropsychology 3 0% 0 0%

Health Psychology 22 2% 2 1%

Industrial/Organizational Psychology 8 1% 0 0%

Neurosciences 1 0% 2 1%

Perception/Learning 2 0% 0 0%

Personality Psychology 2 0% 1 1%

Physiological Psychology/Psychobiology 3 0% 0 0%

Psychopharmacology 1 0% 0 0%

Quantitative/Mathematical/Psychometrics/Statistics 0 0% 0 0%

Rehabilitation Psychology 2 0% 0 0%

School Psychology 60 6% 17 9%

Social Psychology 4 0% 3 2%

Sports Psychology 0 0% 0 0%

Other 9 1% 2 1%

Total 984 100% 187 100%


Table 14 Current Major Area of Practice by Country


n % n %



























Other 35 4% 3 2%

Total 962 100% 187 100%

Respondents who indicated that they had been trained in an Other area were most likely to have trained in a combined program including two of the specifically delineated areas. Respondents who indicated that they were currently practicing in an Other area were most likely to indicate that they were in administration/management. U.S. respondents were more likely to indicate an Other area of training than Canadian respondents. Only one area—pediatric psychology—was specifically identified by more than two respondents.


Consistent with the data regarding major area of training and current major area of practice, Table 15 documents the number and percent of U.S. and Canadian respondents who have participated in formal post-doctoral specialization or re-specialization. As can be noted in Table 15, nearly one third of the U.S. respondents have participated in such training and 17% of Canadian respondents have done so as well.

Table 15 Participated in Formal Post-Doctoral Specialization or

Re-specialization Program by Country


n % n %

Yes 303 31% 32 17%

No 679 69% 154 83%

Total 982 100% 186 100%

Viewed from the perspective of years of experience as a licensed/registered psychologist, Table 16 and Table 17 document a somewhat consistent picture. Recently licensed respondents are more likely to have trained in clinical, clinical child, and school psychology, and less likely to have trained in community, counseling, developmental, educational, experimental, industrial/organizational, and social psychology than less-recently licensed/registered respondents.

Table 16 Major Area of Training by Years of Licensure/Registration

1 – 3 4+

n % N %
















1 – 3 4+

n % N %













Other 4 1% 7 1%

Total 422 100% 734 100%

Regardless of year of initial licensure/registration, respondents were about equally likely to be currently practicing in clinical psychology, clinical neuropsychology , forensic psychology, geropsychology, health psychology, and rehabilitation psychology (see Table 17). Recently licensed/registered psychologists were more likely to be practicing clinical child psychology and counseling psychology than less-recently licensed/registered respondents, and less likely to be practicing school psychology than less-recently licensed/registered respondents.

Table 17 Current Major Area of Practice by Years of Licensure/Registration

1 - 3 4+

n % n %













1 - 3 4+

n % n %
















Other 10 2% 28 4%

Total 414 100% 721 100%

Finally, recently licensed/registered respondents are somewhat more likely to have participated in formal post-doctoral specialization or re-specialization than less-recently licensed/registered respondents (see Table 18). This finding is perhaps more remarkable given the fact that the less-recently licensed/registered respondents may not have had sufficient time to complete the eligibility requirements for formal post-doctoral specialization or re-specialization.

Table 18 Participated in Formal Post-Doctoral Specialization or Re-specialization Program

by Years of Licensure/Registration

1 – 3 4+

n % n %

Yes 128 30% 201 27%

No 293 70% 531 73%

Total 421 100% 732 100%

Appendix 10 documents the areas wherein respondents indicated where they had participated in formal post-doctoral specialization or re-specialization. The area of specialization or re-specialization most frequently cited was neuropsychology, with 54 respondents obtaining post-doctoral training in some aspect of this discipline.


Respondents indicated whether or not they had earned a certification from the American Board of Professional Psychology (ABPP). As shown in Table 19, small numbers of respondents indicated they had earned such a certification; as expected, more U.S. respondents had earned an ABPP certification than had Canadian respondents.

Table 19 Certified by ABPP by Country


n n% n n%

Yes 47 5% 6 3%

No 936 95% 181 97%

Consistent with the experience requirements for ABPP certification, less-recently licensed/registered respondents were more likely to have earned an ABPP certification than recently licensed/registered respondents.

Table 20 Certified by ABPP by Years of Experience

1 – 3 4+

n % n %

Yes 8 2% 44 6%

No 413 98% 690 94%

Total 421 100% 734 100%

Table 21and Table 22 document the specific areas wherein U.S. and Canadian respondents, and the recently and less-recently licensed/registered respondents had earned ABPP certification. As documented in Table 21, U.S. respondents were most likely to have earned certification in clinical psychology (51%), clinical neuropsychology (25%), or clinical health psychology (12%). The very few Canadian respondents who had earned an ABPP certification were most likely to have done so in clinical psychology and counseling psychology.

Table 21 Area(s) where certified by Country


n % n %

Cognitive and Behavioral Psychology 4 7% 0 0%

Clinical Child and Adolescent Psychology 4 7% 1 13%

Clinical Health Psychology 7 12% 1 13%




n % n %


Counseling 3 5% 3 38%

Couple & Family Psychology 2 4% 1 13%


Group Psychology 1 2% 1 13%

Organizational Business Consulting Psychology 0 0% 0 0%

Psychoanalysis Psychology 1 2% 0 0%



As seen in Table 22, more-recently licensed/registered respondents are more likely to have earned their ABPP certification in clinical psychology than in any other area.

Table 22 Area(s) where certified by Years of Experience

1 – 3 4+

n % n %

Cognitive and Behavioral Psychology 0 0% 4 8%

Clinical Child and Adolescent Psychology 0 0% 5 10%

Clinical Health Psychology 0 0% 8 15%



Counseling 0 0% 6 12%

Couple & Family Psychology 0 0% 3 6%


Group Psychology 0 0% 2 4%

Organizational Business Consulting Psychology 0 0% 0 0%

Psychoanalysis Psychology 0 0% 1 2%



Table 23 shows that U.S. and Canadian respondents work in a variety of primary employment settings within three overall categories—educational institutions/school systems (3 settings), human services settings (20 settings), and other employment settings (12 settings).


As can be seen in Table 23, approximately 61% of the U.S. respondents and 48% of the Canadian respondents indicated that their primary employment setting fell within the general category of human services settings (including individual practice and group psychological practice), whereas 21% and 23% of U.S. respondents and Canadian respondents, respectively, indicated an educational institution and school system as their primary employment setting. Fewer than 14% of indicated any other employment setting.

Table 23 Employment Setting by Country


n % n %

Educational Setting

–University (not medical or professional school) 116 12% 13 7%

– Four-year college 19 2% 0 0%

– Other educational setting 65 7% 29 16%

Human services setting

–Public general hospital 45 5% 25 13%

–Private general hospital 18 2% 0 0%

–City/county/state psychiatric hospital 31 3% 2 1%

–Not for profit, private psychiatric hospital 8 1% 0 0%

–For profit, private psychiatric hospital 1 0% 0 0%

–VA hospital 55 6% 0 0%

–Military hospital 9 1% 0 0%

–Individual private practice 196 20% 35 19%

–Group psychological practice 86 9% 9 5%

–Medical psychological group practice 24 2% 1 1%

–Outpatient mental health clinic, free-standing 20 2% 1 1%

–Community mental health center or clinic (CMHC) 53 5% 7 4%

–Independent Practice Association (IPA) 2 0% 0 0%

–Health Maintenance Organization (HMO) 3 0% 0 0%

–Preferred Provider Organization (PPO) 0 0% 0 0%

–Other managed care setting 0 0% 1 1%

–Counseling or guidance center (not university or college) 5 1% 3 2%

–Nursing home 6 1% 1 1%

–Specialized health service (e.g., substance abuse or developmental disabilities) 22 2% 7 4%

–Other human services setting 15 2% 9 5%

Other Employment Settings

–Self-employed, not private practice (e.g., consulting) 50 5% 17 9%

–Consulting firm 14 1% 7 4%



n % n %

–Private research organization or laboratory 1 0% 0 0%

–Government research organization or laboratory 3 0% 0 0%

–Business or industry (not consulting firm or research organization) 3 0% 0 0%

–Criminal justice system 37 4% 8 4%

–Military service (not military hospital) 1 0% 1 1%

–Federal government agency (other than above settings) 6 1% 0 0%

–State government agency (other than above settings) 15 2% 3 2%

–Local government agency (other than above settings) 4 0% 1 1%

–Other non–profit organization 22 2% 1 1%

Other non-educational or non-service setting not listed above 11 1% 4 2%

Not currently employed 21 2% 1 1%

Total 987 100% 186 100%

In general, the list of employment settings appears quite comprehensive. Fewer than 12% of the U.S. respondents and 16% of the Canadian respondents indicated any other setting as their primary employment setting. The majority of respondents in both the U.S. (88%) and Canada (87%) identified themselves as health service providers (see Table 24). This result is consistent with the fact that, in many jurisdictions, licensure/registration laws specifically designate licensees as health service providers.

Table 24 Health Service Provider by Country


n % N %

Yes 751 88% 139 87%

No 98 12% 20 13%

Total 849 100% 159 100%

As documented in Table 25, employment setting varies slightly in relation to years of licensure/registration. Recently licensed/registered respondents are more likely to be employed in educational-related settings than less-recently licensed/registered respondents (26% and 18%, respectively). In terms of human service settings, recently licensed respondents are far less likely to be in individual private practice and somewhat more likely to be in employed in a VA hospital or group psychological practice. Finally, in regard to other employment settings,


recently licensed/registered respondents are more likely to be employed in the criminal justice system or other non-profit setting, and less-recently licensed/registered respondents are more likely to be self-employed.

Table 25 Employment Setting by Years of Licensure/Registration

1 – 3 4+

n % n %

Educational setting

–University (not medical or professional school) 64 15% 64 9%

–Four-year college 13 3% 6 1%

–Other educational setting 33 8% 60 8%

Human services setting

–Public general hospital 28 7% 40 5%

–Private general hospital 6 1% 11 1%

–City/county/state psychiatric hospital 15 4% 17 2%

–Not for profit, private psychiatric hospital 6 1% 2 0%

–For profit, private psychiatric hospital 1 0% 0 0%

–VA hospital 30 7% 22 3%

–Military hospital 4 1% 5 1%

–Individual private practice 40 9% 189 26%

–Group psychological practice 40 9% 54 7%

–Medical psychological group practice 11 3% 13 2%

–Outpatient mental health clinic, free-standing 3 1% 18 2%

–Community mental health center or clinic (CMHC) 24 6% 36 5%

–Independent Practice Association (IPA) 1 0% 1 0%

–Health Maintenance Organization (HMO) 1 0% 2 0%

–Preferred Provider Organization (PPO) 0 0% 0 0%

–Other managed care setting 0 0% 1 0%

–Counseling or guidance center (not university or college) 3 1% 5 1%

–Nursing home 6 1% 1 0% –Specialized health service (e.g., substance abuse or developmental disabilities) 8 2% 21 3%

–Other human services setting 10 2% 14 2%

Other Employment Settings

–Self-employed, not private practice (e.g., consulting) 11 3% 55 7%

–Consulting firm 6 1% 15 2%

–Private research organization or laboratory 0 0% 1 0%

–Government research organization or laboratory 2 0% 1 0% –Business or industry (not consulting firm or research organization) 0 0% 3 0%


1 – 3 4+

n % n %

–Criminal justice system 24 6% 21 3%

–Military service (not military hospital) 0 0% 2 0%

–Federal government agency (other than above settings) 4 1% 2 0%

–State government agency (other than above settings) 9 2% 9 1%

–Local government agency (other than above settings) 1 0% 4 1%

–Other non-profit organization 12 3% 11 1%

Other non-educational or non-service setting not listed above 2 0% 13 2%

Not currently employed 5 1% 16 2%

Total 423 100% 735 100%

Less- and more-recently licensed/registered respondents were generally equally likely to describe themselves as a health service provider (see Table 26).

Table 26 Health Service Provider by Years of Experience

1 – 3 4+

n % n %

Yes 309 87% 572 89%

No 47 13% 69 11%

Total 356 100% 641 100%

Respondents identified a primary and a secondary theoretical orientation that best described their approach to professional practice. Table 27 and Table 28 document the primary and secondary theoretical orientations of the U.S. and Canadian respondents, respectively. In the U.S., more than one-half of the respondents indicated cognitive/behavioral psychology (58%), and 9% and 11% indicated interpersonal psychology and psychodynamic psychology as their primary orientation, respectively. No more than 6% of the U.S. respondents indicated any of the other four specifically-delineated theoretical orientations as primary. In terms of secondary theoretical orientation, 24% of the U.S. respondents indicated cognitive/behavioral and 18% indicated interpersonal, and about 10% to 12% each indicated behavioral, existential/humanistic, psychodynamic, and systems. In Canada, about two-thirds of the respondents indicated cognitive/behavioral psychology (66%), and 8% indicated interpersonal psychology as their primary orientation. No more than 4% of the Canadian respondents indicated any of the other specifically-delineated orientations as primary. In terms of secondary theoretical orientation, 20% and 19% indicated cognitive/behavioral and interpersonal psychology, respectively, and between 8% and 13% each indicated each of the remaining five specifically-delineated theoretical orientations.


In eliciting information about theoretical orientation, respondents were asked to select from seven designated categories and to refrain from selecting other orientation if at all possible. At least three respondents each indicated the following other orientations: Adlerian, DBT, developmental, eclectic, integrative, mindfulness, and neuropsychological.

Table 27 Primary Orientation by Country


n % n %

Behavioral 61 6% 6 3%

Cognitive/Behavioral 568 58% 123 66%

Existential/Humanistic 51 5% 12 6%

Interpersonal 86 9% 15 8%

Psychodynamic 110 11% 8 4%

Social learning 11 1% 4 2%

Systems 37 4% 4 2%

Other orientation 38 4% 8 4%

Not relevant 18 2% 5 3%

Total 980 100% 185 100%

Table 28 Secondary Orientation by Country


n % n %







Systems 96 10% 17 10%

Other 38 4% 7 4%


Total 916 100% 167 100%

The pattern of responses for recently licensed/registered respondents is similar to that of less-recently licensed/registered respondents in regard to cognitive/behavior psychology, although


slightly fewer of the recently licensed/registered respondents indicated that either psychodynamic psychology or systems psychology was their primary theoretical orientation (see Table 29). They were slightly more likely than the less-recently licensed/registered respondents to indicate behavioral psychology or interpersonal psychology as their primary orientation. The recently licensed/registered respondents were slightly more likely to indicate behavioral psychology and slightly less likely to indicate social learning, systems psychology or other orientation as their secondary theoretical orientation than were the less-recently licensed/registered respondents (see Table 30).

Table 29 Primary Orientation by Years of Licensure/Registration

1 - 3 4+

n % n %







Systems 9 2% 32 4%


Not relevant 4 1% 17 2%

Total 419 100% 731 100%

Table 30 Secondary Orientation by Years of Experience

1 – 3 4+

n % n %







Systems 35 9% 78 11%

Other 13 3% 32 5%


Total 391 100% 679 100%


Table 31 presents the self-described areas of expertise of U.S. and Canadian respondents. As can be seen, both U.S. and Canadian respondents were most likely to describe themselves as being experts in clinical psychology and in assessment/diagnosis/evaluation, and somewhat less likely to indicate clinical child psychology; counseling psychology; and treatment, intervention, and prevention. With very few exceptions, respondents indicated that they had expertise in one or more of each of the 57 specifically-delineated areas of expertise.

Table 31 Area(s) in Which Respondents Have Expertise by Country


n % n %

Assessment/Diagnosis/Evaluation 454 46% 105 56%

Behavioral Psychology 68 7% 9 5%

Biological Bass of Behavior 16 2% 0 0%

Career/Vocational Psychology 15 2% 4 2%

Child Psychology 52 5% 18 10%


Clinical Neuropsychology-Adult 84 9% 18 10%

Clinical Neuropsychology-Child 34 3% 7 4%



Cognitive-Affective Basis of Behavior 24 2% 3 2%



Competency Assessment 9 1% 1 1%

Consultation 35 4% 2 1%

Consulting Psychology 14 1% 5 3%

Consumer Psychology 1 0% 0 0%


Cross Cultural Studies/Issues 12 1% 2 1%




Ethical/Legal/Professional Issues 24 2% 4 2%



Gay/Lesbian Issues 27 3% 2 1%


Geropsychology/Aging 45 5% 2 1%



n % n %

Growth and Lifespan Development 11 1% 2 1%



Interdisciplinary Systems 0 0% 0 0%

Management 17 2% 2 1%

Marriage and Family Psychology 70 7% 9 5%

Mental Retardation/Developmental Disabilities 43 4% 5 3%

Multicultural Psychology/Ethnic minority issues 42 4% 2 1%

Neuroassessment 16 2% 2 1%

Neuroscience 5 1% 2 1%

Pain/Pain Management 35 4% 8 4%

Pediatric Psychology 38 4% 2 1%



Psychoanalysis 14 1% 1 1%

Psychology of Women 28 3% 1 1%



Recently emerged areas (e.g., collaborative care) 9 1% 0 0%


Research and Evaluation 16 2% 7 4%

Research Methods and Statistics 24 2% 1 1%


Social and Multicultural Basis of Behavior 5 1% 0 0%


Specialized Assessment Techniques 8 1% 2 1%


Substance Abuse 77 8% 4 2%

Treatment, Intervention, and Prevention 122 12% 35 19%

Comparisons between the self-described areas of expertise of recently and less-recently licensed/registered respondents are difficult to make in that newly licensed/registered psychologists may not have had sufficient opportunity to develop any additional areas of expertise.


Table 32 presents the wide array of experiences as a psychologist that U.S. and Canadian respondents had in the most recent three year period. In both the U.S. and Canada, the most frequently reported experiences were providing health/mental health services to a client pro bono; providing other services (e.g., expert witness, research, teaching, organization consultation) on a pro bono basis; and serving as a paid consultant to another institution, business, government agency, or organization. With only one exception (serving on a regulatory board), a minimum of 10% of the sample of Canadian respondents indicated that they had participated in each of the 13 specifically-delineated experiences. The pattern is similar for the sample of U.S. respondents, although the U.S. respondents appear to be slightly less likely than the Canadian respondents to engage in most of these experiences.

Table 32 Experiences as a Psychologist Within the Past Three Years by Country


n % n %

Served as the principal investigator or co-investigator on a research grant/contract 168 20% 36 23%

Provided health/mental health services to a client pro bono (free of charge) 547 67% 100 63%

Provided other services (e.g. expert witness, research, teaching, organization consultation) on a pro bono basis 332 40% 63 39%

Served as a paid consultant to another institution, business, government agency, or organization (exclude direct clinical)

275 33% 65 41%

Served as a consultant on a grant/contract 123 15% 19 12%

Served as an expert witness in a court proceeding 236 29% 38 24%

Appeared on television or radio as a psychologist 118 14% 39 24%

Served on a review group for a government agency or private foundation, to review grant proposals for research, training

61 7% 18 11%

Been involved in legislative/lobbying activities 127 15% 24 15%

Presented at an international convention 140 17% 42 26%

Attended an international convention 195 24% 94 59%

Participated in any professional psychology association activities as committee, board, or other governance group member

203 25% 62 39%

Served on a licensure or regulatory board 24 3% 24 15%

In considering participation in various activities by experience level, with only two exceptions, recently licensed/registered respondents are generally less likely to have participated when compared to less-recently licensed/registered respondents (see Table 33). Recently licensed/registered respondents were more likely to have presented or attended an international convention than the less-recently licensed/registered respondents.


Table 33

Experiences as a Psychologist Within the Past Three Years by Years of Experience

1 - 3 4+

n % n %

Served as the principal investigator or co-investigator on a research grant/contract 96 29% 107 17%

Provided health/mental health services to a client pro bono (free of charge) 198 59% 445 70%

Provided other services (e.g. expert witness, research, teaching, organization consultation) on a pro bono basis

133 40% 261 41%

Served as a paid consultant to another institution, business, government agency, or organization (exclude direct clinical)

90 27% 248 39%

Served as a consultant on a grant/contract 46 14% 96 15%

Served as an expert witness in a court proceeding 52 16% 219 34%

Appeared on television or radio as a psychologist 43 13% 113 18%

Served on a review group for a government agency or private foundation, to review grant proposals for research, training

27 8% 52 8%

Been involved in legislative/lobbying activities 31 9% 119 19%

Presented at an international convention 75 23% 103 16%

Attended an international convention 102 31% 182 28%

Participated in any professional psychology association activities as committee, board, or other governance group member

73 22% 188 29%

Served on a licensure or regulatory board 4 1% 43 7%

As documented in Table 34 and Table 35, in both the U.S. and Canada, the majority of respondents are female (62% and 70%, respectively), and the recently licensed/registered respondents are more likely to be female than less-recently licensed/registered respondents (72% and 58%).


Table 34 Sex by Country


n % n %

Female 603 62% 125 70%

Male 367 38% 53 30%

Total 970 100% 178 100%

Table 35 Sex by Years of Experience

1 - 3 4+

n % n %

Female 302 72% 415 58%

Male 115 28% 302 42%

Total 417 100% 717 100%

As documented in Table 36, about 85% of the U.S. respondents and 93% of the Canadian respondents described themselves as Caucasian/White. In both the U.S. and Canada, no more than 4% identified themselves as best described by any of the other racial/ethnic categories. More-recently licensed/registered respondents were less likely to describe themselves as Caucasian/White than less-recently licensed/registered respondents (see Table 36). Consistent with that finding, proportionately more-recently licensed/registered respondents were likely to describe themselves as each of the specifically-delineated racial/ethnic categories than less-recently licensed/registered respondents.

Table 36 Racial/Ethnic Background by Country


n % n %

African American/Black 35 4% 1 1%

Asian 30 3% 10 6%

Caucasian/White 804 85% 159 93%

Hispanic/Latino 53 6% 0 0%

Multi-ethnic 17 2% 1 1%

Native American/Inuit 10 1% 0 0%

Native Hawaiian or Pacific Islander 1 0% 0 0%

Total 950 100% 171 100%


Table 37 Racial/Ethnic Background by Years of Experience

1 - 3 4+

n % n %

African American/Black 19 5% 16 2%

Asian 24 6% 15 2%

Caucasian/White 333 81% 619 89%

Hispanic/Latino 22 5% 31 4%

Multi-racial 8 2% 10 1%

Native American/Inuit 5 1% 4 1%

Native Hawaiian or Pacific Islander 1 0% 0 0%

Total 412 100% 695 100%

As can be seen in Table 38, very few U.S. and Canadian respondents indicated that they had a disability as defined by the Americans with Disabilities Act. U.S. respondents were most likely to indicate a physical/orthopedic disability and Canadian respondents were most likely to indicate a deaf/hearing-related disability (see Table 39).

Table 38 Disability (as defined by the Americans with Disabilities Act) by Country


n % n %

Yes 36 4% 8 4%

No 920 96% 170 96%

Total 956 100% 178 100%

Table 39 Type of Disability by Country


n % n %

Blind/Visually Impaired 0 0% 0 0%

Deaf/Hard of Hearing 6 19% 3 43%

Physical/Orthopedic Disability 17 53% 2 29%

Learning/Cognitive Disability 6 19% 1 14%

Mental/Psychological Disorder 4 13% 0 0%

Other 4 13% 1 14%


Less-recently licensed/registered respondents were only slightly more likely to indicate that they had a disability than recently licensed respondents (see Table 40).

Table 40 Disability by Years of Experience

1 – 3 4+

n % n %

Yes 13 3.1% 30 4.1%

No 396 93.4% 680 92.1%

Total 409 100.0% 710 100.0%

Comparisons Between Target Sample and Samples Identified in Previous ASPPB Practice Analysis Studies and by Other Research Initiatives

During the final meeting of the PATF in November 2009, PES led a discussion about the degree to which the sample of respondents completing the survey presented a valid representation of the population of licensed/registered psychologists or any other formally drawn sample of licensed/registered psychologists. Accordingly, members of the PATF reflected on the degree to which the demographic and professional descriptions of the respondents in the current study matched their best understandings of the population—from the perspectives of their own experiences in training programs, as supervisors, as practitioners, as board members, and as regulators. Participants agreed that the description of the sample matched their best understandings of the demographic and professional characteristics of the members of the profession. For key professional background and demographic questions, PES compared the current data set with the practice analysis data sets collected in both 1995 (Greenberg, Smith, and Muenzen, 1996) and 2003 (Greenberg and Jesuitus). Across the span of 14 years and three studies of practice, specific trends were noted in regard to the types of institutions attended, major area of training, current area of practice, primary and secondary theoretical orientation, participation in formal post-doctoral specialization or re-specialization, and the identification of both sex and race/ethnicity. As seen in Table 41 although small in number and regardless of country, respondents were increasingly likely to indicate that they had been awarded their highest degree in psychology by a free-standing school of professional psychology. The percentage of respondents doing so doubled in the U.S. (8.6% versus 15.9%) and tripled in Canada (1.9% versus 6.4%) between 1995 and 2009 when the data was collected.


Table 41 Institution Where Awarded Highest Degree in Psychology by Study Year and Country

1995 2003 2009

United States Canada United States Canada United States Canada

n % n % n % n % n % n % Free-standing school of professional psychology

301 8.6% 14 1.9% 32 11.8% 3 4.8% 157 15.9% 12 6.4%

University (including an affiliated school of professional psychology)

3192 91.4% 706 98.1% 240 88.2% 59 95.2% 828 84.1% 175 93.6%

Total 3493 100.0% 720 100.0% 272 100.0% 62 100.0% 985 100.0% 187 100.0%

Table 42 and Table 43 document trends in regard to major areas of training and current major area of practice by study year, respectively. As seen in Table 42, in the U.S., there was a general increase in the percent of respondents identifying their major areas of training as: clinical psychology, clinical child psychology, clinical neuropsychology, health psychology, and school psychology, and a decrease in regard to both counseling psychology and educational psychology. The small number of respondents from Canada makes direct comparisons more difficult, although there was an increase in the percent of respondents identifying their major area of training as either educational psychology or school psychology. Table 43 documents the current major area of practice of the respondents—regardless of their initial training. In the U.S., there was a general increase in the percent of respondents identifying clinical child psychology, clinical neuropsychology, forensic psychology, geropsychology, and health psychology as their major area of practice between 1995 and 2009, and a decrease in the percent identifying clinical psychology and counseling psychology. Comparisons between Table 42 and Table 43 seem to indicate that major area of training may not predict current area of practice.


Table 42 Major Area of Training by Study Year and Country

1995 2003 2009


n % n % n % n % n % n %

Clinical Psychology 1896 54.1% 351 48.5% 153 56% 37 61% 562 57% 84 45%

Clinical Child Psychology 103 2.9% 31 4.3% 12 4% 1 2% 63 6% 14 7%

Clinical Neuropsychology 25 0.7% 11 1.5% 6 2% 1 2% 30 3% 8 4%

Cognitive Psychology 9 0.3% 7 1.0% 0 0% 1 2% 6 1% 1 1%

Community Psychology 11 0.3% 6 0.8% 0 0% 0 0% 6 1% 1 1%

Comparative Psychology 4 0.1% 20 2.8% 1 0% 0 0% 0 0% 2 1%

Counseling Psychology 739 21.1% 90 12.4% 44 16% 5 8% 170 17% 26 14%

Developmental Psychology 87 2.5% 20 2.8% 8 3% 3 5% 11 1% 5 3%

Educational Psychology 128 3.7% 34 4.7% 10 4% 5 8% 10 1% 18 10%

Environmental Psychology 0 0.0% 1 0.1% 0 0% 0 0% 0 0% 0 0%

Experimental Psychology 41 1.2% 32 4.4% 0 0% 2 3% 5 1% 1 1%

Forensic Psychology n/a n/a n/a n/a 2 1% 0 0% 2 0% 0 0% General Psychology/ Methods & Systems 29 0.8% 6 0.8% 4 1% 0 0% 2 0% 0 0%

Geropsychology 1 0.0% 0 0.0% 0 0% 0 0% 3 0% 0 0%

Health Psychology 11 0.3% 7 1.0% 3 1% 0 0% 22 2% 2 1% Industrial/Organizational Psychology 33 0.9% 12 1.7% 2 1% 0 0% 8 1% 0 0%

Neurosciences 4 0.1% 1 0.1% 0 0% 1 2% 1 0% 2 1%

Perception/Learning n/a n/a n/a n/a n/a n/a n/a n/a 2 0% 0 0%

Personality Psychology 14 0.4% 5 0.7% 1 0% 1 2% 2 0% 1 1% Physiological Psychology /Psychobiology 15 0.4% 12 1.7% 2 1% 2 3% 3 0% 0 0%

Psychopharmacology 1 0.0% 0 0.0% 1 0% 0 0% 1 0% 0 0% Quantitative/ Mathematical/ Psychometrics/ Statistics

4 0.1% 3 0.4% 0 0% 0 0% 0 0% 0 0%

Rehabilitation Psychology 13 0.4% 2 0.3% 0 0% 0 0% 2 0% 0 0%

School Psychology 163 4.7% 20 2.8% 0 0% 1 2% 60 6% 17 9%

Social Psychology 33 0.9% 12 1.7% 8 3% 1 2% 4 0% 3 2%

Sports Psychology n/a n/a n/a n/a n/a n/a n/a n/a 0 0% 0 0% Combined (e.g., counseling/school) 71 2.0% 19 2.6% 3 1% 0 0% n/a n/a n/a n/a

Other 54 1.5% 12 1.7% 4 1% 0 0% 9 1% 2 1%

Not specified 15 0.4% 10 1.4% 2 1% 0 0% n/a n/a n/a n/a

Total 3504 100.0% 724 100.0% 272 100% 61 100% 984 100% 187 100%


Table 43 Current Major Area of Practice by Study Year and Country

1995 2003 2009


n % n % n % n % n % n %

Clinical Psychology 1850 52.8% 309 42.7% 149 55% 30 49% 449 47% 59 32%

Clinical Child Psychology 259 7.4% 62 8.6% 21 8% 6 10% 119 12% 22 12%

Clinical Neuropsychology 140 4.0% 28 3.9% 17 6% 4 7% 56 6% 14 7%

Cognitive Psychology 12 0.3% 8 1.1% 0 0% 2 3% 8 1% 0 0%

Community Psychology 27 0.8% 4 0.6% 3 1% 0 0% 8 1% 3 2%

Comparative Psychology 2 0.1% 7 1.0% 1 0% 0 0% 1 0% 0 0%

Counseling Psychology 402 11.5% 63 8.7% 16 6% 4 7% 94 10% 16 9% Developmental Psychology 17 0.5% 6 0.8% 3 1% 1 2% 4 0% 3 2%

Educational Psychology 11 0.3% 21 2.9% 0 0% 0 0% 4 0% 10 5%

Environmental Psychology 3 0.1% 1 0.1% 0 0% 0 0% 0 0% 0 0%

Experimental Psychology 4 0.1% 2 0.3% 0 0% 0 0% 0 0% 0 0%

Forensic Psychology n/a n/a n/a n/a 11 4% 1 2% 51 5% 15 8% General Psychology/ Methods & Systems 9 0.3% 4 0.6% 1 0% 0 0% 0 0% 0 0%

Geropsychology 20 0.6% 5 0.7% 3 1% 0 0% 17 2% 0 0%

Health Psychology 82 2.3% 21 2.9% 4 1% 5 8% 46 5% 5 3% Industrial/Organizational Psychology 59 1.7% 19 2.6% 4 1% 0 0% 14 1% 4 2%

Neurosciences 9 0.3% 4 0.6% 0 0% 0 0% 0 0% 0 0%

Perception/Learning n/a n/a n/a n/a n/a n/a n/a n/a 1 0% 0 0%

Personality Psychology 6 0.2% 1 0.1% 0 0% 0 0% 1 0% 0 0% Physiological Psychology/Psychobiology 1 0.0% 0 0.0% 2 1% 0 0% 2 0% 0 0%

Psychopharmacology 1 0.0% 0 0.0% 0 0% 0 0% 2 0% 0 0% Quantitative/ Mathematical/ Psychometrics/ Statistics

5 0.1% 0 0.0% 0 0% 0 0% 1 0% 1 1%

Rehabilitation Psychology 52 1.5% 18 2.5% 2 1% 1 2% 6 1% 6 3%

School Psychology 102 2.9% 35 4.8% 5 2% 1 2% 40 4% 26 14%

Social Psychology 9 0.3% 8 1.1% 1 0% 0 0% 2 0% 0 0%

Sports Psychology n/a n/a n/a n/a n/a n/a n/a n/a 1 0% 0 0% Combined (e.g., counseling/school) 71 2.0% 21 2.9% 4 1% 1 2% n/a n/a n/a n/a

Other 115 3.3% 24 3.3% 8 3% 0 0% 35 4% 3 2%

Not specified 236 6.7% 53 7.3% 17 6% 5 8% n/a n/a n/a n/a

Total 3504 100.0% 724 100.0% 272 100% 61 100% 962 100% 187 100%


Major shifts have occurred in regard to the theoretical orientation of the survey respondents across the period encompassed by the three studies of practice. As documented in Table 44, the percentage of respondents identifying themselves as having a cognitive/behavioral orientation has increased in both the U.S. (from 44% to 58%) and Canada (from 43% to 66%). Conversely, the percentage of respondents identifying psychodynamic psychology as their primary theoretical orientation has decreased in both the U.S. (from 24% to 11%) and Canada (from 18% to 4%).

Table 44 Primary Theoretical Orientation by Study Year and Country

1995 2003 2009


n % n % n % n % n % n %

Behavioral 195 5.6% 38 5.2% 17 6% 4 6% 61 6% 6 3%

Cognitive/Behavioral 1539 43.9% 313 43.2% 144 53% 37 60% 568 58% 123 66%

Existential/Humanistic 241 6.9% 125 17.3% 13 5% 4 6% 51 5% 12 6%

Interpersonal 263 7.5% 33 4.6% 19 7% 1 2% 86 9% 15 8%

Psychodynamic 822 23.5% 129 17.8% 48 18% 11 18% 110 11% 8 4%

Social learning 56 1.6% 13 1.8% 3 1% 1 2% 11 1% 4 2%

Systems 225 6.4% 43 5.9% 12 4% 3 5% 37 4% 4 2%

Other orientation 110 3.1% 17 2.3% 13 5% 1 2% 38 4% 8 4% Not relevant/not specified 53 1.5% 13 1.8% 3 1% 0 0% 18 2% 5 3%

Total 3504 100.0% 724 100.0% 272 100% 0 0% 980 100% 185 100%

Finally, the years 1995 to 2009 represent a time of demographic shifts within the profession. In regard to sex, Table 45 documents the shift in the proportion of female respondents in both the U.S. and Canada—from a minority to a majority.

Table 45 Sex by Study Year and Country

1995 2003 2009


n % n % n % n % n % n %

Female 1525 43.5% 358 49.4% 138 50.7% 27 43.5% 603 60.8% 125 66.1%

Male 1953 55.7% 363 50.1% 132 48.5% 35 56.5% 367 37.0% 53 28.0%

Not specified 26 0.7% 3 0.4% 2 0.7% 0 0.0% 21 2.1% 11 5.8%

Total 3504 100.0% 724 100.0% 272 100.0% 62 100.0% 991 100.0% 189 100.0%


Finally, with regard to racial/ethnic background, across the period of the three studies, both U.S. and Canadian respondents are less likely to identify themselves as Caucasian/White (see Table 46). In the U.S. they were also more likely to identify themselves as either Hispanic/Latino or Asian, whereas in Canada, they were more likely to identify themselves as Asian.

Table 46 Racial/Ethnic Background by Study Year and Country

19951 20032 20093


n % n % n % n % n % n %

African American/Black 54 1.5% 4 0.6% 2 0.7% 0 0.0% 35 3.5% 1 .5%

Asian 32 0.9% 16 2.2% 2 0.7% 2 3.2% 30 3.0% 10 5.3%

Caucasian/White 3268 93.3% 642 88.7% 254 93.4% 57 91.9% 804 81.1% 159 84.1%

Hispanic/Latino 54 1.5% 10 1.4% 4 1.5% 2 3.2% 53 5.3% 0 .0% Multi-racial/multi-ethnic n/a n/a n/a n/a n/a n/a n/a n/a 17 1.7% 1 .5%

Native American/Inuit 20 0.6% 3 0.4% 0 0.0% 1 1.6% 10 1.0% 0 .0% Native Hawaiian or Pacific Islander n/a n/a n/a n/a n/a n/a n/a n/a 1 .1% 0 .0%

Other 48 1.4% 30 4.1% 3 1.1% 2 3.2% n/a n/a n/a n/a

Not specified 28 0.8% 19 2.6% 7 2.6% 0 0.0% 41 4.3% 18 9.6%

Total 3504 100.0% 724 100.0% 272 100.0% 62 100.0% 950 100.0% 189 100.0% 1 Categories in 1995 study:

American Indian/Alaskan Native Asian or Pacific Islander Black or African American Caucasian/White Hispanic/Latino Other Not specified

2 Categories in 2003 study: African-American Asian/Pacific Islander Caucasian, not of Hispanic Origin Native American Spanish/Hispanic Origin Other Not specified

3 Categories in 2009 study: African American/Black Asian Caucasian/White Hispanic/Latino Multi-ethnic Native American/Inuit Native Hawaiian or Pacific Islander Not specified

Results Related to Content- and Competency-based Delineations Content-based Delineation

Content areas. Table 47 presents for the total sample of respondents the mean, standard deviation, and number of respondents completing the Importance, Frequency, and Criticality ratings on the eight content areas, and the frequency distributions for the associated scale points. (See pages 17 to 18 of this report for a complete description of the rating scales.) As can be seen in Table 47, the Importance ratings for the eight content areas range from 2.6 for the Research Methods and Statistics content area, to 3.9 for the Ethical/Legal/Professional Issues content area, indicating that the content areas range from being minimally-to-moderately important to the practice of licensed/registered psychologists, to moderately-to-very important to


the practice of licensed psychologists. In fact, seven of the eight content areas were rated as moderately-to-very important to practice, and only one content area, Research Methods and Statistics, was rated as somewhat less important to practice. For the total sample of respondents, the mean Frequency ratings for the content areas range from 2.3 for the Research Methods and Statistics content area, to 3.4 for four content areas (Cognitive–Affective Bases of Behavior; Assessment and Diagnosis; Treatment, Intervention, Prevention, and Supervision; and Ethical, Legal, and Professional Issues), indicating that respondents call upon knowledge from the content areas from infrequently-to-frequently, to frequently-to-very frequently. In fact, consistent with the Importance ratings previously described, respondents call upon the knowledge from seven of the eight content areas moderately-to-very frequently, and the knowledge from one content area, Research Methods and Statistics, infrequently-to-frequently. For the total sample of respondents, the mean Criticality ratings for the eight content areas range from 2.6 for the Research Methods and Statistics content area, to 3.8 for the Ethical/Legal/Professional Issues content area, indicating that all of the content areas are rated as falling between minimally-to-moderately and moderately-to-highly critical to protecting the patient/client/public from harm. In general, the Criticality ratings were very similar to the Importance ratings, indicating that respondents both recognized the importance of the knowledge to their own practice and the critical contribution that the knowledge associated with the content area makes to protecting the patient/client/public from harm.


Table 47 Knowledge Content Areas

Frequency, Criticality, and Importance Ratings for Total Sample

Frequency Ratings Criticality Ratings Importance Ratings

Never/ rarely

Infre- quently

Fre- quently

Very Freq

Summary Statistics Not Min Mod High


Summary Statistics

% % % % n Mean SD % % % % n Mean SD % % % % n Mean SD


3% 24% 41% 32% 384 3.0 (.8) 1% 14% 44% 41% 180 3.3 (.7) 1% 19% 36% 44% 196 3.2 (.8)


1% 12% 36% 51% 383 3.4 (.7) 1% 11% 42% 47% 180 3.3 (.7) 0% 8% 30% 62% 197 3.5 (.6)


2% 18% 45% 35% 384 3.1 (.8) 1% 18% 38% 44% 180 3.3 (.8) 1% 15% 34% 50% 196 3.3 (.8)


3% 22% 37% 38% 384 3.1 (.8) 1% 14% 43% 43% 180 3.3 (.7) 2% 15% 32% 52% 198 3.3 (.8)

Assessment and Diagnosis — knowledge of (a) psychometrics, (b) assessment models and instruments, (c) assessment methods for initial status of and change by individuals, couples, families, groups, and organizations/systems, and (d)

2% 15% 24% 59% 384 3.4 (.8) 2% 8% 25% 65% 181 3.5 (.7) 1% 8% 21% 71% 197 3.6 (.6)



Never/ rarely

Infre- quently

Fre- quently

Very Freq



Summary Statistics

% % % % n Mean SD % % % % n Mean SD % % % % n Mean SD diagnostic classification systems and their limitations Treatment, Intervention, Prevention, and Supervision — knowledge of (a) individual, couple, family, group, organizational, or community interventions for specific problems/disorders in diverse populations, (b) intervention and prevention theories, (c) best practices and practice guidelines, (d) consultation and supervision models, and (e) evidence supporting efficacy and effectiveness of interventions

3% 16% 23% 59% 385 3.4 (.8) 1% 9% 25% 65% 181 3.5 (.7) 3% 8% 19% 71% 198 3.6 (.7)


19% 45% 25% 12% 384 2.3 (.9) 11% 36% 34% 19% 180 2.6 (.9) 12% 32% 37% 18% 198 2.6 (.9)


1% 10% 33% 56% 385 3.4 (.7) 2% 1% 11% 86% 181 3.8 (.5) 0% 3% 10% 88% 198 3.9 (.4)


Table 48 presents the responses of the recently and less-recently licensed/registered U.S. and Canadian respondents, including the mean and the standard deviation for the Frequency, ratings on the content areas. As can be seen, the Frequency ratings of the recently licensed/registered Canadian respondents were more likely to be higher than those of the U.S. respondents, whereas the Frequency ratings of the U.S. and Canadian less-recently licensed/registered respondents were likely to be relatively similar to each other. Table 49 compares the Criticality ratings of the recently and less-recently licensed/registered U.S. and Canadian respondents. With only two exceptions (Social and Cultural Bases of Behavior, and Research Methods and Statistics), recently licensed/registered Canadian respondents were more likely to see these eight content areas as critical to protecting the patient/client/public from harm. The ratings of less-recently licensed/registered Canadian respondents indicated that in all content areas, they were as likely or more likely than the U.S. respondents to rate the areas as making a moderate contribution to protecting the patient/client/public from harm. Regardless of year of licensure/registration, seven of the eight areas were seen as making at least a moderate contribution; only in the case of one content area—Research Methods and Statistics—did the ratings suggest that the area made a lower contribution to protecting the public/patient/client from harm. Finally, in regard to the Importance ratings, Table 50 indicates that both recently and less-recently licensed/registered Canadian respondents were as likely or somewhat less likely to see each content area as important to their own practice of psychology than were the U.S. respondents. Regardless of year of licensure/registration, seven of the eight content areas were rated as moderately important to practice. As was the case in regard to the Criticality ratings, only the importance ratings for the content area of Research Methods and Statistics fell between minimally-to-moderately important to practice.


Table 48 Knowledge Content Areas—

Frequency Ratings by Country and Years of Experience as Licensed/Registered Psychologist

Country and Years as Licensed/Registered. Psychologist

U.S. 1-3 years (n=380)

U.S. 4+ years (n=597)

Canada 1-3 years (n=44)

Canada 4+ years (n=)141

M SD M SD M SD M SD


3.0 (.8) 3.0 (.9) 3.6 (.7) 2.8 (.8)

Cognitive-Affective Bases of Behavior — knowledge of (a) cognition, (b) theories and empirical bases of learning, memory, motivation, affect, emotion, and executive function, and (c) factors that influence cognitive performance and/or emotional experience and their interaction

3.4 (.7) 3.3 (.8) 3.6 (.8) 3.4 (.7)


3.3 (.7) 3.1 (.8) 2.9 (1.1) 2.8 (.7)


3.1 (.8) 3.1 (.9) 3.3 (.9) 3.2 (.8)


3.4 (.8) 3.4 (.8) 3.7 (.7) 3.4 (.8)


3.6 (.7) 3.3 (.9) 3.5 (.8) 3.2 (.9)


2.3 (1.0) 2.3 (.9) 2.5 (1.1) 2.3 (.8)


3.6 (.6) 3.3 (.8) 3.7 (.7) 3.4 (.8)



Criticality Ratings by Country and Years of Experience as Licensed/Registered Psychologist

Country and Years as Lic./Reg. Psychologist

U.S. 1-3 years (n=380)

U.S. 4+ years (n=597)



M SD M SD M SD M SD


3.3 (.7) 3.2 (.7) 4.0 (.0) 3.2 (.7)


3.3 (.7) 3.3 (.7) 3.8 (.5) 3.6 (.5)


3.4 (.7) 3.2 (.8) 3.0 (1.2) 3.3 (.7)


3.3 (.7) 3.2 (.7) 3.8 (.5) 3.4 (.7)


3.6 (.7) 3.5 (.7) 4.0 (.0) 3.5 (.7)


3.6 (.7) 3.5 (.7) 3.8 (.5) 3.5 (.8)


2.8 (.9) 2.5 (.9) 2.5 (1.3) 2.6 (.8)


3.9 (.4) 3.8 (.6) 3.8 (.5) 3.8 (.6)



Importance Ratings by Country and Years of Experience as Licensed/Registered Psychologist

Country and Years as Lic./Reg. Psychologist

U.S. 1-3 years (n=380)

U.S. 4+ years (n=597)



M SD M SD M SD M SD


3.2 (.7) 3.3 (.8) 3.6 (.9) 2.9 (.8)


3.5 (.7) 3.6 (.6) 3.3 (1.0) 3.5 (.7)


3.4 (.7) 3.3 (.8) 2.8 (1.0) 3.1 (.9)


3.3 (.7) 3.3 (.8) 3.2 (1.0) 3.3 (.8)


3.6 (.7) 3.7 (.6) 3.3 (1.2) 3.6 (.6)


3.8 (.5) 3.5 (.8) 3.8 (.4) 3.5 (.7)


2.6 (.9) 2.7 (.9) 2.5 (1.0) 2.4 (.9)


3.9 (.3) 3.8 (.5) 3.8 (.4) 3.8 (.5)


Knowledge in the practice of psychology. This section presents the results of the ratings related to the 78 knowledge statements delineated in connection with the eight content areas in the practice of psychology. Results related to Acquisition, and Criticality, and Usage ratings for each knowledge statement are presented for the total sample of respondents. (See pages 17 to 18 of this report for a complete description of the rating scales.) Table 51 documents the frequency distributions for the scale points associated with the Acquisition, Criticality, and Usage rating scales, and the mean, standard deviation, and number of respondents completing the Criticality rating for each knowledge statement for the total sample of respondents completing the versions of the survey containing the knowledge statements. As documented in Table 51 on the Acquisition scale, 85% or more of the respondents rated 64 of the 78 knowledge statements as necessary either primarily before or primarily after initial licensure—indicating they are key to the practice of the profession. All but nine statements received 50% or more support for acquisition primarily before initial certification, indicating that they represent valid targets for item development within the context of a licensure/registration examination. Nine knowledge statements were rated by at least 15% of the respondents as being acquired primarily after specialization, and one knowledge statements (i.e., evolutionary perspectives on social behavior) was rated by 15% as not necessary at any point in time. Table 51 also presents the frequency distribution for the knowledge statement Criticality ratings and the mean and standard deviation for each set of ratings. Inspection of Table 51 indicates that all but four statements were rated as 2.5 or higher on the Criticality scale, indicating that the overwhelming majority of the statements made at least a minimally-to-moderately important contribution to protection of the patient/client/public from harm. Exhibit 4 documents the four knowledge statements which fell below the arbitrary criterion of 2.5 on the Criticality scale.

Exhibit 4 Knowledge Statements Not Meeting Criterion for the Criticality Rating Scale

Knowledge of evolutionary perspectives on social behavior (2.2)

Knowledge of environmental/ecological psychology (2.1)

Knowledge of telecommunications-assisted psychological assessment, intervention, research, and documentation (2.4)

Knowledge of academic and vocational counseling (2.4)

As documented in Table 51, at least 85% of the respondents used 58 of the 78 knowledge statements at either the recognize/recall or the apply/interpret/integrate level. Between 16% and 46% of the respondents identified each of the remaining 20 knowledge statements as knowledge they did not have. A row-by-row inspection of the results indicates that these 20 knowledge statements were frequently rated as a target for acquisition primarily after specialization or were not necessary. Moreover, the Criticality ratings for these knowledge statements were among the


lowest ratings. Eight of these 20 statements were associated with one content area—Treatment, Intervention, Prevention, and Supervision. The knowledge least likely to be used by the respondents—Knowledge of telecommunication-assisted psychological assessment, intervention, research, and documentation—was also rated low on the Criticality scale and likely to be acquired primarily after specialization.


Table 51 Frequency Distribution for Acquisition, Criticality, and Usage Ratings, and Mean and Standard Deviation for Criticality Ratings for Total

Sample of Respondents Acquisition ratings: 0=Never; 1=Before licensure/registration; 2= After licensure/registration; 3=After specialization

Usage ratings: 0=Do not have; 1=Recognize/recall; 2=Apply/interpret/integrate

Acquisition Ratings Criticality Ratings Usage Ratings

0 1 2 3 Not Min Mod High Summary Statistics 0 1 2

% % % % n % % % % n Mean SD % % % n

Biological Bases of Behavior

Correlates and determinants of the biological and neural bases of behavior (e.g., neuroanatomy, neurophysiology, neuroendocrinology) pertaining to perception, cognition, personality, and mood and affect in normal, acute and chronic disordered states, (e.g., pain, stroke, focal lesion, traumatic brain injury, toxic metabolic states); and/or acute and chronic disease (e.g., diabetes, mood and psychotic spectrum disorders, dementias)

1% 80% 14% 5% 378 2% 12% 52% 34% 170 3.2 0.7 6% 38% 57% 196

Drug classification of therapeutic agents and drugs of abuse (e.g., anxiolytics, antidepressants, antipsychotics, mood stabilizers, cognitive-enhancing agents, opiates, psychostimulants); pharmacokinetics (administration, distribution, metabolism, elimination) and pharmacodynamics as they relate to the desired and non-desired, acute and chronic effects of therapeutic drugs, abused drugs, and common interactions with other drugs, foods, and herbal or alternative remedies

2% 74% 18% 6% 379 1% 14% 43% 42% 169 3.3 0.7 5% 37% 59% 196

Results from multi-center trials and guidelines for pharmacological, somatic, and combined treatment of psychological and neuropsychological disorders

6% 53% 24% 16% 374 8% 24% 46% 22% 166 2.8 0.9 18% 44% 38% 193

Behavioral genetics, transmission and expression of genetic information and its modification (e.g., gene-environment interactions) and the role of this information in understanding diseases and disorders (e.g., substance abuse disorders, Huntington's, pervasive developmental disorders, Alzheimer's) with consideration of co-morbidities and population differences in genetic information

6% 60% 20% 14% 374 7% 38% 40% 15% 169 2.6 0.8 19% 48% 33% 192




% % % % n % % % % n Mean SD % % % n

Interaction of developmental, gender, ethnic, cultural, environmental, and experiential factors with the biological and neural bases of behavior

2% 87% 8% 2% 372 4% 10% 43% 44% 169 3.3 0.8 4% 27% 69% 188

Applications and limitations of brain imaging methods that describe structure and function; electrophysiological methods (e.g., biofeedback, evoked potentials); therapeutic drug monitoring techniques; genetic screening methodologies

8% 34% 33% 25% 378 11% 41% 39% 9% 170 2.5 0.8 30% 49% 22% 193

Cognitive-Affective Bases of Behavior

Elements of cognition (e.g., sensation and perception, attention, learning (adult learning theory and other contemporary models), memory, language (reception and expression), intelligence, information processing, problem-solving, executive function)

0% 95% 4% 1% 384 2% 16% 29% 52% 176 3.3 0.8 1% 15% 85% 200

Major research-based theories and models of intelligence (e.g., Wechsler, Naglieri) and their application

1% 95% 3% 1% 384 2% 17% 35% 46% 176 3.2 0.8 2% 36% 63% 199

Major research-based theories, models, and principles of learning and their application (e.g., contingency reinforcement, interventions, cognitive behavioral therapy, training strategies, sports performance strategies)

1% 95% 3% 1% 379 2% 9% 27% 62% 175 3.5 0.7 1% 15% 85% 197

Major research-based theories and models of memory (e.g., multiple memory systems, expectancy theory, constructivist theory, levels of processing) and their application (e.g., use of mnemonics, neurorehabilitation following injury or in disease states).

4% 70% 21% 5% 382 4% 26% 42% 28% 175 2.9 0.8 5% 55% 41% 200

Major research-based theories and models of motivation and emotion (e.g., need/value approaches, cognitive appraisal, James-Lange theory of emotion) and their application (e.g., self-regulation, work motivation, anger management, social skills training, sports performance

1% 91% 7% 1% 381 3% 19% 42% 37% 175 3.1 0.8 3% 39% 59% 200




% % % % n % % % % n Mean SD % % % n

Interrelationships among cognitions/beliefs, behavior, affect, temperament, and mood (e.g., healthy functioning, performance anxiety, performance enhancement, job satisfaction, stress, and depression)

1% 95% 4% 0% 381 1% 7% 34% 58% 176 3.5 0.7 1% 14% 85% 199

Influence of psychosocial factors (e.g., life span development, gender, diversity characteristics) on beliefs/cognitions and behaviors

1% 96% 3% 0% 380 1% 11% 32% 56% 176 3.4 0.7 1% 16% 83% 198

Social and Cultural Bases of Behavior

Social cognition and perception (e.g., categorization and attribution theories, person perception, development of stereotypes, prejudice)

2% 95% 2% 1% 384 5% 28% 40% 27% 176 2.9 0.9 3% 44% 54% 200

Social interaction (e.g., interpersonal relationships, attraction, aggression, altruism, organizational justice, verbal and non-verbal communication, internet communication)

2% 92% 5% 1% 377 5% 25% 37% 34% 175 3.0 0.9 2% 30% 68% 198

Group/team dynamics and organizational structures (e.g., school, work, and family systems, job satisfaction, team functioning, conformity, persuasion) and social influences on individual functioning

2% 78% 15% 4% 382 7% 31% 40% 22% 177 2.8 0.9 5% 42% 53% 200

Environmental/ecological psychology (e.g., person-environment fit, job design, rural-urban contexts) 12% 44% 28% 16% 382 16% 50% 29% 5% 177 2.2 0.8 32% 47% 21% 197

Evolutionary perspectives on social behavior (e.g., mate selection, empathy) 15% 63% 16% 7% 378 23% 49% 23% 5% 171 2.1 0.8 19% 58% 23% 196

Major research-based theories of personality (e.g., psychodynamic, humanistic/existential, cognitive, behavioral, trait theory, interpersonal)

1% 99% 1% 0% 380 5% 13% 32% 51% 176 3.3 0.9 0% 20% 81% 200

Social-contextual issues (e.g., privilege/oppression, cross-cultural comparisons, political differences, international and global awareness, religion and spirituality)

7% 81% 8% 4% 381 6% 24% 38% 33% 174 3.0 0.9 8% 35% 57% 198




% % % % n % % % % n Mean SD % % % n Impact of race/ethnicity on psychosocial, political, and economic development of individuals, families, groups, organizations, and communities (e.g., theories of racial/ethnic identity; effects of culture on motivation and communication)

5% 89% 5% 1% 383 6% 15% 36% 44% 176 3.2 0.9 5% 29% 66% 200

Causes, manifestations, effects of oppression (e.g., racism, sexism, heterosexism, ethnic conflicts, colonization, classism, political persecution)

7% 78% 12% 3% 384 8% 23% 32% 37% 176 3.0 1.0 12% 35% 54% 200

Sexual orientation and identity (e.g., sexual identity development, workplace identity management, heterosexual/gay/lesbian/bisexual perspectives, parenting and family constellations)

2% 88% 8% 2% 384 5% 16% 38% 42% 177 3.2 0.9 5% 32% 64% 200

Psychology of gender (e.g., women/men/transgender, gender identity development) 4% 79% 14% 3% 383 6% 17% 43% 34% 177 3.0 0.9 8% 40% 52% 198

Disability and rehabilitation issues (e.g., psychological impact of disability on individuals and families, conceptual models and assumption of disability, incorporation or inclusion of persons with disabilities in the workplace)

1% 57% 30% 12% 379 3% 25% 45% 27% 177 2.9 0.8 14% 33% 54% 198

Acculturation of immigrant, refugee and political asylum seeking populations (e.g., stages of acculturation models, trauma, mental health)

6% 50% 31% 13% 379 10% 25% 45% 20% 177 2.8 0.9 27% 46% 26% 197

Culturally mediated communication patterns 7% 62% 22% 9% 381 10% 30% 35% 25% 175 2.7 0.9 25% 38% 37% 199

Growth and Lifespan Development

Normal growth and development (biological, physical, sexual functioning, cognitive, perceptual, social, personality, moral/spiritual, emotional, career development, and end of life issues) across the full lifespan

0% 98% 2% 0% 381 1% 7% 33% 60% 178 3.5 0.6 0% 18% 82% 200.0

Influence of individual-environment interaction over time (e.g., the relationship between the individual and the social, academic, or work environment) on development

2% 83% 13% 2% 380 4% 25% 45% 26% 176 2.9 0.8 5% 37% 59% 198

Major research-based theories of development 1% 95% 4% 1% 382 3% 22% 33% 42% 178 3.1 0.9 3% 41% 57% 200




% % % % n % % % % n Mean SD % % % n Influence of culture on development (e.g., cultural moderation of individual-environment interactions, acculturation and development)

2% 86% 10% 3% 380 4% 22% 42% 32% 177 3.0 0.8 6% 41% 53% 196

Family development and functioning and its impact on the individual across the full lifespan 0% 91% 8% 1% 381 1% 15% 41% 43% 177 3.3 0.7 4% 31% 65% 199

Family configuration (e.g., single parent, reconstituted, extended, gay/lesbian) and its effects on child and adolescent development

2% 81% 15% 3% 382 3% 22% 42% 33% 178 3.1 0.8 9% 34% 58% 200

Normative and non-normative life event changes (e.g., injury, trauma, illness, interpersonal loss, birth of a child, retirement) that can influence the normal course of development

1% 86% 11% 3% 379 1% 13% 47% 40% 176 3.3 0.7 6% 23% 72% 200

System factors that predict a problematic developmental course (e.g., nutritional deficiencies, poor prenatal care, poor health care, lack of social support, poverty, exposure to violence and abuse/victimization)

1% 84% 12% 3% 381 2% 12% 50% 36% 178 3.2 0.7 6% 32% 63% 200

Disorders/diseases (e.g., cognitive, genetic, psychological) that impact expected course of development over the full lifespan

1% 81% 12% 5% 378 2% 16% 44% 38% 176 3.2 0.8 5% 29% 67% 200

Assessment and Diagnosis

Psychometric theory (e.g., classical test theory, item response theory), generalizability theory, item and test characteristics and related concepts (e.g., test construction and standardization procedures, reliability and validity, sensitivity and specificity, examination of test fairness and bias, test and item characteristics)

0% 93% 6% 1% 379 4% 16% 28% 52% 178 3.3 0.9 4% 32% 64% 199

Assessment theories and models (e.g., psychometric, behavioral, ecological, neuropsychological) 0% 93% 6% 1% 381 2% 22% 33% 43% 178 3.2 0.8 3% 33% 65% 198

Assessment methods (e.g., standardized administration, assessment centers, self-report, report by others, psychophysiological, work sample, direct observation, structured and semi-structured interviews)

1% 92% 7% 1% 382 4% 13% 28% 55% 177 3.3 0.9 1% 22% 78% 199




% % % % n % % % % n Mean SD % % % n Tests for the measurement of characteristics and behaviors of individuals (e.g., social, relational, emotional, and behavioral functioning; cognitive and neuropsychological functioning; ability, aptitude, and achievement; personality; vocational interest; health behavior; assessment of symptom validity, competence, criminal responsibility, risk of future violence, suicide evaluation), and the adaptation of these tests for use with various populations

1% 83% 14% 2% 380 2% 8% 30% 60% 176 3.5 0.7 2% 17% 82% 199

Issues of differential diagnosis and integration of non-psychological information (e.g., medical evaluations, educational history, genomics) into psychological assessment

1% 81% 16% 2% 378 2% 8% 35% 55% 177 3.4 0.7 4% 18% 79% 197

Instruments and methods for the measurement of characteristics and performance of jobs, organizations and systems of care, and educational and other social institutions (e.g., performance appraisal, academic skill development , job analysis, job evaluation

6% 34% 42% 18% 382 10% 34% 40% 16% 176 2.6 0.9 26% 45% 29% 198

Criteria for selection and adaptation of assessment methods (e.g., cultural appropriateness, trans-cultural adaptation, language accommodation, cost effectiveness, incremental validity, relevance to referral concern, evidence-based knowledge of assessment

1% 72% 20% 7% 379 6% 14% 44% 37% 178 3.1 0.8 11% 36% 54% 199

Classification systems (e.g., DSM, WHO, ICD) and their underlying rationales and limitations for evaluating client functioning; dimensional vs. categorical approaches to diagnosis

1% 91% 8% 0% 382 3% 7% 30% 61% 178 3.5 0.7 2% 15% 83% 198

Factors influencing interpretation of data and decision-making (e.g., base rates, group differences, cultural biases and differences, heuristics, evidence base)

1% 86% 11% 3% 380 3% 19% 32% 46% 177 3.2 0.9 4% 33% 64% 198

Epidemiology of behavioral disorders, base rates of disorders in clinical or demographic populations; comorbidity of mental illness with substance abuse; comorbidity of behavioral disorders with medical disorders; comorbidity rates, age ranges affected; natural history disorders

1% 79% 15% 5% 382 2% 16% 37% 45% 179 3.3 0.8 3% 37% 60% 198




% % % % n % % % % n Mean SD % % % n Methods for the measurement of individual, couples, family, group, and organizational change due to intervention or prevention efforts (e.g., continuous monitoring, behavioral analysis, pre-, post-, and follow-up assessment, detection of relapse, patient adherence, organizational benchmarking, response to intervention)

2% 60% 27% 12% 381 5% 21% 45% 28% 179 3.0 0.8 16% 43% 42% 199

Use of computers, the internet, and related technology in implementing tests, surveys, and other forms of assessment and diagnostic evaluation; validity, cost effectiveness, consumer acceptability, privacy issues and confidentiality

4% 63% 25% 9% 379 6% 31% 36% 27% 178 2.8 0.9 20% 35% 45% 198

Treatment, Intervention, Prevention, and Supervision

Treatment decision making based on best available data (e.g., matching treatment to assessment/diagnosis, matching client/patient and therapist characteristics, cost and benefit, readiness to change)

1% 87% 11% 1% 379 2% 5% 33% 61% 176 3.5 0.7 3% 14% 83% 198

Contemporary theories/models of treatment/intervention and their evidence base 1% 93% 5% 1% 379 2% 7% 34% 57% 176 3.5 0.7 3% 23% 74% 197

Treatment techniques/interventions and the evidence for their comparative efficacy and effectiveness for specific disorders or functional concerns

1% 83% 15% 1% 377 1% 6% 31% 62% 176 3.5 0.7 3% 23% 75% 198

Adaptation of techniques for prevention and intervention with special populations (e.g., refugees, forensic, survivors of catastrophic injuries, linguistic and cultural minorities)

2% 39% 42% 17% 378 3% 11% 51% 35% 175 3.2 0.7 20% 40% 40% 198

Interventions to enhance growth and performance of individuals, couples, families, groups, systems, and organizations (e.g., enhancement of athletic performance, conflict resolution skills, career performance, coaching, organizational development and change, training and development, effects of work/overwork, human resource management interventions)

5% 35% 40% 20% 378 6% 31% 44% 18% 175 2.7 0.8 26% 42% 33% 195

Consultation models and processes for individuals, couples, families, groups, organizations, and 4% 52% 33% 11% 377 6% 30% 50% 14% 175 2.7 0.8 19% 41% 40% 196




% % % % n % % % % n Mean SD % % % n communities

Academic and vocational counseling 13% 28% 38% 21% 375 13% 41% 39% 8% 170 2.4 0.8 30% 42% 29% 196 Adjunctive interventions (e.g., support groups, individual self-help, indigenous support systems, spirituality)

7% 55% 26% 11% 376 8% 34% 42% 16% 172 2.7 0.8 16% 32% 52% 193

Telecommunication-assisted psychological assessment, intervention, research, and documentation 14% 32% 33% 21% 374 15% 44% 32% 9% 171 2.4 0.8 46% 30% 24% 194

Healthcare systems, structures, and economics, and how these impact intervention choice (e.g., funding sources and trends, cost/benefit considerations, medical cost-offset; health care resource allocation)

7% 38% 40% 15% 377 8% 30% 41% 21% 174 2.7 0.9 29% 34% 38% 194

Health promotion, risk reduction, resilience and wellness activities and methods 4% 61% 27% 8% 378 6% 25% 41% 29% 175 2.9 0.9 12% 32% 56% 196

Contemporary theories/models of supervision and their evidence base 4% 53% 32% 12% 375 7% 27% 41% 25% 175 2.8 0.9 16% 37% 47% 198

Research Methods and Statistics

Sampling and data collection (e.g., recruitment, instrument selection and calibration, questionnaire design, proximal-distal data collection, role of culture and other individual difference variables in choosing methods)

3% 86% 9% 2% 383 9% 24% 43% 24% 177 2.8 0.9 8% 56% 37% 200

Design of case studies, correlational, quasi-experimental and experimental studies (e.g., hypothesis generation; randomized controlled trials; cross-sequential designs, efficacy and effectiveness trials)

3% 87% 7% 2% 382 9% 24% 40% 27% 175 2.8 0.9 5% 60% 35% 199

Analytic methods 3% 87% 7% 3% 381 8% 26% 37% 29% 174 2.9 0.9 7% 59% 35% 200

Statistical interpretation (e.g., power, effect size, causation vs. association, sensitivity and specificity, generalizability, clinical versus statistical significance)

1% 92% 5% 2% 384 7% 16% 39% 39% 176 3.1 0.9 4% 48% 48% 199




% % % % n % % % % n Mean SD % % % n

Critical appraisal and application of research findings (e.g., adequacy of design and statistics, limitations to generalizations, threats to internal and external validity, design flaws, level of evidence)

1% 93% 4% 2% 383 4% 12% 38% 46% 177 3.3 0.8 5% 43% 53% 198

Evaluation strategies and techniques (e.g., needs assessment, process/implementation evaluation, formative and summative assessment program evaluation, outcome evaluation, cost-benefit analysis)

3% 64% 26% 7% 382 8% 24% 49% 19% 176 2.8 0.8 12% 54% 35% 200

Considerations underlying community involvement and participation in research, particularly for underrepresented populations (e.g. use of community advisory boards, community involvement and research planning)

8% 49% 26% 17% 380 11% 37% 38% 14% 175 2.5 0.9 25% 51% 24% 199

Dissemination and presentation of research findings (e.g., publication standards) 4% 76% 16% 4% 381 11% 32% 29% 28% 177 2.7 1.0 10% 51% 39% 199

Ethical/Legal/Professional Issues

Ethical principles/codes of psychologists (APA, CPA), and codes of conduct (e.g., ASPPB) 0% 99% 0% 0% 382 1% 1% 3% 95% 177 3.9 0.3 0% 3% 97% 198

Professional standards for the practice of psychology (e.g., standards for providers of psychology services, standards for educational and psychological testing)

0% 98% 1% 0% 378 1% 1% 6% 93% 175 3.9 0.4 0% 4% 96% 199

Federal, state and/or provincial laws/statutes and/or judicial decisions that affect psychological practice 1% 90% 9% 1% 383 0% 3% 16% 81% 176 3.8 0.5 0% 13% 87% 199

Potential ethical dilemmas (e.g., practice management, supervision, multiple relationships that impair judgment, problems and ethics of practice on the internet, group and family counseling); ethical issues associated with specific areas of practice (e.g., forensic, law enforcement, psychopharmacology, dealing with third party payors)

0% 94% 6% 0% 384 0% 2% 9% 89% 177 3.9 0.4 0% 7% 93% 198

Models of ethical decision-making process (e.g., Josephson’s “Five Steps of Principled Reasoning,” Rest model of ethical decisions, consensus model)

9% 79% 11% 2% 383 9% 16% 32% 44% 174 3.1 1.0 24% 35% 41% 199




% % % % n % % % % n Mean SD % % % n Models and approaches for professional development (e.g., methods for developing, updating, and enhancing knowledge in proficiencies and specialties, peer consultation and supervision; appropriateness of credentials)

3% 66% 27% 4% 382 5% 14% 44% 37% 177 3.1 0.8 9% 25% 66% 200

Consideration of social, legal, ethical, and policy implications of emerging issues (e.g., neuroethics, genetics, end of life issues)

3% 59% 26% 11% 384 6% 22% 39% 33% 176 3.0 0.9 11% 40% 49% 198

Patient’s/client’s rights (e.g., informed assent and consent, right to refuse treatment in least restrictive environment, privacy, privileged communication)

0% 97% 3% 0% 384 0% 1% 8% 91% 178 3.9 0.3 1% 6% 94% 200

Ethical issues in the conduct of research (e.g., protection of human participants, secondary use of data, conflict of interest)

1% 92% 6% 2% 383 0% 8% 22% 69% 177 3.6 0.6 3% 40% 57% 200

Ethical issues in supervision (e.g., multiple role relationships, client welfare, availability to supervisee) 0% 81% 16% 2% 383 1% 10% 19% 71% 178 3.6 0.7 3% 21% 76% 198

Acquisition ratings: 0=Never; 1=Before licensure/registration; 2= After licensure/registration; 3=After specialization Usage ratings: 0=Do not have; 1=Recognize/recall; 2=Apply/interpret/integrate


Appendix 11 and Appendix 12 present the knowledge Acquisition, Criticality, and Usage ratings of the U.S. and Canadian respondents, and the recently and less-recently licensed/registered respondents, respectively. Careful inspection of the results indicates general consistency in the ratings regardless of country or year of licensure/registration. Table 52 presents the mean of means for the Criticality rating scale for the knowledge statements associated with each content area for the total sample of respondents. The overall pattern for ratings reported in Table 52 is somewhat consistent with the pattern of ratings in Table 49, wherein respondents rated each content area on Criticality, although the absolute value of the ratings are consistently lower on the specific knowledge statements that comprise the areas than on the associated content areas themselves.

Table 52 Mean of Means for Criticality of Knowledge Statements

Content Areas Mean

Biological Bases of Behavior 3.0

Cognitive-Affective Bases of Behavior 3.3

Social and Cultural Bases of Behavior 2.9

Growth and Lifespan Development 3.2

Assessment and Diagnosis 3.2

Treatment, Intervention, Prevention, and Supervision 2.9

Research Methods and Statistics 2.9

Ethical/Legal/Professional Issues 3.6

For the total sample, the mean of means range from 2.9 for the seven knowledge statements delineated in connection with the Research Methods and Statistics content area, to 3.6 for the five knowledge statements delineated in association with Ethical/Legal/Professional Issues content area. Overall, the knowledge statements associated with three content areas (Social and Multicultural Bases of Behavior; Treatment, Intervention, Prevention, and Supervision; and Research Methods and Statistics) approached the level of moderately critical to protecting the patient/client/public from harm, while the knowledge statements associated with five content areas (Biological Bases of Behavior, Cognitive-Affective Bases of Behavior, Growth and Lifespan Development, Assessment and Diagnosis, Ethics/Legal/Professional Issues) were moderately-to-highly critical.

Implications of the Ratings Related to the Content-based Delineation. The patterns of ratings on the eight content areas and 78 knowledge statements can be reviewed in regard to their potential incorporation into the credentialing process for licensed/registered psychologist in the U.S. and Canada. The overall pattern of ratings on the eight content areas suggests that these content areas are important to the respondents, called upon with some frequency, and make a critical contribution to the protection of the patient/client/public from harm. The knowledge ratings provide guidance in regard to the content of the test questions that are associated with these knowledge statements. For example, the Acquisition ratings suggest that whereas most of


the statements were supported by an absolute majority of the respondents as being primarily acquired before licensure/registration, a number of statements may be more appropriately assessed primarily after licensure/registration. Test questions classified in these areas should be reviewed for depth and breadth to ensure that they are not beyond the scope of minimally-qualified candidates for licensure/registration. Moreover, while the Criticality ratings for the knowledge statements were generally high, a small number of statements make only a minimal contribution to the protection of the patient/client/public from harm. The absolute number of questions developed in conjunction with these knowledge statements should be monitored and a cap placed on the number of test items included within any examination. Finally, the level of usage ratings should be considered when developing and reviewing test questions for use in the credentialing program. Knowledge statements that are rated by a high percentage of respondents as used at either the recognize/ recall or apply/interpret/integrate level should serve as the basis for question development requiring the same cognitive level of processing.

Qualitative Results Related to Content Areas and Knowledge. Respondents to the Survey were asked to identify (a) tasks or knowledge that may have been missing from the delineations included in the various versions of the survey, and (b) long-range changes occurring in the practice of the profession—especially those that may affect the knowledge required of psychologists in the future. A careful review of the respondents’ comments indicated that no tasks or knowledge had been omitted from the survey. Rather, respondents who had been exposed to one version of the survey identified content that had been included in one or more other versions of the survey. The following summary characterizes the areas of change identified by the U.S. and Canadian respondents.

• Respondents were most likely to comment on the acquisition of prescription privileges by psychologists and the impact of psychopharmacology

• Large numbers of respondents commented on the impact of; the neurosciences, neuropsychology, and research into the biological basis for behavior; evidence-based practice; the business aspects of practice; the increasing diversity of the patient/client population; and technology.

Both U.S. and Canadian respondents identified the following types of knowledge and skills as needed in the future:

• Knowledge regarding neurology, biology, genetic determinants, psychopharmacology, medical imaging, technology-based assessments, and a growing number of specialties.

• Professional skills associated with procedures to integrate medicine into psychology,

including working with primary care physicians, procedures to work with increasingly diverse patients/clients, and procedures to implement evidence-based practice.

• Generic management skills, such as obtaining business training in order to run a practice

or work within a larger healthcare system.


Appendix 13 contains the respondents’ verbatim responses to the open-ended questions about long-range changes occurring in the practice of the profession—especially those that may affect the knowledge required of psychologists in the future. Content-Based Profile of Practice and Test Specifications for the EPPP

Profile of Practice. The purpose of developing profiles of practice from the data

collected and described previously is to provide empirical descriptions of the critical knowledge required by licensed/registered psychologists in practice. Following review by subject-matter experts, the profiles become the basis for the development of final test specifications for the EPPP. A content-based profile uses the data associated with the content-based delineation—the ratings on the content areas and knowledge statements. In the case of the content-based delineation, decisions were made regarding the most relevant sets of ratings from which to generate the profiles. To develop the content-based profile of practice, the following steps were implemented: • In Step 1, the Frequency and Criticality ratings for all respondents on each content area were

multiplied together and added across respondents to form a sum for each content area. The sums for the eight content areas were added together to form a grand sum. The Frequency × Criticality sum for each content area was then divided by the grand sum to produce a percentage or weight for each content area.

• In Step 2, the Acquisition and Criticality ratings of the knowledge statements were combined

to generate a mean value for the 78 statements. Exhibit 5 presents the weights assigned to each respondent’s ratings of each knowledge statement. The procedure was designed to highlight the knowledge base identified as necessary primarily before licensure, and to incorporate the relative criticality of each statement.


Exhibit 5

Weighting of Knowledge Statements

Weight Acquisition Criticality

1 Not necessary at any point Not critical

2 Not necessary at any point Minimally critical

3 Not necessary at any point Moderately critical

4 Not necessary at any point Highly critical

2 Primarily after specialization Not critical

4 Primarily after specialization Minimally critical

6 Primarily after specialization Moderately critical

8 Primarily after specialization Highly critical

3 Primarily before specialization Not critical

6 Primarily before specialization Minimally critical

9 Primarily before specialization Moderately critical

12 Primarily before specialization Highly critical

4 Primarily before licensure Not critical

8 Primarily before licensure Minimally critical

12 Primarily before licensure Moderately critical

16 Primarily before licensure Highly critical

The overall process produced a profile of practice in which the percentages for the content areas total 100%, and the mean value of each knowledge statement highlights the critical knowledge to be acquired—either before or after licensure. Table 53 and Table 54 presents the results of the procedures designed to produce the profiles of practice for the content-based delineation. As can be seen in Table 53 , at the content area level, seven of eight content areas each contribute between 11.8% and 15.1% to the profile, while one content area, Research Methods and Statistics, made a smaller contribution to the profile (7.5%). Table 54 documents the knowledge statements as they were presented in the survey of practice and their mean weightings as derived from each respondent’s Acquisition and Criticality ratings. Both within and across each of the eight content areas, statements with higher weightings would be considered to contribute more to professional performance than statements with lower mean weightings.


Table 53 Empirically Derived Test Specifications for Content Areas

% of exam Biological Bases of Behavior –– knowledge of (a) biological and neural bases of behavior, (b) psychopharmacology, and (c) methodologies supporting this body of knowledge

11.8%


13.3%


12.3%


12.3%


14.0%


14.1%


7.5%


15.1%

Table 54 Knowledge Statements—Mean Weightings

Mean Weighting

01 Biological Bases of Behavior

KN1 1.01


11.97

KN2 1.02


12.09

KN3 1.03 Results from multi-center trials and guidelines for pharmacological, somatic, and combined treatment of psychological and neuropsychological disorders 9.77


Mean Weighting

KN4 1.04

Behavioral genetics, transmission and expression of genetic information and its modification (e.g., gene-environment interactions) and the role of this information in understanding diseases and disorders (e.g., substance abuse disorders, Huntington's, pervasive developmental disorders, Alzheimer's) with consideration of co-morbidities and population differences in genetic information

9.12

KN5 1.05 Interaction of developmental, gender, ethnic, cultural, environmental, and experiential factors with the biological and neural bases of behavior 12.69

KN6 1.06 Applications and limitations of: brain imaging methods that describe structure and function; electrophysiological methods (e.g., biofeedback, evoked potentials); therapeutic drug monitoring techniques; genetic screening methodologies

7.71

02 Cognitive-Affective Bases of Behavior

KN7 2.01 Elements of cognition (e.g., sensation and perception, attention, learning (adult learning theory and other contemporary models), memory, language (reception and expression), intelligence, information processing, problem-solving, executive function

13.14

KN8 2.02 Major research-based theories and models of intelligence (e.g., Wechsler, Naglieri) and their application 12.84

KN9 2.03 Major research-based theories, models, and principles of learning and their application (e.g., contingency reinforcement, interventions, cognitive behavioral therapy, training strategies, sports performance strategies)

13.78

KN10 2.04 Major research-based theories and models of memory (e.g., multiple memory systems, expectancy theory, constructivist theory, levels of processing) and their application (e.g., use of mnemonics, neurorehabilitation following injury or in disease states).

10.80

KN11 2.05

Major research-based theories and models of motivation and emotion (e.g., need/value approaches, cognitive appraisal, James-Lange theory of emotion) and their application (e.g., self-regulation, work motivation, anger management, social skills training, sports performance)

12.12

KN12 2.06 Interrelationships among cognitions/beliefs, behavior, affect, temperament, and mood (e.g., healthy functioning, performance anxiety, performance enhancement, job satisfaction, stress, and depression)

13.74

KN13 2.07 Influence of psychosocial factors (e.g., life span development, gender, diversity characteristics) on beliefs/cognitions and behaviors 13.58

03 Social and Cultural Bases of Behavior

KN14 3.01 Social cognition and perception (e.g., categorization and attribution theories, person perception, development of stereotypes, prejudice) 11.34

KN15 3.02 Social interaction (e.g., interpersonal relationships, attraction, aggression, altruism, organizational justice, verbal and non-verbal communication, internet communication) 11.61

KN16 3.03 Group/team dynamics and organizational structures (e.g., school, work, and family systems, job satisfaction, team functioning, conformity, persuasion) and social influences on individual functioning

10.37

KN17 3.04 Environmental/ecological psychology (e.g., person-environment fit, job design, rural-urban contexts) 6.96

KN18 3.05 Evolutionary perspectives on social behavior (e.g., mate selection, empathy) 7.31

KN19 3.06 Major research-based theories of personality (e.g., psychodynamic, humanistic/existential, cognitive, behavioral, trait theory, interpersonal) 13.13

KN20 3.07 Social-contextual issues (e.g., privilege/oppression, cross-cultural comparisons, political differences, international and global awareness, religion and spirituality) 11.31

KN21 3.08 Impact of race/ethnicity on psychosocial, political, and economic development of individuals, families, groups, organizations, and communities (e.g., theories of racial/ethnic identity; effects of culture on motivation and communication)

12.32

KN22 3.09 Causes, manifestations, effects of oppression (e.g., racism, sexism, heterosexism, ethnic conflicts, colonization, classism, political persecution) 11.31


Mean Weighting

KN23 3.10 Sexual orientation and identity (e.g., sexual identity development, workplace identity management, heterosexual/gay/lesbian/bisexual perspectives, parenting and family constellations)

12.29

KN24 3.11 Psychology of gender (e.g., women/men/transgender, gender identity development) 11.45

KN25 3.12 Disability and rehabilitation issues (e.g., psychological impact of disability on individuals and families, conceptual models and assumption of disability, incorporation or inclusion of persons with disabilities in the workplace)

10.23

KN26 3.13 Acculturation of immigrant, refugee and political asylum seeking populations (e.g., stages of acculturation models, trauma, mental health) 9.30

KN27 3.14 Culturally mediated communication patterns 9.75

04 Growth and Lifespan Development

KN28 4.01 Normal growth and development (biological, physical, sexual functioning, cognitive, perceptual, social, personality, moral/spiritual, emotional, career development, and end of life issues) across the full lifespan

14.03

KN29 4.02 Influence of individual-environment interaction over time (e.g., the relationship between the individual and the social, academic, or work environment) on development

11.31

KN30 4.03 Major research-based theories of development 12.37

KN31 4.04 Influence of culture on development (e.g., cultural moderation of individual-environment interactions, acculturation and development) 11.53

KN32 4.05 Family development and functioning and its impact on the individual across the full lifespan 12.68

KN33 4.06 Family configuration (e.g., single parent, reconstituted, extended, gay/lesbian) and its effects on child and adolescent development 11.53

KN34 4.07 Normative and non-normative life event changes (e.g., injury, trauma, illness, interpersonal loss, birth of a child, retirement) that can influence the normal course of development

12.59

KN35 4.08 System factors that predict a problematic developmental course (e.g., nutritional deficiencies, poor prenatal care, poor health care, lack of social support, poverty, exposure to violence and abuse/victimization)

12.21

KN36 4.09 Disorders/diseases (e.g., cognitive, genetic, psychological) that impact expected course of development over the full lifespan 12.05

05 Assessment and Diagnosis

KN37 5.01


12.92

KN38 5.02 Assessment theories and models (e.g., psychometric, behavioral, ecological, neuropsychological) 12.53

KN39 5.03 Assessment methods (e.g., standardized administration, assessment centers, self-report, report by others, psychophysiological, work sample, direct observation, structured and semi-structured interviews)

13.11

KN40 5.04

Tests for the measurement of characteristics and behaviors of individuals (e.g., social, relational, emotional, and behavioral functioning; cognitive and neuropsychological functioning; ability, aptitude, and achievement; personality; vocational interest; health behavior; assessment of symptom validity, competence, criminal responsibility, risk of future violence, suicide evaluation), and the adaptation of these tests for use with various populations

13.29

KN41 5.05 Issues of differential diagnosis and integration of non-psychological information (e.g., medical evaluations, educational history, genomics) into psychological assessment 13.05


Mean Weighting

KN42 5.06

Instruments and methods for the measurement of characteristics and performance of jobs, organizations and systems of care, and educational and other social institutions (e.g., performance appraisal, academic skill development , job analysis, job evaluation, need assessment, functional analysis of behavior) and for the measurement of the environmental/ecological influences on individuals, groups or organizations

8.24

KN43 5.07

Criteria for selection and adaptation of assessment methods (e.g., cultural appropriateness, trans-cultural adaptation, language accommodation, cost effectiveness, incremental validity, relevance to referral concern, evidence-based knowledge of assessment limitations)

11.66

KN44 5.08 Classification systems (e.g., DSM, WHO, ICD) and their underlying rationales and limitations for evaluating client functioning; dimensional vs. categorical approaches to diagnosis

13.60

KN45 5.09 Factors influencing interpretation of data and decision-making (e.g., base rates, group differences, cultural biases and differences, heuristics, evidence base) 12.30

KN46 5.10

Epidemiology of behavioral disorders, base rates of disorders in clinical or demographic populations; comorbidity of mental illness with substance abuse; comorbidity of behavioral disorders with medical disorders; comorbidity rates, age ranges affected; natural history disorders

12.22

KN47 5.11

Methods for the measurement of individual, couples, family, group, and organizational change due to intervention or prevention efforts (e.g., continuous monitoring, behavioral analysis, pre-, post-, and follow-up assessment, detection of relapse, patient compliance, organizational benchmarking, response to intervention (RTI))

10.57

KN48 5.12 Use of computers, the internet, and related technology in implementing tests, surveys, and other forms of assessment and diagnostic evaluation; validity, cost effectiveness, consumer acceptability, privacy issues and confidentiality

10.05

06 Treatment, Intervention, Prevention, and Supervision

KN49 6.01 Treatment decision making based on best available data (e.g., matching treatment to assessment/diagnosis, matching client/patient and therapist characteristics, cost and benefit, readiness to change)

13.51

KN50 6.02 Contemporary theories/models of treatment/intervention and their evidence base 13.66

KN51 6.03 Treatment techniques/interventions and the evidence for their comparative efficacy and effectiveness for specific disorders or functional concerns 13.46

KN52 6.04 Adaptation of techniques for prevention and intervention with special populations (e.g., refugees, forensic, survivors of catastrophic injuries, linguistic and cultural minorities) 10.26

KN53 6.05


8.44

KN54 6.06 Consultation models and processes for individuals, couples, families, groups, organizations, and communities 9.01

KN55 6.07 Academic and vocational counseling 6.95

KN56 6.08 Adjunctive interventions (e.g., support groups, individual self-help, indigenous support systems, spirituality) 8.88

KN57 6.09 Telecommunication-assisted psychological assessment, intervention, research, and documentation 7.02

KN58 6.10 Healthcare systems, structures, and economics, and how these impact intervention choice (e.g., funding sources and trends, cost/benefit considerations, medical cost-offset; health care resource allocation)

8.79

KN59 6.11 Health promotion, risk reduction, resilience and wellness activities and methods 10.27

KN60 6.12 Contemporary theories/models of supervision and their evidence base 9.50


Mean Weighting

07 Research Methods and Statistics

KN61 7.01 Sampling and data collection (e.g., recruitment, instrument selection and calibration, questionnaire design, proximal-distal data collection, role of culture and other individual difference variables in choosing methods)

10.85

KN62 7.02 Design of case studies, correlational, quasi-experimental and experimental studies (e.g., hypothesis generation; randomized controlled trials; cross-sequential designs, efficacy and effectiveness trials)

11.04

KN63 7.03

Analytic methods: qualitative (e.g., thematic, phenomenological), descriptive and inferential statistics (e.g., probability theory, parametric statistics, meta analysis, exploratory and confirmatory factor analysis, non-parametric statistics, causal modeling)

11.06

KN64 7.04 Statistical interpretation (e.g., power, effect size, causation vs. association, sensitivity and specificity, generalizability, clinical versus statistical significance) 12.13

KN65 7.05 Critical appraisal and application of research findings (e.g., adequacy of design and statistics, limitations to generalizations, threats to internal and external validity, design flaws, level of evidence)

12.82

KN66 7.06 Evaluation strategies and techniques (e.g., needs assessment, process/implementation evaluation, formative and summative assessment program evaluation, outcome evaluation, cost-benefit analysis)

9.91

KN67 7.07 Considerations underlying community involvement and participation in research, particularly for underrepresented populations (e.g. use of community advisory boards, community involvement and research planning)

8.33

KN68 7.08 Dissemination and presentation of research findings (e.g., publication standards) 10.17

08 Ethical/Legal/Professional Issues

KN69 8.01 Ethical principles/codes of psychologists (APA, CPA), and codes of conduct (e.g., ASPPB) 15.72

KN70 8.02 Professional standards for the practice of psychology (e.g., standards for providers of psychology services, standards for educational and psychological testing) 15.59

KN71 8.03 Federal, state and/or provincial laws/statutes and/or judicial decisions that affect psychological practice 14.65

KN72 8.04

Potential ethical dilemmas (e.g., practice management, supervision, multiple relationships that impair judgment, problems and ethics of practice on the internet, group and family counseling); ethical issues associated with specific areas of practice (e.g., forensic, law enforcement, psychopharmacology, dealing with third party payors)

15.13

KN73 8.05 Models of ethical decision-making process (e.g., Josephson’s “Five Steps of Principled Reasoning,” Rest model of ethical decisions, consensus model) 11.74

KN74 8.06 Models and approaches for professional development (e.g., methods for developing, updating, and enhancing knowledge in proficiencies and specialties, peer consultation and supervision; appropriateness of credentials)

11.20

KN75 8.07 Consideration of social, legal, ethical, and policy implications of emerging issues (e.g., neuroethics, genetics, end of life issues) 10.23

KN76 8.08 Patient’s/client’s rights (e.g., informed assent and consent, right to refuse treatment in least restrictive environment, privacy, privileged communication) 15.46

KN77 8.09 Ethical issues in the conduct of research (e.g., protection of human participants, secondary use of data, conflict of interest) 14.11

KN78 8.10 Ethical issues in supervision (e.g., multiple role relationships, client welfare, availability to supervisee) 13.47


Development of Test Specifications. Following a systematic review of the content-based profile of practice, recommendations regarding revised test specifications for the EPPP were prepared. The goal of the revision was to develop test specifications that focus on the knowledge needed by licensed/registered psychologists that serves the public protection function of regulation. To develop the recommendations, PES presented all relevant data to the PATF and the PAAC. On the basis of discussions and review of the data, the participants supported the following decisions regarding the EPPP test specifications: 1. The continued use of the content areas and knowledge statements as the primary organizing

structure for the EPPP test specifications and feedback to the candidates. 2. The revision of 16 knowledge statements, including additions, deletions, and modifications

of the examples included as part of the knowledge statement. Members of the PAAC and the PATF suggested each revision after (a) carefully reviewing the qualitative comments of the respondents, including comments about statements that were not clear or reflective of contemporary perspectives, and (b) the patterns of ratings by various subgroups.

Exhibit 6 documents the 16 knowledge statements that were modified, including the original statement as it was included in the survey, and the revised statement as it is to be included in the test specifications for the EPPP).

3. The adoption of percentage weights for the eight content areas reflecting minor modifications

of the percentage weights adopted in 2003 and reflective of the 2009 results. Table 55 documents the recommended final test specifications at the content area level, including the recommended percentage weights at the content area level.

4. The use of the derived weighting for each knowledge statement, indicating the degree to

which related questions might be included in each version of the EPPP. In order of priority, the system incorporated the Criticality ratings and the Acquisition ratings for the knowledge statements. Accordingly, the 78 knowledge statements were ranked by their mean derived weighting. Top ranked knowledge statements (i.e., knowledge statements with derived weightings ranked 1 to 26) are to be selected most frequently as the basis for item writing activities, while mid-ranked knowledge statements (i.e., 27 to 52) and bottom-ranked knowledge statements (53 to 78) are to be selected less frequently and infrequently, respectively, as the basis for item writing activities.

Table 56 documents the rank of each of the 78 knowledge statements. In order to highlight their use in regard to examination construction activities, each statement is color-coded as to its rank within the top, middle, or bottom of the distribution for all knowledge statements across the content areas. Accordingly, statements ranked 1 to 26 are shaded in green to identify the subset of the highest ranked statements, while statements ranked 27 to 52, and 53 to 78 are shaded in pink and blue, respectively, to identify the subsets of mid- and lowest-ranked statements. Overall, the content area with the greatest percentage of highly ranked statements is Content Area 8, Ethical/Legal/Professional Issues, and the content area with the


lowest ranked statements is Content Area 6, Treatment, Intervention, Prevention, and Supervision. Content Area 6 is also the only area with no mid-ranked statements.

5. The use of a rubricking system by which: (a) test items would be classified by content area and knowledge statement; and (b) forms of the EPPP would be constructed so as to reflect the percentage allocations for the eight content areas, including within each content area an appropriate balance of test questions reflective of the ranked values of the associated knowledge. Accordingly, at the content-area level, each form of the EPPP should include a preponderance of test questions associated with higher-ranked knowledge statements, a lesser number associated with mid-ranked knowledge statements, and only a few test questions associated with low-ranked knowledge statements.

The PAAC and the PATF concluded that the development of the EPPP in accordance with the foregoing information and decisions is most likely to provide the continuing basis for a generic or core licensing examination for psychologists in the U.S. and Canada. Such an examination will assess the critical knowledge base required at the point of licensure in a practice-relevant framework.

Exhibit 6 16 Modified Knowledge Statements

Knowledge statement as presented in survey Final revision of knowledge statement

1.04 Behavioral genetics, transmission and expression of genetic information and its modification (e.g., gene-environment interactions) and the role of this information in understanding diseases and disorders (e.g., substance abuse disorders, Huntington's, pervasive developmental disorders, Alzheimer's) with consideration of co-morbidities and population differences in genetic information

1.04 Behavioral genetics, transmission and expression of genetic information and its modification (e.g., gene-environment interactions) and the role of this information in understanding diseases and disorders (e.g., substance abuse disorders, neurodegenerative disorders, pervasive developmental disorders) with consideration of co-morbidities and population differences in genetic information

1.06 Applications and limitations of: brain imaging methods that describe structure and function; electrophysiological methods (e.g., biofeedback, evoked potentials); therapeutic drug monitoring techniques; genetic screening methodologies

1.06 Applications of brain imaging methods that describe structure and function, electrophysiological methods, therapeutic drug monitoring techniques, and genetic screening methodologies, and the evidence for their effectiveness

2.01 Elements of cognition (e.g., sensation and perception, attention, learning (adult learning theory and other contemporary models), memory, language (reception and expression), intelligence, information processing, problem-solving, executive function

2.01 Elements of cognition, including sensation and perception, attention, learning (adult learning theory and other contemporary models), memory, language (reception and expression), intelligence, information processing, problem-solving, executive function

2.02 Major research-based theories and models of intelligence (e.g., Wechsler, Naglieri) and their application

2.02 Major research-based theories and models of intelligence and their application

2.05 Major research-based theories and models of motivation and emotion (e.g., need/value approaches, cognitive appraisal, James-Lange

2.05 Major research-based theories and models of motivation and emotion (e.g., need/value approaches, cognitive appraisal, contemporary



theory of emotion) and their application (e.g., self-regulation, work motivation, anger management, social skills training, sports performance)

theories of emotion) and their application (e.g., self-regulation, work motivation, anger management, social skills training, sports performance)

3.04 Environmental/ecological psychology (e.g., person-environment fit, job design, rural-urban contexts)

3.04 Environmental/ecological psychology (e.g., person-environment fit, job design, rural-urban contexts) and human impact of environmental and societal events (e.g., natural disasters, armed conflict, job loss and unemployment)

5.08 Classification systems (e.g., DSM, WHO, ICD) and their underlying rationales and limitations for evaluating client functioning; dimensional vs. categorical approaches to diagnosis

5.08 Classification systems (e.g., DSM, ICF, ICD) and their underlying rationales and limitations for evaluating client functioning; dimensional vs. categorical approaches to diagnosis

5.10 Epidemiology of behavioral disorders, base rates of disorders in clinical or demographic populations; comorbidity of mental illness with substance abuse; comorbidity of behavioral disorders with medical disorders; comorbidity rates, age ranges affected; natural history disorders

5.10 Epidemiology of behavioral disorders, base rates of disorders in clinical or demographic populations; comorbidity of mental illness with substance abuse; comorbidity of behavioral disorders with medical disorders; comorbidity rates, age ranges affected

5.11 Methods for the measurement of individual, couples, family, group, and organizational change due to intervention or prevention efforts (e.g., continuous monitoring, behavioral analysis, pre-, post-, and follow-up assessment, detection of relapse, patient compliance, organizational benchmarking, response to intervention (RTI))

5.11 Methods for the measurement of individual, couples, family, group, and organizational change due to intervention or prevention efforts (e.g., continuous monitoring, behavioral analysis, pre-, post-, and follow-up assessment, detection of relapse, patient adherence, organizational benchmarking, response to intervention)

6.04 Adaptation of techniques for prevention and intervention with special populations (e.g., refugees, forensic, survivors of catastrophic injuries, linguistic and cultural minorities)

6.04 Adaptation of techniques for prevention and intervention with special populations (e.g., linguistic and cultural minorities, refugees, prisoners, survivors of catastrophic injuries)

6.07 Academic and vocational counseling 6.07 Career counseling

6.09 Telecommunication-assisted psychological assessment, intervention, research, and documentation

6.09 Technology-assisted psychological assessment, intervention, research, and documentation

7.01 Sampling and data collection (e.g., recruitment, instrument selection and calibration, questionnaire design, proximal-distal data collection, role of culture and other individual difference variables in choosing methods)

7.01 Sampling and data collection (e.g., recruitment, community partnerships, instrument selection and calibration, questionnaire design, proximal-distal data collection, role of culture and other individual difference variables in choosing methods)

8.04 Potential ethical dilemmas (e.g., practice management, supervision, multiple relationships that impair judgment, problems and ethics of practice on the internet, group and family counseling); ethical issues associated with specific areas of practice (e.g., forensic, law enforcement, psychopharmacology, dealing with third party payors)

8.04 Potential ethical dilemmas (e.g., practice management, supervision, multiple relationships, group and family counseling) and ethical issues associated with specific areas of practice (e.g., forensic, law enforcement, psychopharmacology, dealing with third party payors)



8.05 Models of ethical decision-making process (e.g., Josephson’s “Five Steps of Principled Reasoning,” Rest model of ethical decisions, consensus model)

8.05 Models of ethical decision-making process

8.07 Consideration of social, legal, ethical, and policy implications of emerging issues (e.g., neuroethics, genetics, end of life issues)

8.07 Consideration of social, legal, ethical, and policy implications of current and emerging issues (e.g., neuroethics, use of technology in practice, genetics, end of life issues)

Table 55 Recommended Test Specifications for Content Areas

% of exam Biological Bases of Behavior –– knowledge of (a) biological and neural bases of behavior, (b) psychopharmacology, and (c) methodologies supporting this body of knowledge

12%


13%


12%


12%


14%


14%


8%


15%


Table 56 Final Knowledge Statements and Ranks for EPPP

Rank

01 Biological Bases of Behavior 12%

KN1 1.01


38

KN2 1.02


36

KN3 1.03 Results from multi-center trials and guidelines for pharmacological, somatic, and combined treatment of psychological and neuropsychological disorders 63

KN4 1.04

Behavioral genetics, transmission and expression of genetic information and its modification (e.g., gene-environment interactions) and the role of this information in understanding diseases and disorders (e.g., substance abuse disorders, neurodegenerative disorders, pervasive developmental disorders) with consideration of co-morbidities and population differences in genetic information

67

KN5 1.05 Interaction of developmental, gender, ethnic, cultural, environmental, and experiential factors with the biological and neural bases of behavior 24

KN6 1.06 Applications of brain imaging methods that describe structure and function, electrophysiological methods, therapeutic drug monitoring techniques, and genetic screening methodologies, and the evidence for their effectiveness

74

02 Cognitive-Affective Bases of Behavior 13%

KN7 2.01 Elements of cognition, including sensation and perception, attention, learning (adult learning theory and other contemporary models), memory, language (reception and expression), intelligence, information processing, problem-solving, executive function

17

KN8 2.02 Major research-based theories and models of intelligence and their application 22

KN9 2.03 Major research-based theories, models, and principles of learning and their application (e.g., contingency reinforcement, interventions, cognitive behavioral therapy, training strategies, sports performance strategies)

8

KN10 2.04 Major research-based theories and models of memory (e.g., multiple memory systems, expectancy theory, constructivist theory, levels of processing) and their application (e.g., use of mnemonics, neurorehabilitation following injury or in disease states).

53

KN11 2.05

Major research-based theories and models of motivation and emotion (e.g., need/value approaches, cognitive appraisal, contemporary theories of emotion) and their application (e.g., self-regulation, work motivation, anger management, social skills training, sports performance)

35

KN12 2.06 Interrelationships among cognitions/beliefs, behavior, affect, temperament, and mood (e.g., healthy functioning, performance anxiety, performance enhancement, job satisfaction, stress, and depression)

9

KN13 2.07 Influence of psychosocial factors (e.g., life span development, gender, diversity characteristics) on beliefs/cognitions and behaviors 12

03 Social and Cultural Bases of Behavior 12%

KN14 3.01 Social cognition and perception (e.g., categorization and attribution theories, person perception, development of stereotypes, prejudice) 45


Rank

KN15 3.02 Social interaction (e.g., interpersonal relationships, attraction, aggression, altruism, organizational justice, verbal and non-verbal communication, internet communication) 41

KN16 3.03 Group/team dynamics and organizational structures (e.g., school, work, and family systems, job satisfaction, team functioning, conformity, persuasion) and social influences on individual functioning

55

KN17 3.04 Environmental/ecological psychology (e.g., person-environment fit, job design, rural-urban contexts) and human impact of environmental and societal events (e.g., natural disasters, armed conflict, job loss and unemployment)

77

KN18 3.05 Evolutionary perspectives on social behavior (e.g., mate selection, empathy) 75

KN19 3.06 Major research-based theories of personality (e.g., psychodynamic, humanistic/existential, cognitive, behavioral, trait theory, interpersonal) 18

KN20 3.07 Social-contextual issues (e.g., privilege/oppression, cross-cultural comparisons, political differences, international and global awareness, religion and spirituality) 47

KN21 3.08 Impact of race/ethnicity on psychosocial, political, and economic development of individuals, families, groups, organizations, and communities (e.g., theories of racial/ethnic identity; effects of culture on motivation and communication)

29

KN22 3.09 Causes, manifestations, effects of oppression (e.g., racism, sexism, heterosexism, ethnic conflicts, colonization, classism, political persecution) 48

KN23 3.10 Sexual orientation and identity (e.g., sexual identity development, workplace identity management, heterosexual/gay/lesbian/bisexual perspectives, parenting and family constellations)

31

KN24 3.11 Psychology of gender (e.g., women/men/transgender, gender identity development) 44

KN25 3.12 Disability and rehabilitation issues (e.g., psychological impact of disability on individuals and families, conceptual models and assumption of disability, incorporation or inclusion of persons with disabilities in the workplace)

58

KN26 3.13 Acculturation of immigrant, refugee and political asylum seeking populations (e.g., stages of acculturation models, trauma, mental health) 66

KN27 3.14 Culturally mediated communication patterns 64

04 Growth and Lifespan Development 12%

KN28 4.01 Normal growth and development (biological, physical, sexual functioning, cognitive, perceptual, social, personality, moral/spiritual, emotional, career development, and end of life issues) across the full lifespan

7

KN29 4.02 Influence of individual-environment interaction over time (e.g., the relationship between the individual and the social, academic, or work environment) on development

46

KN30 4.03 Major research-based theories of development 28

KN31 4.04 Influence of culture on development (e.g., cultural moderation of individual-environment interactions, acculturation and development) 42

KN32 4.05 Family development and functioning and its impact on the individual across the full lifespan 25

KN33 4.06 Family configuration (e.g., single parent, reconstituted, extended, gay/lesbian) and its effects on child and adolescent development 43

KN34 4.07 Normative and non-normative life event changes (e.g., injury, trauma, illness, interpersonal loss, birth of a child, retirement) that can influence the normal course of development

26

KN35 4.08 System factors that predict a problematic developmental course (e.g., nutritional deficiencies, poor prenatal care, poor health care, lack of social support, poverty, exposure to violence and abuse/victimization)

33

KN36 4.09 Disorders/diseases (e.g., cognitive, genetic, psychological) that impact expected course of development over the full lifespan 37


Rank

05 Assessment and Diagnosis 14%

KN37 5.01


21

KN38 5.02 Assessment theories and models (e.g., psychometric, behavioral, ecological, neuropsychological) 27

KN39 5.03 Assessment methods (e.g., standardized administration, assessment centers, self-report, report by others, psychophysiological, work sample, direct observation, structured and semi-structured interviews)

19

KN40 5.04

Tests for the measurement of characteristics and behaviors of individuals (e.g., social, relational, emotional, and behavioral functioning; cognitive and neuropsychological functioning; ability, aptitude, and achievement; personality; vocational interest; health behavior; assessment of symptom validity, competence, criminal responsibility, risk of future violence, suicide evaluation), and the adaptation of these tests for use with various populations

16

KN41 5.05 Issues of differential diagnosis and integration of non-psychological information (e.g., medical evaluations, educational history, genomics) into psychological assessment 20

KN42 5.06

Instruments and methods for the measurement of characteristics and performance of jobs, organizations and systems of care, and educational and other social institutions (e.g., performance appraisal, academic skill development , job analysis, job evaluation, need assessment, functional analysis of behavior) and for the measurement of the environmental/ecological influences on individuals, groups or organizations

73

KN43 5.07

Criteria for selection and adaptation of assessment methods (e.g., cultural appropriateness, trans-cultural adaptation, language accommodation, cost effectiveness, incremental validity, relevance to referral concern, evidence-based knowledge of assessment limitations)

40

KN44 5.08 Classification systems (e.g., DSM, ICF, ICD) and their underlying rationales and limitations for evaluating client functioning; dimensional vs. categorical approaches to diagnosis

11

KN45 5.09 Factors influencing interpretation of data and decision-making (e.g., base rates, group differences, cultural biases and differences, heuristics, evidence base) 30

KN46 5.10

Epidemiology of behavioral disorders, base rates of disorders in clinical or demographic populations; comorbidity of mental illness with substance abuse; comorbidity of behavioral disorders with medical disorders; comorbidity rates, age ranges affected

32

KN47 5.11

Methods for the measurement of individual, couples, family, group, and organizational change due to intervention or prevention efforts (e.g., continuous monitoring, behavioral analysis, pre-, post-, and follow-up assessment, detection of relapse, patient adherence, organizational benchmarking, response to intervention)

54

KN48 5.12 Use of computers, the internet, and related technology in implementing tests, surveys, and other forms of assessment and diagnostic evaluation; validity, cost effectiveness, consumer acceptability, privacy issues and confidentiality

61

06 Treatment, Intervention, Prevention, and Supervision 14%

KN49 6.01 Treatment decision making based on best available data (e.g., matching treatment to assessment/diagnosis, matching client/patient and therapist characteristics, cost and benefit, readiness to change)

13

KN50 6.02 Contemporary theories/models of treatment/intervention and their evidence base 10

KN51 6.03 Treatment techniques/interventions and the evidence for their comparative efficacy and effectiveness for specific disorders or functional concerns 15


Rank

KN52 6.04 Adaptation of techniques for prevention and intervention with special populations (e.g., linguistic and cultural minorities, refugees, prisoners, survivors of catastrophic injuries)

57

KN53 6.05


71

KN54 6.06 Consultation models and processes for individuals, couples, families, groups, organizations, and communities 68

KN55 6.07 Career counseling 78

KN56 6.08 Adjunctive interventions (e.g., support groups, individual self-help, indigenous support systems, spirituality) 69

KN57 6.09 Technology-assisted psychological assessment, intervention, research, and documentation 76

KN58 6.10 Healthcare systems, structures, and economics, and how these impact intervention choice (e.g., funding sources and trends, cost/benefit considerations, medical cost-offset; health care resource allocation)

70

KN59 6.11 Health promotion, risk reduction, resilience and wellness activities and methods 56

KN60 6.12 Contemporary theories/models of supervision and their evidence base 65

07 Research Methods and Statistics 8%

KN61 7.01 Sampling and data collection (e.g., recruitment, community partnerships, instrument selection and calibration, questionnaire design, proximal-distal data collection, role of culture and other individual difference variables in choosing methods)

52

KN62 7.02 Design of case studies, correlational, quasi-experimental and experimental studies (e.g., hypothesis generation; randomized controlled trials; cross-sequential designs, efficacy and effectiveness trials)

51

KN63 7.03

Analytic methods: qualitative (e.g., thematic, phenomenological), descriptive and inferential statistics (e.g., probability theory, parametric statistics, meta analysis, exploratory and confirmatory factor analysis, non-parametric statistics, causal modeling)

50

KN64 7.04 Statistical interpretation (e.g., power, effect size, causation vs. association, sensitivity and specificity, generalizability, clinical versus statistical significance) 34

KN65 7.05 Critical appraisal and application of research findings (e.g., adequacy of design and statistics, limitations to generalizations, threats to internal and external validity, design flaws, level of evidence)

23

KN66 7.06 Evaluation strategies and techniques (e.g., needs assessment, process/implementation evaluation, formative and summative assessment program evaluation, outcome evaluation, cost-benefit analysis)

62

KN67 7.07 Considerations underlying community involvement and participation in research, particularly for underrepresented populations (e.g. use of community advisory boards, community involvement and research planning)

72

KN68 7.08 Dissemination and presentation of research findings (e.g., publication standards) 60

08 Ethical/Legal/Professional Issues 15%

KN69 8.01 Ethical principles/codes of psychologists (APA, CPA), and codes of conduct (e.g., ASPPB) 1

KN70 8.02 Professional standards for the practice of psychology (e.g., standards for providers of psychology services, standards for educational and psychological testing) 2

KN71 8.03 Federal, state and/or provincial laws/statutes and/or judicial decisions that affect psychological practice 5


Rank

KN72 8.04

Potential ethical dilemmas (e.g., practice management, supervision, multiple relationships, group and family counseling) and ethical issues associated with specific areas of practice (e.g., forensic, law enforcement, psychopharmacology, dealing with third party payors)

4

KN73 8.05 Models of ethical decision-making process 39

KN74 8.06 Models and approaches for professional development (e.g., methods for developing, updating, and enhancing knowledge in proficiencies and specialties, peer consultation and supervision; appropriateness of credentials)

49

KN75 8.07 Consideration of social, legal, ethical, and policy implications of current and emerging issues (e.g., neuroethics, use of technology in practice, genetics, end of life issues) 59

KN76 8.08 Patient’s/client’s rights (e.g., informed assent and consent, right to refuse treatment in least restrictive environment, privacy, privileged communication) 3

KN77 8.09 Ethical issues in the conduct of research (e.g., protection of human participants, secondary use of data, conflict of interest) 6

KN78 8.10 Ethical issues in supervision (e.g., multiple role relationships, client welfare, availability to supervisee) 14

As can be seen in Table 56, at the content-area level, the Ethical/Legal/Professional Issues content area included six of the top-10 ranked knowledge statements; the Cognitive-Affective Bases of Behavior content area included two of the top 10 ranked statements; and both Growth and Lifespan Development and Treatment, Intervention, Prevention, and Supervision content areas each included one of the top-10 highest-ranked statements. Competencies The following sets of results relate to the competency-based delineation comprising six competency clusters, 37 competency statements, and 276 exemplars describing the theoretical evolution of performance in regard to the specific competencies. As described previously, one subset of the total target sample rated the competency clusters and competency statements on two sets of rating scales—either Frequency and Criticality, or Frequency and Importance, and a second subset rated the behavioral exemplars associated with the competency statements on one rating scale—Point of Acquisition.

Competency Clusters. Table 57 presents the mean, standard deviation, and number of respondents completing the Frequency, Criticality, and Importance ratings for the six competency clusters, and the frequency distributions for the associated scale points. As can be seen, the means for the three sets of ratings are quite high, indicating that these clusters each represent competencies which are performed frequently-to-very frequently, are moderately-to-highly critical to optimizing outcomes for patient/client/public, and are moderately-to-very important to the practice of the respondents. The ratings of three clusters—Professionalism/Ethics, Scientific Knowledge, and Cultural and Interpersonal Competence—are ranked first, second, and third, regardless of rating scale. Conversely, the ratings for one cluster—Evidence-based Decision Making/Critical Reasoning—are ranked last, regardless of rating scale.


Review of the frequency distributions for the results related to the scale points for the competency clusters indicates 80% or more of the respondents perform the competencies associated with each cluster at least frequently and consider the competencies at least moderately important to their practice. Finally, 90% or more of the respondents consider the competencies at least moderately important to optimizing outcomes.


Table 57 Competency Clusters

Frequency, Criticality, and Importance Ratings for Total Sample


Never/ rarely

Infre- quently

Fre- quently

Very Freq



Summary Statistics

% % % % n Mean SD % % % % n Mean SD % % % % n Mean SD Scientific Knowledge 1% 4% 36% 60% 400 3.6 (.6) 0% 3% 26% 71% 393 3.7 (.5) 1% 3% 29% 67% 398 3.6 (.6)

Evidence-Based Decision Making/Critical Reasoning 3% 14% 45% 39% 399 3.2 (.8) 1% 9% 40% 50% 387 3.4 (.7) 2% 13% 37% 48% 397 3.3 (.8)

Cultural and Interpersonal Competence 1% 9% 37% 54% 399 3.4 (.7) 0% 4% 28% 68% 389 3.6 (.6) 2% 7% 30% 62% 399 3.5 (.7)

Professionalism/Ethics 1% 1% 20% 79% 400 3.8 (.5) 0% 2% 16% 82% 385 3.8 (.5) 1% 1% 15% 83% 397 3.8 (.5)

Assessment 6% 14% 31% 49% 399 3.2 (.9) 2% 7% 32% 59% 386 3.5 (.7) 6% 12% 27% 54% 397 3.3 (.9)

Intervention/Supervision/ Consultation 6% 12% 36% 45% 398 3.2 (.9) 2% 4% 33% 61% 383 3.5 (.7) 5% 9% 32% 54% 399 3.3 (.9)


Competency Statements. Table 58 presents the mean of means for the competency

statements within each cluster on the Frequency, Criticality, and Importance rating scales. Regardless of rating scale, ratings related to the competencies within the Evidence-based Decision Making/Critical Reasoning cluster are lower than the ratings for any other set of competencies. Comparison of Table 57 and Table 58 also indicates that on each of the three rating scales, ratings made in regard to the competency clusters are higher than the comparable ratings for the specific competency statements associated with the cluster. (Note: respondents were not asked to rate specific competencies associated with the Scientific Knowledge competency cluster.)

Table 58 Mean of Means for Competency Statements

on the Frequency, Criticality, and Importance Rating Scales

Mean of Means Competency Cluster Frequency Criticality Importance

Evidence-Based Decision Making/Critical Reasoning 2.6 3.0 2.9

Cultural and Interpersonal Competence 3.3 3.6 3.6

Professionalism/Ethics 3.3 3.5 3.5

Assessment 3.0 3.5 3.3

Intervention/Supervision/Consultation 3.0 3.5 3.3

Table 59 presents the mean, standard deviation, and number of respondents completing the Frequency ratings for the 37 competency statements , and the frequency distributions for the associated scale points. A review of the results indicates wide variability in the mean Frequency ratings, from a low of 2.1 for four competencies to a high of 3.8 for one competency. One cluster, Professionalism/Ethics, contained two competency statements rated lowest and highest on the Frequency scale. In the case of one cluster, Evidence-based Decision Making/Critical Reasoning, all six of the Frequency ratings are ≤3.0,indicating that these competencies are called upon infrequently-to-frequently. Finally, a review of the four competencies with the lowest Frequency ratings reveals that these statements were most likely to demonstrate a pattern of ratings associated with high standard deviations, and, further, between 25% and 41% of the respondents indicated that they never/rarely performed these competencies.


Table 59 Frequency Distribution, Number, Mean, and Standard Deviation for

Frequency Ratings for Competency Statements

Never/ rarely

Infre- quently

Fre- quently

Very Freq Summary Statistics

% % % % n Mean SD

Evidence-Based Decision Making/Critical Reasoning

Select relevant research literature and critically review its assumptions, conceptualization, methodology, interpretation, and generalizability

7% 35% 42% 16% 402 2.7 (.8)

Interpret, evaluate, and integrate results of information-gathering and assessment processes within the context of scientific/professional knowledge to formulate and reformulate working hypotheses, conceptualizations, and recommendations (e.g., ongoing data collection)

9% 19% 39% 33% 401 3.0 (.9)

Articulate a rationale for decisions and psychological services that relies on objective supporting data (e.g., research results, base rates, epidemiological data)

7% 23% 44% 26% 400 2.9 (.9)

Use evidence-based algorithms and decision trees and other decision making tools in framing problems and solutions to minimize influence of biases and heuristics in decision making

25% 42% 24% 8% 398 2.1 (.9)

Utilize scientific method to conduct research in an ethical and legal manner 41% 25% 15% 18% 397 2.1 (1.1)

Cultural and Interpersonal Competence Never/ rarely

Infre- quently

Fre- quently


% % % % n Mean SD

Integrate and apply theory, research, professional guidelines, and personal understanding about social contexts to work effectively with diverse individuals, families, groups, communities, organizations, and research participants

1% 11% 45% 43% 402 3.3 (.7)

Communicate effectively with individuals, families, groups, communities, and/or organizations 1% 3% 18% 78% 399 3.7 (.5)

Integrate a collaborative perspective with all aspects of professional life 2% 8% 41% 48% 396 3.4 (.7)

Identify and manage interpersonal conflict between self and others 4% 36% 38% 22% 400 2.8 (.8)

Professionalism/Ethics Never/ rarely

Infre- quently

Fre- quently


% % % % n Mean SD

Demonstrate and promote values and behaviors commensurate with standards of practice, including ethics codes, laws, and regulations, in professional practice

0% 1% 21% 77% 401 3.8 (.5)

Advocate for changes in ethical, legal, regulatory, and/or practice standards, as necessary 25% 44% 23% 7% 401 2.1 (.9)

Identify and observe boundaries of competence in all areas of professional practice 1% 7% 37% 55% 394 3.5 (.6)


Never/ rarely

Infre- quently

Fre- quently


% % % % n Mean SD

Establish and maintain a process that promotes ethical decision-making 1% 7% 39% 52% 401 3.4 (.7)

Critically evaluate one’s own professional practice through self-reflection and feedback from others 1% 7% 52% 41% 399 3.3 (.6)

Develop and disseminate knowledge in accord with scientific and ethical principles 5% 18% 42% 36% 400 3.1 (.8)

Demonstrate respect for others in all areas of professional practice, including those with conflicting viewpoints

0% 5% 41% 54% 399 3.5 (.6)

Advocate for clients’ access to services (e.g. mental health, social services, employment) 6% 20% 38% 37% 400 3.0 (.9)

Accurately represent and document work performed in scholarship and professional practice 4% 9% 27% 61% 401 3.4 (.8)

Implement sound practice management skills (e.g., scheduling, resource management) 3% 6% 37% 54% 401 3.4 (.7)

Assessment Never/ rarely

Infre- quently

Fre- quently


% % % % n Mean SD

Apply knowledge of individual and cultural characteristics in assessment and diagnosis 2% 8% 39% 51% 399 3.4 (.7)

Apply evidence-based criteria in selection and use of assessment methods (e.g., psychometric properties, cost effectiveness, relevance)

6% 21% 36% 37% 399 3.0 (.9)

Demonstrate familiarity with models and techniques of interviewing 4% 16% 39% 41% 391 3.2 (.8)

Select or develop assessment instruments based on available normed data and/or criterion-reference standards, and address the limitations associated with the selection or development of such instruments

14% 25% 28% 34% 400 2.8 (1.0)

Administer and score instruments following current guidelines and psychometric research 11% 18% 20% 52% 399 3.1 (1.1)

Interpret and synthesize results from multiple sources (e.g., multiple methods of assessment, written documentation, interviewees) following current guidelines and psychometric research

7% 12% 29% 52% 397 3.3 (.9)

Formulate a diagnosis, recommendation, and/or professional opinion using relevant criteria and considering all assessment data (including information provided by other sources or professionals from other disciplines)

3% 7% 24% 66% 398 3.5 (.8)

Communicate assessment results in an integrative manner (e.g., psychological evaluation reports, feedback to clients, expert testimony)

6% 14% 28% 52% 398 3.3 (.9)


Never/ rarely

Infre- quently

Fre- quently


% % % % n Mean SD

Evaluate effectiveness of psychological services for individuals, couples, families, groups, communities, and/or organizations

16% 33% 30% 21% 398 2.6 (1.0)

Develop and communicate formative and summative supervisee evaluations 41% 19% 24% 16% 398 2.1 (1.1)

Intervention/Supervision/ Consultation Never/ rarely

Infre- quently

Fre- quently


% % % % n Mean SD

Select interventions to address needs and/or treat specific disorders in individuals, families, groups, organizations and/or communities based on available research evidence as well as contextual and other factors

4% 9% 40% 47% 401 3.3 (.8)

Apply interventions to treat specific disorders or address issues in individuals, families, groups, organizations and/or communities based on available research evidence and contextual factors

8% 13% 35% 44% 401 3.1 (.9)

Select evidence-based interventions to promote health and well-being (e g , positive psychology), enhance growth and performance (e g , coaching, performance management, conflict resolution, and organizational change), reduce risk factors, or increase resilience

7% 26% 35% 32% 399 2.9 (.9)

Apply evidence-based interventions to promote health and well-being (e g , positive psychology), enhance growth and performance (e g , coaching, performance management, conflict resolution, organizational change),and reduce risk factors and increase resilience

13% 29% 32% 27% 399 2.7 (1.0)

Engage in collaborative intervention planning with client(s) and stakeholders 7% 17% 40% 37% 399 3.1 (.9)

Modify interventions based on knowledge of individual and cultural characteristics, situational and environmental variables, emerging information, outcomes data, and current research

5% 9% 40% 46% 400 3.3 (.8)

Engage in consultation and collaboration across professions 3% 17% 32% 48% 399 3.3 (.8)

Develop and implement a training plan for supervisees, students, and trainees 41% 23% 19% 16% 396 2.1 (1.1)

Table 60 presents the mean, standard deviation, and number of respondents completing the Criticality ratings for the 37 competency statements, and the frequency distributions for the associated scale points. A review of the results indicates less variability in the mean Criticality ratings than for the Frequency ratings, from a low of 2.5 or one competency to a high of 3.9 for one competency. Twenty-one of the 37 competencies were rated ≥3.5, suggesting that these competencies were considered to be highly critical to optimizing outcomes for the patient/client/public. Only in one cluster, Evidence-based Decision Making/Critical Reasoning, did all of the ratings fall below a mean of 3.5 on the Criticality scale.


Table 60

Frequency Distribution, Number, Mean, and Standard Deviation for Criticality Ratings for Competency Statements


Not Min Mod High Summary Statistics

% % % % n Mean SD


2% 15% 44% 40% 193 3.2 (.8)

Interpret, evaluate, and integrate results of information-gathering and assessment processes within the context of scientific/professional knowledge to formulate and reformulate working hypotheses, conceptualizations, and recommendations (e.g., ongoing data

3% 13% 32% 53% 190 3.3 (.8)


2% 15% 43% 41% 190 3.2 (.8)


9% 43% 33% 14% 189 2.5 (.8)


Cultural and Interpersonal Competence


% % % % n Mean SD


2% 6% 22% 71% 194 3.6 (.7)

Communicate effectively with individuals, families, groups, communities, and/or organizations

1% 2% 15% 82% 194 3.8 (.5)




Professionalism/Ethics Not Min Mod High Summary Statistics

% % % % n Mean SD


0% 2% 8% 90% 194 3.9 (.4)

Advocate for changes in ethical, legal, regulatory, and/or practice standards, as necessary

2% 19% 47% 32% 194 3.1 (.8)

Identify and observe boundaries of competence in all areas of professional practice

1% 3% 16% 81% 192 3.8 (.5)


Critically evaluate one’s own professional practice through self-reflection and feedback from others

0% 3% 26% 72% 195 3.7 (.5)



0% 5% 31% 64% 195 3.6 (.6)


Accurately represent and document work performed in scholarship and professional practice

2% 7% 28% 64% 196 3.5 (.7)


Assessment Not Min Mod High Summary Statistics

% % % % n Mean SD



2% 8% 30% 60% 194 3.5 (.7)



4% 9% 30% 58% 196 3.4 (.8)

Administer and score instruments following current guidelines and psychometric research 4% 5% 15% 76% 196 3.6 (.8)



2% 3% 19% 76% 195 3.7 (.6)


2% 1% 15% 82% 196 3.8 (.6)


2% 4% 18% 76% 195 3.7 (.6)


3% 10% 45% 42% 195 3.3 (.8)


Intervention/Supervision/ Consultation


% % % % n Mean SD


2% 1% 22% 75% 196 3.7 (.6)


1% 5% 23% 71% 194 3.6 (.6)


2% 10% 41% 47% 194 3.3 (.7)


3% 12% 37% 48% 193 3.3 (.8)




3% 3% 31% 63% 194 3.6 (.7)


Develop and implement a training plan for supervisees, students, and trainees 10% 8% 41% 42% 192 3.1 (.9)

Table 61 presents the mean, standard deviation, and number of respondents completing the Importance ratings for the 37 competency statements, and the frequency distributions for the associated scale points. The results indicates a mid-level of variability in the mean Importance ratings compared to either the Frequency or the Criticality ratings, from a low of 2.4 for two competencies to a high of 3.9 for two competencies. Twenty of the 37 competencies were rated ≥3.5 on the Importance scale, indicating that these competencies were viewed as making a very important contribution to performance in practice during the past year. Finally, a side-by-side comparison of the Criticality and Importance ratings indicates that the respondents were somewhat more likely to consider the specific competencies as critical for optimizing outcomes for patients/clients/public than they were to consider the competencies as important to their own practice during the past year.

Table 61 Frequency Distribution, Number, Mean, and Standard Deviation for

Importance Ratings for Competency Statements


% % % % n Mean SD



4% 16% 44% 36% 202 3.1 (.8)

Interpret, evaluate, and integrate results of information-gathering and assessment processes within the context of scientific/professional knowledge to formulate and reformulate working hypotheses, conceptualizations, and recommendations (e.g., ongoing data

7% 16% 26% 52% 199 3.2 (.9)



% % % % n Mean SD


5% 16% 36% 44% 199 3.2 (.9)


20% 38% 23% 18% 197 2.4 (1.0)


Cultural and Interpersonal Competence


% % % % n Mean SD


0% 6% 27% 67% 207 3.6 (.6)

Communicate effectively with individuals, families, groups, communities, and/or organizations

0% 2% 10% 88% 206 3.9 (.4)



Professionalism/Ethics Not Min Mod High Summary Statistics

% % % % n Mean SD


0% 1% 7% 92% 206 3.9 (.4)

Advocate for changes in ethical, legal, regulatory, and/or practice standards, as necessary

8% 26% 33% 33% 205 2.9 (.9)

Identify and observe boundaries of competence in all areas of professional practice

0% 2% 23% 75% 204 3.7 (.5)


Critically evaluate one’s own professional practice through self-reflection and feedback from others

0% 3% 25% 73% 204 3.7 (.5)



% % % % n Mean SD



0% 2% 28% 70% 206 3.7 (.5)


Accurately represent and document work performed in scholarship and professional practice

3% 4% 21% 72% 205 3.6 (.7)


Assessment


% % % % n Mean SD



6% 11% 37% 46% 204 3.2 (.9)



9% 15% 28% 48% 204 3.1 (1.0)

Administer and score instruments following current guidelines and psychometric research 9% 8% 18% 65% 204 3.4 (1.0)


6% 7% 19% 67% 202 3.5 (.9)


2% 3% 12% 83% 203 3.8 (.6)


4% 6% 18% 71% 202 3.6 (.8)



% % % % n Mean SD


12% 17% 35% 36% 202 2.9 (1.0)


Intervention/Supervision/ Consultation


% % % % n Mean SD


3% 7% 24% 66% 205 3.5 (.8)


6% 4% 25% 64% 203 3.5 (.9)


7% 12% 35% 46% 205 3.2 (.9)


11% 15% 31% 43% 204 3.1 (1.0)



3% 4% 29% 64% 205 3.5 (.7)


Develop and implement a training plan for supervisees, students, and trainees 27% 17% 22% 34% 204 2.6 (1.2)


Appendix 14 and Appendix 15 present the Frequency, Importance, and Criticality ratings of the competencies for the U.S. and Canadian respondents, and the recently and less-recently licensed/registered psychologists.

Competency Exemplars. As described previously, for each competency statement, the PATF developed a hypothetical thread of behavioral exemplars, including a minimum of four exemplars typifying the level of competency to be demonstrated at each of the following four levels:

• During entry-level supervised practice (practicum), level 1 • During advanced-level supervised practice (internship), level 2 • At entry to practice through the first 3 years of independent practice, level 3 • After 3 years of independent practice, level 4

Table 62 documents the percent of support for acquisition of the behavioral exemplars at each level, and presents a graphic representation of the support of the development of the hypothetical thread. Validation of the hypothetical threads can be evaluated by the relative differences in percentage support for acquisition at various levels; for example,

• highest percentage of support for acquisition of level 1 behaviors during entry-level supervised practice,

• highest percentage of support for acquisition of level 2 behaviors during advanced-level supervised practice,

• highest percentage of support for acquisition of level 3 behaviors during entry to practice through 3 years of practice, and

• highest percentage of support for acquisition of level 4 behaviors after 3 years of practice. Similarly, support for the hypothetical threads can be visually depicted by the patterns of the bar graphs; that is,

• longest bars associated with level 1 behavioral exemplars under the column labeled “during entry-level supervised practice” and decreasingly shorter length bars for levels 2, 3, and 4 behavioral exemplars, respectively;

• longest bars associated with level 2 behavioral exemplars under the column labeled “during advanced-level supervised practice,” shorter length bars for levels 1 and 3, and shortest length bar for level 4 behavioral exemplars;

• longest bars associated with level 3 behavioral exemplars under the column labeled “entry to practice through 3 years” and shorter length bars for levels 4 and 2, and shortest length bar for level 1 behavioral exemplars; and

• longest bars associated with level 4 behavioral exemplars under the column labeled “after three years practice” and decreasingly shorter length bars for levels 3, 2, and 1 behavioral exemplars, respectively.


Table 62 documents the results related to the demonstration of various behaviors associated with the 37 competencies. A detailed review of the results indicates a general level of support for the developmental unfolding of the competencies as operationalized by the exemplars. For example, in regard to the Evidence-based Decision Making/Critical Reasoning competency cluster, respondents were most likely to support the acquisition of level 1 exemplars at entry-level, supervised practice (practicum), and the acquisition of levels 2 and 3 exemplars during either advanced supervised practice (internship) or the first three years of independent practice. In the case of level 4 exemplars, respondents were most likely to indicate that the behavior should be demonstrated either during the first three years of practice or subsequent to the first three years. In regard to the Professional/Ethics competency cluster, the unfolding patterns of competence vary as a function of the specific competency. For example,

• Competency 4.01—Demonstrate and promote value and behaviors commensurate with standards of practice, including ethics, codes, laws, and regulations, in professional practice: Respondents were most likely to support the acquisition of level 1 exemplars at entry-level, supervised practice (practicum), and the acquisition of levels 2 and 3 exemplars during either advanced supervised practice (internship) or the first three years of independent practice, and finally, in the case of level 4 exemplars, respondents were most likely to indicate that the behavior should be demonstrated either during the first three years of practice or subsequent to the first three years.

• Competency 4.02—Advocate for changes in ethic, legal, and regulatory, and/or practice standards as necessary: Respondents supported the later acquisition of level 1, 2, 3, and 4 behaviors than was initially believed to be the case. Specifically, level 1 and 2 behaviors were rated as emerging at the third milestone, and level 3 behaviors were rated as emerging at the time of the fourth milestone.

Accordingly, whereas a review of the results supports the general assumptions made about the emergence of competence across the four milestones, care should be taken in regard to the evaluation of the emergence of each set of exemplars related to the various competencies. Care should be taken in regard to the application of the results and the assumptions made about the emergence of various exemplars.


Table 62 Support for Acquisition of Behavioral Exemplars

Entry- level supervised (practicum)

Advanced supervised

(internship)

Entry to practice thru

3 yearsAfter 3 years

practice

Sample Size

L1 Identify and summarize relevant empirical articles in peer-reviewed journals 64% 23% 8% 5% 193L2 Critically evaluate the quality of the research reported 47% 37% 12% 4% 193L2 Identify implications for practice 22% 50% 20% 7% 191L3 Identify best quality and most appropriate studies and integrate findings across studies taking into account

strengths and weaknesses of the methods26% 43% 23% 8% 192

L3 Interpret and communicate empirical research results in a manner that is easily understood by non-scientific audiences

26% 42% 26% 7% 192

L3 Apply critical research findings with attention to challenges in generalizability 24% 43% 24% 9% 193L4 Guide others in the critical evaluation and application of relevant empirical literature 8% 29% 41% 22% 192

L1 Identify and obtain, with supervision, applicable data (e.g., clinical, observational, organizational, and/or empirical) and integrate with existing theoretical models to develop working hypotheses

46% 44% 8% 3% 193

L2 Integrate situation-specific data and applicable research to develop case conceptualizations and recommendations for uncomplicated situations

32% 56% 9% 3% 192

L2 Articulate the practical implications of empirical research 35% 45% 18% 3% 192L2 Formulate new or alternative hypotheses based on applicability of existing data 28% 53% 16% 3% 191L2 Develop preliminary hypotheses to discuss with supervisors 79% 19% 1% 1% 193L3 Independently synthesize clinical and scientific data with contextual factors to develop working hypotheses

and recommendations across a range of typical problems 9% 38% 44% 9% 193

L3 Reformulate working hypotheses and recommendations based on emerging data 24% 55% 19% 2% 193L3 Supervise others in the integration of situation-specific data and applicable research to develop case

conceptualizations and recommendations1% 14% 45% 40% 193

L4 Train, consult with, and supervise others in the integration and synthesis of clinical and scientific data with contextual factors to develop working hypotheses and recommendations across a range of typical problems complex or unusual situations.

3% 10% 38% 48% 193

0201. Select relevant research literature and critically review its assumptions, conceptualization, methodology, interpretation, and generalizability

Cluster 02. Evidence-Based Decision Making/Critical Reasoning

0202. Interpret, evaluate, and integrate results of information-gathering and assessment processes within the context of scientific/professional knowledge to formulate and reformulate working hypotheses, conceptualizations, and recommendations (e.g., ongoing data collection)


Entry- level supervised (practicum)

Advanced supervised

(internship)


3 yearsAfter 3 years

practice

Sample Size

L1 Articulate the need for evidence-based rationales to support recommendations 43% 42% 13% 1% 193L2 Articulate an evidence-based rationale for routine or uncomplicated decisions and recommendations 38% 51% 10% 2% 192L3 Articulate a rationale for decisions and recommendations that integrates situation or case- specific,

social/contextual, and scientific data in typical cases 20% 63% 17% 1% 192

L3 Train or supervise others to articulate evidence-based reasoning for decisions and recommendations 1% 13% 45% 41% 192L3 Articulate evidence-based rationale for decisions, recommendations, and opinions to clients, professionals,

and the public10% 45% 37% 8% 192

L4 Articulate a rationale for decisions and recommendations that integrates situation or case- specific, social/contextual, and scientific data in both typical and unusual cases

6% 44% 38% 12% 193

L1 Identify biases and heuristics that impact general decision-making 45% 38% 16% 2% 193L2 Under supervision, apply decision-making tools designed to minimize the impact biases and heuristics 53% 46% 1% 1% 192L3 Independently identify and apply decision-making tools designed to minimize the impact of biases and

heuristics9% 44% 43% 5% 193

L4 Assist others to identify biases and heuristics and to apply appropriate decision-making tools designed to minimize their impact

3% 28% 44% 25% 193

L1 Formulate researchable questions regarding the care of clients/patients as individuals, families , systems, organizational systems or communities

42% 40% 16% 3% 193

L1 State current ethical and legal responsibilities in the conduct of research 67% 27% 4% 1% 193L2 Identify gaps in current scientific/professional knowledge and articulate implications for practice 17% 49% 24% 10% 193L2 Under supervision, design and conduct research activities that conform to current ethical and legal

requirements57% 38% 4% 1% 193

L3Independently design and conduct research activities that conform to current ethical and legal requirements 10% 33% 48% 10% 193

L4 Supervise empirical tests of hypotheses 2% 24% 53% 20% 193L4 Supervise others’ research according to current ethical and legal best practices 2% 8% 56% 34% 193L4 Communicate evidence-based and qualitative research findings and/or feedback in an understandable manner

to applicable populations16% 60% 22% 3% 193

0203. Articulate a rationale for decisions and psychological services that relies on objective supporting data (e.g., research results, base rates, epidemiological data)

0204. Use evidence-based algorithms and decision trees and other decision making tools in framing problems and solutions to minimize influence of biases and heuristics in decision making

0205. Utilize scientific method to conduct research in an ethical and legal manner


Entry- level supervised

(practicum)

Advanced supervised

(internship)

Entry to practice

thru 3 years

After 3 years

practice Sample

Size

L1 Demonstrate scientific and personal knowledge about how social contexts shape behaviors 42% 43% 13% 2% 191L1 Engage in respectful interactions that reflect knowledge of the scientific literature regarding individual and

cultural diversity61% 29% 8% 2% 191

L1 Engage in self-reflection regarding personal attitudes, assumptions, and behaviors 78% 17% 5% 0% 190L2 With supervision, modify own behavior based on scientific and personal knowledge about how social

contexts shape behaviors 61% 32% 6% 1% 191

L2 Monitor personal attitudes, assumptions, and behaviors when working with diverse populations and in interpersonal relationships

57% 33% 9% 2% 190

L2 Follow professional guidelines for providing professional services to specific diverse populations, when available

53% 33% 13% 1% 190

L2 Understand where research shows culture is not a significant factor (e.g., differential validity) 37% 48% 12% 4% 191L2 Understand where research findings about culture are inappropriate or biased (e.g., Bell curve hypothesis) 47% 43% 8% 2% 190L3 Independently monitor and modify own and others’ behaviors in professional work by applying scientific

and personal knowledge about how social contexts shape behaviors 13% 27% 46% 14% 190

L3 When guidelines for working with specific, diverse populations are either unavailable or limited, communicate their strengths and limitations

16% 50% 31% 3% 189

L3 Apply culturally appropriate repertoire of skills, techniques, and behaviors to maximize effectiveness of professional services

22% 43% 29% 5% 189

L4 Facilitate professional and scientific dialogue about how social contexts shape behaviors 11% 34% 44% 11% 191L4 Provide consultation/training/supervision about individual and cultural diversity issues 1% 16% 48% 35% 189L4 Develop and disseminate theory, research, and professional guidelines about the level of impact of social

context, culture, and diversity on individuals, families, groups, communities, organizations, and research 6% 21% 36% 37% 190

0301. Integrate and apply theory, research, professional guidelines, and personal understanding about social contexts to work effectively with diverse individuals, families, groups, communities, organizations, and research participants



(practicum)

Advanced supervised

(internship)

Entry to practice

thru 3 years

After 3 years

practice Sample

Size

L1 Listen and communicate respectfully while showing empathy for others 92% 8% 0% 0% 191L1 Demonstrate knowledge of importance of verbal and non-verbal cross-cultural, social, and communication

cues55% 36% 8% 1% 192

L2 Collaborate effectively in routine professional interactions with individuals, families, groups, communities, and/or organizations

34% 46% 19% 1% 192

L2 Recognize when verbal and non-verbal cross-cultural social and communication cues are occurring 41% 46% 11% 2% 193L3 Collaborate effectively in complex situations with individuals, families, groups, communities, and/or

organizations12% 35% 45% 8% 193

L3 Use appropriate verbal and non-verbal cross cultural social and communication cues 45% 41% 11% 3% 193L3 Develop insight regarding emerging situations that involve cross-cultural, social and interpersonal differences 37% 48% 12% 3% 193L4 Collaborate effectively in conflictual situations with individuals, families, groups, communities, and/or

organizations15% 41% 33% 11% 193

L4 Provide consultation/training/supervision on appropriate verbal and non-verbal cross cultural social and communication cues

2% 16% 48% 35% 193

L1 Identify multiple systems involved in the situation 40% 44% 15% 1% 193L1 Recognize the perspectives of diverse groups or systems who are the subject of intervention or research 41% 42% 17% 1% 192L2 Communicate with key individuals within the multiple systems 24% 47% 29% 1% 193L2 Under supervision, collaborate with diverse groups or systems that are the subject of intervention or

research 42% 46% 11% 1% 193

L3 Collaborate and participate in change across systems 18% 42% 28% 12% 192L3 Incorporate perspectives of diverse groups or systems that are the subject of intervention or research 22% 47% 27% 4% 193L4 Initiate and manage change across systems 6% 24% 41% 30% 191L4 Involve representatives of diverse groups or systems in the development of interventions or research 11% 39% 38% 12% 190

0302. Communicate effectively with individuals, families, groups, communities, and/or organizations

0303. Integrate a collaborative perspective with all aspect of professional life



(practicum)

Advanced supervised

(internship)


3 years

After 3 years

practice Sample

Size

L1 Recognize interpersonal conflict 84% 16% 1% 0% 192L2 Demonstrate active problem solving to manage interpersonal conflict 45% 42% 12% 1% 192L2 Seek supervision or consultation to examine one’s own reactions and behaviors to manage interpersonal

conflict73% 23% 5% 0% 191

L3 Negotiate difficult and complex interpersonal relationships between self and others 21% 41% 32% 6% 192L3 Use experienced peers’ feedback to examine and modify one’s own reactions and behavior when managing

interpersonal conflict60% 30% 10% 1% 192

L4 Mediate interpersonal conflict between others 12% 41% 36% 11% 192L4 Implement strategies to mitigate conflict between self and others 39% 43% 15% 4% 192L4 Provide consultation to others on how to mediate interpersonal conflict 5% 21% 51% 23% 192

L1 Demonstrate integrity in practicum experience (e.g., keep accurate and timely records, utilize constructive criticism, seek supervision and guidance)

85% 14% 1% 1% 198

L1 Demonstrate ethical behavior in interactions with clients 83% 15% 2% 1% 198L1 Demonstrate knowledge of ethics codes and legal principles affecting psychologists 51% 42% 6% 1% 198L2 Recognize ethical dilemmas that arise in psychological practice and notify supervisor 55% 43% 1% 1% 197L2 Apply ethics code and laws in all situations, seeking guidance on more complex issues 22% 48% 26% 5% 196L2 Communicate ethical issues (e.g., limits of confidentiality, dual relationships, third party payment, uses of

data or professional services) to non-psychologists 30% 43% 25% 2% 197

L3 Integrate ethical code and law, including local jurisprudence, when faced with ethical dilemmas 8% 30% 52% 11% 195L3 Demonstrate understanding of when professional consultation is appropriate and necessary 24% 47% 28% 1% 198L3 Update knowledge and skills relevant to psychological practice on an ongoing basis 21% 18% 51% 10% 198L4

Interpret complex issues and dilemmas when ethical codes and relevant laws may be ambiguous or in conflict 1% 14% 45% 40% 198

L4 Provide consultation to other members of the profession 1% 5% 28% 66% 198

0304. Identify and manage interpersonal conflict between self and others

Cluster 04: Professionalism/Ethics0401. Demonstrate and promote values and behaviors commensurate with standards of practice, including ethics codes, laws, and regulations, in professional practice



(practicum)

Advanced supervised

(internship)

Entry to practice thru 3 years

After 3 years

practice Sample

Size

L1 Demonstrate basic awareness of how to advocate for change in ethical, legal and practice standards 6% 22% 42% 30% 196L2 Identify areas where ethics code or changes of law may benefit clients 2% 21% 37% 41% 196L2 With supervision, work for change in ethical, legal and practice standards 4% 32% 45% 19% 196L3 Develop alliances with relevant individuals and groups to formulate strategies for change in ethical, legal,

regulatory, or practice standards2% 6% 34% 58% 194

L3 Advocate for change in ethical, legal and practice standards 2% 6% 38% 54% 196L4 Take leadership role in advocating for change in ethical, legal and practice standards 0% 4% 20% 77% 196L4

Provide expertise to and/or serve on boards or committees within your profession or within the public arena 0% 1% 21% 78% 196

L1 Engage in discussions with faculty and supervisors to define the concept of competence in the practice of psychology

53% 35% 8% 5% 196

L1 Accept direction from supervisors when greater expertise is needed 81% 15% 4% 1% 197L2 Seek knowledge of mechanisms to enhance competencies 34% 34% 29% 4% 197L2 Define personal limits of competence by recognizing strengths and weaknesses in practice areas 16% 43% 39% 2% 196L3 Develop insight into areas of strength and weaknesses and use this knowledge to guide scope of practice. 23% 40% 31% 6% 196L3 Develop mechanisms for specialization and plans to update knowledge. 7% 46% 38% 10% 197L3 Seek consultation with subject-matter experts to ascertain needed training and development before engaging

in specialty practice 17% 40% 38% 5% 196

L4 Identify and acquire specialty skills and expertise 8% 44% 34% 15% 197L4 Provide consultation or supervision in areas of specialty skills and expertise 0% 6% 36% 59% 197

L1 Engage actively in the learning process to acquire knowledge about ethical decision making processes 73% 21% 6% 0% 197L2 Develop an ethical decision making system that takes into account effects on others 47% 39% 10% 3% 198L2 Actively involve peers and supervisors to assess decision-making process 65% 23% 11% 1% 196L3 Demonstrate ethical decision-making in an ongoing manner in the professional environment 36% 39% 24% 1% 198L4 Develop environment for ethical practice 22% 24% 51% 3% 198L4 Model ethical decision making to others 20% 28% 32% 19% 198

0402. Advocate for changes in ethical, legal, regulatory, and/or practice standards, as necessary

0403. Identify and observe boundaries of competence in all areas of professional practice

0404. Establish and maintain a process that promotes ethical decision-making



(practicum)

Advanced supervised

(internship)


After 3 years

practice Sample

Size

L1 Establish basic skills or a framework for evaluating competence 38% 39% 19% 5% 197L2 Demonstrate knowledge of one’s own professional practice evaluative strategies. 11% 38% 45% 6% 197L2 Use supervision to resolve increasingly complex practice issues to enhance professional development 36% 43% 20% 2% 197L3 Identify and reconcile differences between one’s own evaluation of professional competence and the

evaluations of others22% 45% 28% 6% 197

L3 Use the literature and peer consultation to enhance self-reflection. 46% 29% 24% 1% 196L3 Use self-reflection to understand limits of current knowledge 47% 32% 20% 2% 197L3 Demonstrate personal responsibility when mistakes or oversights occur. 66% 23% 10% 1% 196L4 Develop insight into the breadth of the knowledge base necessary to practice in an increasingly sophisticated

manner13% 37% 40% 11% 197

L4 Use self-reflection to develop a plan to increase current knowledge and skills 46% 28% 23% 3% 197

L1 Consult the literature to enhance one’s knowledge of ethical and scientific foundations of the practice of psychology

60% 23% 14% 4% 198

L2 Critically evaluate the literature relevant to professional practice 26% 36% 25% 13% 196L2 Develop a personal heuristic integrating scientific knowledge and ethical precepts to inform practice 7% 34% 42% 17% 197L3 Share psychological knowledge with diverse groups (e.g., students, colleagues, clients, other professionals,

the public) in an unbiased manner14% 33% 37% 16% 196

L4 Advance the profession through active production and dissemination of knowledge. 5% 18% 39% 38% 197L4 Serve as expert in consultation with other psychologists 0% 1% 14% 85% 197

L1 Listen non-judgmentally to those with conflicting viewpoints 70% 27% 3% 1% 197L1 Develop an understanding of conflicting viewpoints within the practice of psychology 36% 45% 17% 2% 196L2 Be open to others’ points of views 90% 8% 2% 0% 197L2 Communicate with colleagues when conflicting viewpoints develop 39% 40% 21% 1% 197L2 With supervision, objectively examine conflicting viewpoints 63% 34% 2% 1% 197L3 Balance conflicting viewpoints by seeking clarification to increase understanding before making judgments 42% 42% 14% 2% 197L4 Integrate conflicting viewpoints into one’s thinking 39% 44% 14% 4% 197L4 Use a broad skill set to resolve conflicts and manage difficult interpersonal situations 20% 41% 32% 7% 197

0405. Critically evaluate one’s own professional practice through self-reflection and feedback from others

0406. Develop and disseminate knowledge in accord with scientific and ethical principles

0407. Demonstrate respect for others in all areas of professional practice, including those with conflicting viewpoints



(practicum)

Advanced supervised

(internship)


After 3 years

practice Sample

Size

L1 Recognize client’s needs for access to multiple services and resources 47% 43% 11% 0% 197L2 Demonstrate knowledge of multiple services and resources 18% 48% 29% 5% 196L2 Under guidance from others, develop a plan for client to access multiple services and resources 39% 49% 11% 1% 197L3 Independently facilitate client’s access to multiple services and resources 7% 31% 56% 5% 197L4 Advocate for systemic change to improve client access to multiple services and resources 3% 21% 43% 33% 196

L1Describe the basic principles of documentation (e.g., clinical documentation, non-clinical technical reports) 61% 34% 5% 1% 197

L1 Accurately document scholarly references 80% 17% 2% 1% 197L2 Keep complete and accurate records, with an understanding of the legal and ethical implications of

professional documentation 47% 43% 11% 0% 197

L2 Develop awareness of personal biases that may impact documentation 52% 42% 6% 1% 197L3

Report clinical, organizational, and research results accurately, avoiding personal biases and preconceptions 43% 45% 12% 1% 197

L3 Ensure adequate and appropriate credit is given to trainees and collaborators in scholarship 39% 29% 29% 3% 197L4 Provide consultation and supervision in the documentation of scholarship and professional work. 4% 14% 48% 35% 196

L1 Articulate basic understanding of practice management 6% 35% 51% 8% 197L2 Identify responsibilities, challenges, and processes of practice management 3% 23% 59% 15% 196L3 Develop a plan commensurate with laws, ethical guidelines, and fiscal constraints to manage professional

activities 2% 15% 67% 16% 197

L3Recognize the realities of client and human resource needs in establishing/maintaining professional practice 5% 22% 60% 13% 196

L4 Deploy resources to practice in fiscally and ethically sound manner. 2% 14% 71% 14% 197L4 Generate and employ strategies to enhance practice in situations with resource challenges/limitations 2% 17% 57% 24% 197

0408. Advocate for clients’ access to services (e.g. mental health, social services, employment)

0409 Accurately represent and document work performed in scholarship and professional practice

0410. Implement sound practice management skills (e.g., scheduling, resource management)



(practicum)

Advanced supervised

(internship)


After 3 years

practice Sample

Size

L1 Describe the research base regarding the robust features of assessment methods and tools (e.g., resistance to test bias, validity generalization)

20% 48% 21% 11% 196

L2 Integrate knowledge of individual and cultural characteristics in selecting assessment methods and tools and interpreting results

16% 51% 27% 6% 196

L3 Ensure that professional opinions, recommendations, and case formulations adequately reflect full consideration of psychometric properties of the assessment and individual and cultural characteristics

10% 52% 34% 5% 196

L4 Provide consultation to others on the impact of individual and cultural characteristics on assessment tools 1% 19% 49% 31% 196L4 Provide consultation to others on psychometric requirements in development of assessment tools. 2% 15% 39% 44% 196L4 Develop new assessment tools taking into account individual and cultural characteristics that can moderate

their validity. 3% 12% 28% 57% 194

L1 Demonstrate basic knowledge of psychometrics, assessment methods, models of assessment, and commonly assessed psychological constructs

56% 34% 9% 1% 195

L2 Select most appropriate tests based upon the constructs to be assessed, the psychometric properties of the tests, and the client population

14% 59% 26% 2% 196

L3 Integrate and analyze client needs, practical constraints, and diagnostic questions into the selection of assessment methods to be used

12% 53% 33% 2% 195

L3 Develop assessments (e.g., tests, structured employment interviews, assessment center exercises, experience records) based upon the appropriate psychometric properties, assessee population, and ethical and legal standards.

4% 14% 39% 43% 194

L4 Provide consultation to others on the evidence-based selection of appropriate assessment instruments 0% 15% 59% 26% 194L4 Provide consultation to others on methods for unique or difficult cases 0% 7% 38% 55% 195

L1 Use semi-structured and unstructured interviewing techniques, as appropriate, to obtain basic data 53% 37% 9% 1% 198L2

Adapt interview behaviors to take into consideration personal characteristics of interviewer and interviewee 32% 47% 19% 2% 198

L3 Demonstrate flexible, empathic, and accurate utilization of a broad range of interview techniques, collecting complex data and adapting questions as required by the situation

15% 54% 26% 6% 198

L3 Incorporate contextual information (e.g., reason for assessment, possible legal or forensic considerations) into the process of conducting an interview

29% 49% 19% 3% 198

L4 Provide consultation to others on interviewing practices 1% 19% 51% 30% 196

Cluster 05: Assessment

0502. Apply evidence-based criteria in selection and use of assessment methods (e.g., psychometric properties, cost effectiveness, relevance)

0503. Demonstrate familiarity with models and techniques of interviewing

0501. Apply knowledge of individual and cultural characteristics in assessment and diagnosis



(practicum)

Advanced supervised

(internship)


After 3 years

practice Sample

Size

L1 Use the norms or criterion-referenced standards, as appropriate, in assessments 52% 38% 10% 1% 198L2 Recognize the limitations of assessment instruments and their associated norms or criterion-referenced

standards for unique or complex assessment situations21% 49% 24% 6% 198

L3 Explicitly note the limitations associated with available norms or criterion-referenced standards and suggest alternative interpretations or explanations in communication of assessment results

15% 48% 31% 7% 196

L3 Review assessment results against multiple available norm sets to inform judgments 20% 52% 25% 4% 198L3 When evaluating assessment instruments, take the relative performance of the instrument and alternatives

into account.12% 51% 32% 5% 198

L4 Provide balanced consultation to others (users or developers) on the limitations of alternatives to assessment instruments, including where the assessment, while flawed, may still be superior to the alternative.

1% 13% 47% 39% 196

L4 When developing assessment instruments, take the relative performance of the instrument and alternatives into account.

6% 19% 39% 36% 197

L1 Administer and score the most commonly used assessment tools 65% 30% 5% 1% 195L2 Administer, score, and interpret, with supervision, a range of standardized assessment tools, including

personality and intellectual measures and self-report and performance measures42% 49% 7% 2% 197

L3 Independently administer, score, and interpret a broad range of assessment instruments 9% 38% 49% 5% 197L3 In situations requiring some alteration or modification of standard practice, administer, score, interpret, and

communicate assessment results 3% 40% 43% 14% 196

L3 Prepare others to administer, score, and interpret the most commonly-used assessment tools. 4% 21% 57% 18% 197L4 Provide consultation to others on the appropriate use of a broad range of assessment tools 1% 14% 51% 35% 195L4 Provide consultation to others on the alteration and modification of standard assessment practices 1% 9% 45% 45% 196L4 Prepare others to administer, score, and interpret more advanced assessment tools, 0% 12% 49% 39% 197

0505. Administer and score instruments following current guidelines and psychometric research

0504. Select or develop assessment instruments based on available normed data and/or criterion-reference standards, and address the limitations associated with the selection or development of such instruments



(practicum)

Advanced supervised

(internship)


After 3 years

practice Sample

Size

L1 With supervision, interpret results from a single instrument 83% 16% 1% 0% 197L1 Identify multiple instruments that assess the same construct 39% 50% 10% 1% 197L2 With supervision, interpret and integrate results from standardized tests results and interviews following

established guidelines53% 44% 3% 0% 197

L2 With supervision, identify the strengths and limitations of various types of assessment data (e.g., self-reports, ratings by others)

53% 45% 3% 0% 197

L3 Reconcile discrepancies between various sources of data when integrating results from multiple methods and informants

13% 60% 24% 3% 197

L3 Evaluate the adequacy of the data available from multiple sources 21% 61% 15% 3% 197L3 Gather additional data, as needed 59% 35% 6% 1% 197L4

Provide consultation to others on the interpretation and synthesis of data from multiple assessment sources 2% 16% 60% 22% 196

L1 Discuss relevant criteria for formulating a diagnosis, professional opinion, and/or professional recommendations

31% 45% 21% 2% 198L2 With supervision, formulate diagnoses, professional opinions, and professional recommendations using

assessment data for a restricted range of populations or clients39% 53% 9% 0% 198

L3 Formulate diagnoses, professional opinions, and professional recommendations using assessment data for a broad range of populations or clients

7% 43% 41% 9% 197

L3 Formulate diagnoses, professional opinions, and professional recommendations incorporating relevant theory and research, assessment data, and information from other professionals

7% 42% 43% 9% 197

L4 Formulate diagnoses, professional opinions, and professional recommendations in complex cases 2% 31% 45% 22% 198L4 Provide consultation to others on formulating diagnoses, professional opinions, and professional

recommendations using assessment data for broad range of populations or clients1% 14% 49% 36% 198

0506. Interpret and synthesize results from multiple sources (e.g., multiple methods of assessment, written documentation, interviewees) following current guidelines and psychometric research

0507. Formulate a diagnosis, recommendation, and/or professional opinion using relevant criteria and considering all assessment data (including information provided by other sources or professionals from other disciplines)



(practicum)

Advanced supervised

(internship)


After 3 years practice

Sample Size

L1 Describe content and organization of assessment reports 64% 31% 5% 1% 196L1 Present results from a single instrument in a simulated setting (e.g., classroom) 84% 14% 2% 1% 195L2 Write complete basic psychological reports, summarizing test data by functional domains and/or sources of

information 47% 46% 6% 2% 196

L2 With supervision, orally communicate the results of psychological assessments 59% 38% 2% 1% 196L3 Prepare assessment reports integrating all available data, while recognizing and describing limitations of the

data26% 61% 11% 3% 195

L3 Verbally communicate the results from assessment reports in an integrative manner 24% 63% 12% 1% 196L3 Reconcile apparent discrepancies in assessment findings 20% 63% 16% 1% 196L4 Provide consultation to others on the communication of integrated assessment results 1% 23% 55% 21% 196

L1 Describe assessment methods and instruments used for program effectiveness evaluation (e.g., psychological services, training and educational programs)

12% 35% 37% 16% 193

L2 Under supervision, evaluate the effectiveness of programs , including one’s own service provision 14% 58% 26% 2% 197L3 Independently plan, design, and conduct comprehensive evaluations of programs, including one’s own

service provision1% 12% 48% 39% 195

L4 Provide consultation to others on the evaluation of programs (e.g., psychological services, training and educational programs).

2% 4% 40% 55% 196

L1 Engage in collaborative evaluation of one’s own performance with supervisor 56% 32% 11% 2% 198L2 Provide ongoing evaluations to supervisees with the assistance of one’s own supervisor 3% 46% 43% 8% 196L3 Formulate evaluation strategies to use with supervisees and regularly conduct formative and summative

evaluations 2% 15% 52% 31% 197

L3 Incorporate knowledge of impact of one’s own biases when evaluating supervisees 4% 34% 46% 16% 197L4 Provide consultation to others on the evaluation of supervisees 1% 8% 40% 52% 197

0508. Communicate assessment results in an integrative manner (e.g., psychological evaluation reports, feedback to clients, expert testimony)

0509. Evaluate effectiveness of psychological services for individuals, couples, families, groups, communities, and/or organizations

0510. Develop and communicate formative and summative supervisee evaluations


L1 Demonstrate basic knowledge of common disorders or functional concerns, and major approaches to intervention

69% 25% 6% 0% 191L1 Identify appropriateness of various interventions or approaches to treatment for different presenting

problems 30% 52% 18% 0% 191

L2 Conceptualize cases or settings with sufficient detail to inform choice of intervention or treatment on the basis of research and clinical literature and supervisory input

23% 65% 11% 1% 190

L2 Use clearly articulated theories of change to inform intervention choices 28% 56% 14% 2% 190L2 Use evidence-based assessment mechanisms to inform selection of interventions 38% 50% 12% 1% 191L2 Develop and present intervention treatment rationales and intervention alternatives as a result of active

participation in the supervision process 26% 53% 14% 7% 191

L3 Independently develop a range of intervention choices and alternatives by using assessment tools, research, theory , consultation, individual experience and judgment, and client preferences

6% 30% 53% 12% 191

L4 Develop or adapt integrated conceptualizations and intervention plans for complex cases that consider a range of interventions and integrate assessment tools, research, theory , consultation, individual experience and judgment, and client preferences in complex cases

3% 32% 40% 25% 191

L4 Independently develop a range of intervention choices and alternatives by using assessment tools, research, theory , consultation, individual experience and judgment, and client preferences

6% 32% 45% 17% 190

L1 Use relational skills to establish and maintain working relationships 72% 19% 8% 1% 190L1 Implement a limited range of selected interventions under supervision 76% 23% 2% 0% 190L2 Apply and modify interventions guided by assessment, research literature and supervisory input 29% 58% 11% 2% 189L2 Identify issues to be addressed in supervision 68% 25% 5% 2% 189L2 Identify personal strengths and weaknesses in the process of conceptualization, assessment, and

intervention 45% 43% 11% 1% 189

L2 Identify personal strengths and weaknesses in establishing and maintaining working relationships 59% 34% 7% 1% 189L3 Independently implement and modify interventions 5% 33% 56% 6% 190L3 Enhance professional intervention skills through awareness of both personal and professional limitations and

strengths 26% 45% 28% 1% 189

L3 Enhance professional intervention skills through participation in continuing education 25% 21% 48% 5% 189L4 Independently implement and modify interventions in complex cases 2% 21% 58% 20% 190L4 Assist others in the development of intervention skills 3% 29% 47% 22% 189

0601. Select interventions to address needs and/or treat specific disorders in individuals, families, groups, organizations and/or communities based on available research evidence as well as contextual and other factors

Cluster 06. Intervention/Supervision/Consultation

0602. Apply interventions to treat specific disorders or address issues in individuals, families, groups, organizations and/or communities based on available research evidence and contextual factors



(practicum)

Advanced supervised

(internship)


After 3 years

practice

Sample Size

L1 Demonstrate basic knowledge of major theories and applications of principles of health promotion, performance enhancement, well-being, and risk reduction

55% 32% 12% 1% 191

L1 Identify appropriateness of various theories and applications for different presenting problems 31% 53% 14% 2% 192L2 Conceptualize interventions, guided by research, theory, and supervisory input 30% 59% 9% 2% 191L2 Articulate an established theory of wellness, performance optimization or organizational change health

promotion, performance enhancement, well-being, and risk reduction to inform intervention choice14% 44% 30% 13% 190

L3 Independently develop, select and present the most appropriate intervention from a range of options based on assessment, research, theory, consultation, empirical evidence, ones’ own professional experience and judgment, client preferences, or situational circumstances

3% 19% 54% 23% 191

L4 Evaluate effectiveness of interventions devised by others 10% 37% 35% 18% 191L4 Guide or direct others in the development, selection, implementation and evaluation of intervention plans 3% 14% 51% 33% 192

L1 Understand and articulate the need to establish and maintain working relationships required for successful implementation of interventions

55% 36% 9% 0% 190

L1 Implement selected techniques or interventions on the basis of research, theory, and supervisory input 38% 52% 8% 1% 190L2 Employ basic relational skills to establish and maintain working relationships needed for successful

implementation of interventions 71% 24% 6% 0% 190

L2 Implement a wide range of techniques and interventions based on an advanced understanding of research, theory, and supervisory input

3% 43% 34% 21% 190

L3 Independently develop, implement and assess interventions 1% 26% 60% 13% 189L3 Modify interventions as indicated by outcomes 16% 59% 24% 1% 190L3 Incorporate professional limitations and enhance professional intervention skills through continuing

education 13% 19% 63% 5% 190

L4 Independently develop, implement and assess interventions in complex or high-stakes circumstances 1% 8% 47% 44% 190L4 Guide or direct others in the conceptualization, implementation, and assessment of interventions 1% 13% 56% 30% 190

0603. Select evidence-based interventions to promote health and well-being (e.g., positive psychology), enhance growth and performance (e.g., coaching, performance management, conflict resolution, and organizational change), reduce risk factors, or increase resilience

0604. Apply evidence-based interventions to promote health and well-being (e g , positive psychology), enhance growth and performance (e g , coaching, performance management, conflict resolution, organizational change),and reduce risk factors and increase resilience (e g , population-level interventions)



(practicum)

Advanced supervised

(internship)


3 years

After 3 years

practice

Sample Size

L1 With supervisory input, describe basic intervention options and processes to clients or stakeholders 59% 36% 3% 2% 192L1 Use stakeholder feedback to inform intervention planning 28% 53% 18% 2% 192L1 Use information derived from multiple sources to develop basic intervention options and processes 36% 51% 13% 1% 191L2 With supervisory input, integrate client and stakeholder opinions, preferences, readiness for change, or

potential for improvement into intervention plan 32% 64% 3% 1% 192

L2 With supervisory input, select an intervention plan in collaboration with client(s) or stakeholder(s) 49% 48% 3% 1% 191L3 Independently develop implement, evaluate, and modify comprehensive, negotiated intervention plans with

client or stakeholders 2% 21% 69% 8% 191

L4 In complex situations, independently develop implement, evaluate, and modify create comprehensive, negotiated intervention plans with client and stakeholders

1% 10% 47% 42% 192

L1 Demonstrate understanding of factors necessitating modification of interventions, relying heavily on supervisory input

57% 38% 5% 0% 191

L2 Recognize when modification of intervention may be required 40% 52% 8% 0% 191L2 Present options for modifying interventions to supervisor 48% 47% 4% 0% 190L2 Modify ongoing interventions in straightforward circumstances 34% 57% 9% 0% 191L3 Integrate individual and cultural client characteristics in intervention plans 46% 42% 12% 0% 191L3 Continually evaluate, modify, and assess the effectiveness of interventions, considering client characteristics,

emerging information, outcomes data, and current research 19% 46% 32% 3% 191

L3Consult with qualified peers when facing the need to modify interventions in new and unfamiliar situations 26% 31% 42% 2% 190

L4 Modify intervention in complex or high stakes situations, considering emerging information, outcomes data, current research literature, and the seriousness of consequences if the intervention is not altered.

3% 20% 49% 28% 191

L4 Guide others in strategies of development, evaluation and modification of interventions 1% 16% 57% 27% 191

L1 Possess knowledge of roles and scope of practice of various professions 27% 39% 25% 9% 191L2 Observe, interact with, and share information with other professionals in multidisciplinary settings 28% 46% 22% 4% 192L3 Tailor consultation requests and provision of information based on knowledge of others’ professional needs

and viewpoint 5% 35% 46% 14% 192

L4 Enhance other professionals’ knowledge of psychological theories, decision-making strategies, and interventions when tailoring consultation requests and providing information

4% 31% 45% 20% 192

0607. Engage in consultation and collaboration across professions

0605. Engage in collaborative intervention planning with client(s) and stakeholders

0606. Modify interventions based on knowledge of individual and cultural characteristics, situational and environmental variables, emerging information, outcomes data, and current research



(practicum)

Advanced supervised

(internship)


After 3 years

practice

Sample Size

L1 State supervisory roles and responsibilities 24% 33% 34% 9% 189L2 Identify own supervisory needs and reactions to supervision 49% 35% 14% 2% 190L2 Describe various supervisory models 18% 39% 33% 10% 191L2 With supervisory support, develop and implement a training plan for a supervisee, student or trainee 5% 45% 45% 6% 191L2 With supervisory support, identify dual or multiple roles and state how these affect supervision 35% 48% 15% 3% 191L3 Using models of supervision, develop and implement training plans tailored to individual needs of

supervisees, students, and trainees 1% 12% 45% 42% 191

L3 Balance demands of dual or multiple roles 13% 32% 46% 10% 190L4 Supervise or mentor others’ training of supervisees, students, and trainees 1% 10% 24% 65% 191L4 Develop and adapt structures and models of supervision or mentorship 2% 12% 42% 45% 191L4 Ensure ongoing evaluation to determine effectiveness of supervisory or mentorship plans 5% 21% 50% 24% 191L4 Guide or direct others in determining action plan to balance demands of dual or multiple roles 4% 13% 48% 35% 189

0608. Develop and implement a training plan for supervisees, students, and trainees


Implications of Findings Regarding the Assessment of Professional Competence At PATF Meeting 3, members of the Task Force began a discussion about the assessment of professional competence. The discussion focused on two future directions—the enhancement of the EPPP, including the use of alternate item types; and the development of new assessments that might complement the EPPP, including those offered during and after the completion of professional training programs. To enrich the discussion, members of the PES staff described various item types, including traditional multiple-choice items that use video and/or audio multimedia; scenario-based multiple-choice items; multiple-choice/multiple-response items; short answer and/or essay questions; drag and drop items; hot-spot items; and linear and branched simulations. Members of the PATF had a very preliminary discussion regarding implementation issues. For example, while these item types are all available as a result of the transition of ASPPB’s testing program to a computer-based delivery system, psychometric, stakeholder, and cost considerations must be factored into their potential use. Discussions touched upon some of the following related considerations:

• Psychometric—reliability and validity of each alternate item type and its defensibility within a high stakes testing program such as that sponsored by ASPPB

• Stakeholder—acceptance by candidates, regulators, credential holders; and the need to “introduce” candidates to each novel item type prior to its incorporation into the high-stakes examination, whether via practice examinations and/or exposure during training programs

• Cost—increased time and dollars to develop, review, and deliver items that may be more memorable than traditional multiple-choice items and, therefore, less long-lived

Subsequently, members of the PAAC described the recently published work of the Competency Assessment Workgroup, including a portfolio of assessment tools known as the Competency Assessment Toolkit for Professional Psychology (Toolkit). In 2007, the APA Board of Educational Affairs (BEA) authorized the workgroup to develop a resource for the assessment of the core foundational and functional competencies within professional psychology (described as the Cube Model by the Assessment of Competency Benchmarks Workgroup [2006]). The Toolkit was designed to provide a best practice resource for graduate, internship, and post-doctoral psychology programs wishing to systematically develop and assess the acquisition of competence in their students and trainees. Each of 15 different assessments were described through the use of fact sheets containing the following elements:

• A brief description • Application guidelines including logically targeted competencies and elements, • Implementation steps • Psychometric properties including comments on reliability, validity, and limitations, • Strengths • Challenges


• Future directions.

Members of the PATF briefly reviewed the assessments identified in the Toolkit and discussed what they believed were any obvious connections between the 15 assessments and either the competency clusters or the competency statements validated as potential targets for assessments in the current practice analysis study. Upon reflection, participants requested that they be given time to carefully review each of the assessments in order to make more informed determinations about the match between the 15 assessments and the competencies related to professional practice. PES described a grant program it was administering on behalf of the ASPPB Foundation. PES had issued a call for research proposals to support the identification, selection, and further refinement of an existing instrument or method that can be used to assess the pre-licensure competencies of psychologists; that is, either during training or internship. The grant application process was open to psychologists working within the professional education and training community. One requirement was that the competency assessment measure should allow for quantification in order to demonstrate sufficient reliability and validity to support it use and further study by other researcher in the future. It should also be of the type to permit its adaptation and application to other levels of professional development in psychology. It was the intention to facilitate the distribution of the assessment materials and research data developed under the grant free of charge to the profession under the auspices of organizations in the U.S. and Canada involved in the education and training of psychologists and/or in the assessment of competence. Under the terms of the grant, two research proposals were supported to stimulate the development of competency assessment initiatives in the profession. The first initiative focused on improving psychology students interview skills and diagnostic acumen through practice with a female adolescent virtual human (VHs) with Posttraumatic Stress Disorder (PTSD). This VH interaction will provide a context where immediate feedback can be provided regarding the student’s interviewing skills in terms of clinical knowledge, sensitivity, and effectiveness. Use of a natural language capable virtual character will be exceptionally beneficial in providing trainees with exposure to clinical diagnoses (in this case: Posttraumatic Stress Disorder, PTSD), prevalent in their live patient populations, and believed to be under-diagnosed due to difficulty in eliciting pertinent information. Virtual reality doctor and patient paradigms, therefore, will provide a unique and important format in which to teach and refine trainees’ interview skills and clinical knowledge. In order to be effective, virtual humans (VHs) must be able to interact in a 3D virtual world, must have the ability to react to dialogues with human-like emotions, and be able to converse in a realistic manner. The combination of these capabilities allows them to serve as unique training tools whose special knowledge and reactions can be continually fed back to trainees. While the initial goal is to focus on a virtual patient with PTSD, a similar strategy could be applied to teaching a broad variety of clinical diagnoses to trainees at every level pre-licensure/registration Parsons, 2009). The second initiative was designed to build upon previous substantial efforts within professional psychology to define competencies and takes the next step to develop and assess specific assessment tools to measure one specific competency delineated in the Benchmarks document


(building relationships) across practicum and internship training. The project also focuses on how supervisors can transform their evaluation of student competencies. Major goals included refinement of the Benchmarks to a more detailed, practical level, suitable for use in everyday training practice; documentation of the work of four internship and practicum sites in developing the conceptual, organizational, educational, and measurement/methodological systems that are required to implement the Competencies approach in practice; demonstrate the effects of these actions on training outcomes; and develop a sourcebook on implementation of competencies- based training for use nationally, that includes a full set of elaborated benchmarks for all of the competencies, and recommendations for implementation (Campbell, Fouad, Grus, Hatcher, McCutcheon, 2009). Finally, at PATF Meeting 3, participants discussed a number of additional data collection initiatives to be implemented in order to explore the assessment of competence. As a result, following the meeting, PES implemented an electronic survey of the PATF and conducted both focus panels and interviews with key stakeholder groups knowledgeable about competency assessment, student assessment, and/or regulatory mandates for competency assessment.

Results of PATF and PAAC Surveys The electronic survey of the members of the PATF and the PAAC required respondents to complete six tasks. In regard to the enhancement of the EPPP, respondents were asked to: • rate the utility of the alternate item types available via CBT for assessing competencies

within each of the six clusters, • identify strengths and weaknesses of each alternate item type, and • identify other novel item types that might be incorporated into the computer-delivered EPPP.

In regard to the development of new assessments to complement the EPPP, respondents were asked to: • rate the utility of a subset of the assessments described within the Toolkit for assessing

competencies associated with five of the six clusters, • identify strengths and weaknesses of each complementary assessment, and • identify other novel complementary assessments. In regard to the latter three tasks, the PATF determined that the Scientific Knowledge Based Competency Cluster was well assessed by the EPPP, and that future attention should be focused on assessing competencies associated with the remaining five clusters via the use of the complementary assessments. As seen in Table 63, an absolute majority of the respondents supported the assessment of the Scientific Knowledge base via multiple-choice, multiple-response items; scenario-based items; the use of supplementary stimulus materials; and point and click/hot-spot items. In regard to the alternate item types, respondents were most likely to support the use of scenario-based items regardless of the target competency cluster. However, an absolute majority of the respondents supported the use of simulations; multiple-choice, multiple-response items ; supplementary stimulus materials; and multimedia video for the assessment of competencies falling within at least two competency clusters other than the Scientific Knowledge base.


Table 63 Methods for Enhancing the EPPP by Competency Cluster

Competency Cluster

Scientific

Knowledge

Evidence-Based Decision

Making/Critical Reasoning

Interpersonal and Multicultural Competence

Professionalism/ Ethics Assessment

Intervention/ Supervision/ Consultation

Question Type n % n % n % n % n % n % Supplementary stimulus materials 11 73% 9 60% 5 33% 6 40% 10 67% 5 33%

Multiple choice–multiple response 13 87% 7 47% 5 33% 8 53% 10 67% 7 47%

Short answer 1 7% 7 47% 7 47% 8 53% 4 27% 7 47% Essay 0 0% 3 20% 5 33% 5 33% 3 20% 2 13% Drag and drop 7 47% 2 13% 2 13% 2 13% 4 27% 5 33% Point & click/hot spot 10 67% 2 13% 2 13% 3 20% 5 33% 2 13% Multimedia–static graphics 7 47% 6 40% 1 7% 0 0% 4 27% 1 7%

Multimedia–animation 2 13% 2 13% 1 7% 0 0% 0 0% 1 7%

Multimedia–audio 2 13% 1 7% 6 40% 5 33% 4 27% 4 27% Multimedia–video 3 20% 5 33% 8 53% 9 60% 6 40% 7 47% Scenarios 10 67% 14 93% 13 87% 15 100% 15 100% 15 100% Simulations 3 20% 6 40% 11 73% 11 73% 10 67% 11 73%


Respondents provided a number of qualitative comments regarding the alternate item types, but the general consensus was that dynamic interfaces (for example, graphical displays, interactive formulas) which present in-depth information may more readily permit the assessment of in-depth thinking than more static displays of information. Respondents were also mindful of the increased costs required for the development and/or scoring of the alternate item types. Table 64 documents the results of the survey of the PATF and PAAC in regard to their support for the application of various assessments described in the Toolkit to the assessment of competencies associated with the competency clusters other than Scientific Knowledge.

Table 64 Methods for Assessing Competency by Cluster (excluding Scientific Knowledge)

Competency Cluster

Evidence-Based Decision

Making/Critical Reasoning

Interpersonal and Multicultural Competence

Professionalism/ Ethics Assessment

Intervention/ Supervision/ Consultation

n % n % n % n % n %

360 evaluations 3 20% 10 67% 4 27% 1 7% 7 47% Case reviews 8 53% 4 27% 9 60% 9 60% 7 47% Conditional reasoning 7 47% 1 7% 10 67% 5 33% 5 33%

Consumer surveys 2 13% 4 27% 2 13% 0 0% 1 7% OSCE 8 53% 8 53% 7 47% 9 60% 8 53% Ratings forms 4 27% 4 27% 3 20% 3 20% 3 20% Simulation/role play 9 60% 13 87% 9 60% 7 47% 12 80% Situational judgment 10 67% 9 60% 10 67% 8 53% 10 67%

Standard oral exam 4 27% 2 13% 6 40% 3 20% 4 27% Standard patient 7 47% 10 67% 7 47% 10 67% 11 73% Self-assessment 2 13% 2 13% 2 13% 1 7% 2 13%

As can be seen, an absolute majority of the respondents supported the use of situational judgment tests for the assessment of competencies related to each cluster. An absolute majority of respondents also supported the use of objective, structured, clinical examinations (OSCE) and simulations/role play in regard to competencies associated with four of the five competency clusters. Finally, assessments such as 360 evaluations, case reviews, conditional reasoning, and standardized patients were supported by an absolute majority of the respondents in regard to competencies associated with three or fewer competency clusters. Respondents provided qualitative comments with regard to the development and implementation of such assessments as complements to the EPPP. Respondents were most likely to express concern about the economic feasibility of developing complementary assessments for use in the licensure/registration process. Several respondents noted that training programs might also be


encouraged to implement a standardized set of competency-based assessments so as to build a more rigorous gate to licensure/registration.

Results of Focus Panels and Interviews PES conducted a series of virtual focus panels to elicit information about the assessment of competencies from targeted stakeholders, namely:

• Educators and those involved in the training and supervision of psychology students, and others experienced in alternative and complementary methods of competency assessment in the US and Canada

• ASPPB key stakeholders, including jurisdictional registrars and those involved in the licensing or registration of psychologists

The purpose of this round of focus panels was to get input on how to assess the competencies in the model developed by the Task Force, and sought feedback from panelists on such issues as:

• Do you formally or informally assess any of the competencies listed? For ELC panel:

How do you assess your students?

• Are there other methods that you would like to be able to develop and implement in order to assess psychologists?

• Is the assessment of any of these competencies a valid role for ASPPB in terms of its public protection mission in regard to the licensure of psychologists?

• If ASPPB were to develop a competency-based approach to assessment, what methods might it use to validly and reliably assess these competencies?

• For ELC panel: Could you envision a formal role for your program in regard to the pre-licensure assessment of these competencies?

• Do you believe that the assessment of these competencies can be accomplished through the use of an enhanced EPPP? What would the enhancements look like?

• Do you believe that an alternate assessment might be more useful for the assessment of these competencies? What would that assessment look like?

• What impediments do you foresee in regard to the defensible assessment of these competencies?

Educators and those involved with the assessment of competency in psychology students, interns, and practica participants were invited to participate in a focus panel with the assistance of the Education Leadership Council (ELC) and the Canadian Council on Professional Psychology Programs (CCPPP). E-mail invitations were sent to attendees at the annual ELC conference asking them to participate in a virtual focus panel to explore how their educational programs assess competency and how competency assessment might be incorporated into training as part of the licensing/registration process. A similar invitation was sent to members of the CCPPP. Nine ELC members agreed to participate in a virtual focus panel, and one CCPPP member agreed to participate in an interview.


To assess how key ASPPB stakeholders viewed incorporating competency assessment into the licensing/registration process, e-mails were sent to ASPPB jurisdictional directors who were attending the 2009 annual meeting of the ASPPB. Eleven agreed to participate in a focus panel in conjunction with that conference. For all panels and interviews, participants were provided stimulus materials consisting of a representation of the competency model. They were asked questions regarding how best to assess such competencies during the course of the panels/interview. (A summary report of the focus panel and interview may be found in Appendix 9). When asked what types of competency assessment they currently use, ELC members generally felt that the existing options were poor or limited. Competency assessment is very resource intensive and expensive, and many rely on informal evaluations by supervisors who have a vested interest in “passing along” their students. They cited some more formal methods used in their programs, including 360o evaluation by colleagues and multidisciplinary panels, use of scenarios and vignettes, OSCEs and role playing, written evaluations by training directors and internship directors, and formal chart, journal, or case reviews, some of which used behaviorally-based anchors as benchmarks. It was suggested that direct clinical skills might best be assessed using OCSE or role playing, while assessment skills might best be measured using standard patients or 360o evaluation, or by formalized chart review. There were a number of problems with all of these methods, including lack of uniform rating scales, issues of reliability and validity, low level of behaviors being assessed, possible bias on the part of evaluators, and questions about the predictive value of the assessments. All participants expressed interest in using a uniform rating scale that operationalized the competencies being assessed, and would like to see “standard patient” scenarios made available. There was general agreement that competency assessment during the training and internship period was desirable, but what is needed is specific standards for such assessments. Participants expressed interest in using summative evaluations involving standard patient scenarios or 360o evaluations if ASPPB were to develop them, along with standardized rating scales and evaluator training materials. They also expressed an interest in taking any summative evaluations developed by ASPPB in the future and modifying them to create formative evaluations for use during the course of training, internship, and practica. Participants expressed concern for the potential for the liability of program directors if the students they train cannot pass the assessments. The ELC participants felt that increased collaboration between ASPPB, the APA, and training programs in the area of competency assessment would benefit the profession. They suggested that APA make competency assessment of students part of the criteria in the approval process of educational and training programs. In general, the concept of integrating some kind of standardized competency assessment into the licensing procedure was well received, as long as someone provided the assessment materials. Participants did have reservations due to the costs of implementing such assessment. Jurisdictional administrators involved in the licensing/registering of psychologists, while approving of the idea of competency assessment in general, argued that such assessments are


extremely expensive to administer and that jurisdictions did not have the resources to implement these. In addition, doing this type of assessment at the point of licensure/registration is difficult, since students should have been demonstrating competency throughout training in order to get to the point of applying for licensure/registration. There was some concern for liability, although it was felt that the boards were shielded due to their public protection mandate. Administrators felt that training programs and jurisdictional boards need to be on the same page regarding competency assessment, and preferably, that such assessment be integrated as part of the training process. Suggestions for the types of competency assessment that could be used at the point of licensure/registration included oral exams or structured interviews, which some jurisdictions currently utilize. These include standard protocols followed by questions, and have been used to rate if candidates are “ready for practice.” Like other methods, however, these rely on subjective ratings by the examiners. Ideally, administrators feel that competency assessment should be done during the course of education. The medical school model, which involves periodic and regular competency assessment as part of the ongoing education process, was cited as a model to emulate. Like their ELC counterparts, these panelists also felt there was a great need for uniform guidelines that programs could use for assessing competence. The concern was raised that training directors and internship supervisors might be biased toward passing through their students regardless of demonstrated competency (or lack thereof) because they have a vested interest in the success of their programs. If competency assessment becomes part of the licensing/registration process, it will be necessary to train the assessors, use standardized or formalized methods, and ensure that the assessments are valid and reliable. ASPPB jurisdictional administrators felt that ASPPB, in collaboration with the APA, should take a pivotal leadership role in making competency assessment part of the licensing/registration process. Standardized assessment could include the use of decision trees and the identification of critical components. Panelists wanted to ensure that ASPPB’s work and the APA’s Benchmarks would be integrated, and suggested that the ASPPB could help guide the APA in developing the behavioral anchors to be used for competency assessment. ASPPB could take the lead in developing standard patient scenarios, rating scales, and other types of questions and assessments for training programs and internship directors to administer, so that programs could ensure that the assessments were transparent and fair. The results of the assessments administered during education and training would be made available to jurisdictions as part of the licensing/registration application. Panelists also felt that the APA should require the administration of such assessments as a condition of accreditation. In general, the consensus was that ,while difficult and expensive to administer, the time was right to incorporate competency assessment into the education and training process as part of the licensure/registration process.


Conclusions

The study of the practice of licensed psychologists in the United States and Canada yielded the following results:

• Professional background and demographic descriptions of recently and less-recently licensed

psychologists in the U.S. and Canada were documented. • Content areas and the knowledge required for professional practice were identified and

validated via a large-scale survey of practice. • Recommendations regarding revised test specifications for the EPPP were developed on the

basis of the survey data related to eight content areas and the knowledge associated with each area. Validated content-based test specifications were developed, focusing primarily on the set of knowledge needed by licensed psychologists at entry to practice, that serves the public protection function of regulation.

• Current and emerging trends in the practice of the profession and the knowledge base that

will be increasingly useful to licensed psychologists in the future were identified as a base for reviewing college and university curricula, professional schools, training programs, continuing education efforts, and specialty recognition initiatives that may be undertaken in professional psychology.

• A unique framework for a competency-based model of professional psychology that might inform decision making at the point of licensure/registration of psychologists was developed. The framework drew upon the work of the 2002 Competencies Conference which identified core foundational and functional competencies within the profession of psychology using a cube model, and further built on the refinement of the cube model by the BEA Task Force on Assessment of Competence in Professional Psychology in 2006.

• The competency-based model includes six competency clusters: Scientific Knowledge,

Evidence-Based Decision Making/Critical Reasoning, Interpersonal and Multicultural Competence, Professionalism/Ethics, Assessment, and Intervention/Supervision/Consultation. The six clusters were validated via the large-scale survey of practice.

• While the first cluster, Scientific Knowledge, is organized by eight content areas and their associated knowledge base, the remaining five clusters are organized by competencies and behavioral exemplars that operationalized the level of behavior expected at four key points: during entry-level supervised practice (practicum), level 1; during advanced-level supervised practice (internship), level 2; at entry to practice through the first 3 years of independent practice, level 3; and after 3 years of independent practice, level 4. The large-scale survey of practice served to validate the competencies and a majority of the behavioral exemplars as being essential to practice at some point on the professional development continuum.

• Methodologies for assessing competence were evaluated.


• The feasibility of enhancing the EPPP with alternate item types was explored as one way of

augmenting the current licensure/registration process.

• The potential for developing new assessments to complement the EPPP was preliminarily discussed as was the use of such assessments at various points of time pre- and post-licensure/registration.

• Discussions among the members of the PATF and the PAAC confirmed the utility of the

EPPP as an effective tool for the assessment of the Scientific Knowledge base underlying the practice of psychology. Future directions in regard to alternate item types that might be built into this computer delivered examination were discussed.

• Discussions among the members of the PATF and the PAAC as well as key stakeholders

involved in the assessment of competency in students, interns, and practica participants, and jurisdictional regulation confirmed interest in the assessment of competency via complementary assessments that might be integrated into pre- and post-licensure/registration activities and/or licensure/registration requirements.

• A substantial investment in time and resources will be required should ASPPB choose to

support the development of entry-level competency assessments for licensed/registered psychologists. The widespread dissemination of this technical report, including information about the development of validated competencies to be acquired pre- and post licensure/registration, should be a priority and might easily be accomplished via the ASPPB’s website.

• ASPPB needs to consider the leadership role it might play in the future development and

support of competency assessment. To that end, a meeting of key stakeholders including other organizations within the profession, may be desirable in order to sort through the issues and develop a game plan for moving the competency assessment agenda forward across the professional development continuum. Among numerous resources available for the discussion are the following:

o The results of the two ASPPB Foundation grant-supported assessment initiatives focusing on the development of competency assessment pre-licensure, as well as numerous other research-based assessment development initiatives occurring within the professional education and training community;

o The recently published work of the Competency Assessment Workgroup, including the portfolio of assessment tools known as the Competency Assessment Toolkit for Professional Psychology (Toolkit); and, now,

o The results of the ASPPB-sponsored practice analysis study in the U.S. and Canada, including not only the validation of the Scientific Knowledge competency cluster, but the validation of five other competency clusters, each including behavioral exemplars representing the evolution of the validated competencies which develop across key milestones.


References and Background Literature American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

American Psychological Association (2006). Report of the Assessment of Competency Benchmarks Work Group. Washington D.C.: Author.

American Psychological Association (2007) Model act for state licensure of psychologists, revised 7/03/2007, Washington, DC: Author.

American Psychological Association Center for Workforce Studies (2008) Early Career Psychologist Survey 2007, Washington, DC: Author.

Bieschke, K., Bell, D., Davies III, C., Hatcher, R., Peterson, R. & Rodolfa, E., (Editors) (2009). Editorial, Training and Education in Professional Psychology, 3(Suppl.) S1-S4.

Campbell, L., Fouad, N., Grus, C., Hatcher, R. & McCutcheon, S. (2009) Training Competent Psychologists in Building Effective Relationships. Grant proposal submitted to the ASPPB Foundation.

Epstein, R. M., & Hundert, E. M. (2002). Defining and assessing professional competence. Journal of the American Medical Association, 287, 226-235.

Fouad, N., Hatcher, R., Hutchings, P.S., Collins, F., Grus, C., Kaslow, N., Madson, M. & Crossman, R. (2009). Competency benchmarks: A model for understanding and measuring competence in professional psychology across training levels. Training and Education in Professional Psychology, 3(Suppl.) S5-S26.

Greenberg, S. & Jesuitus, L. (2003) Study of the practice of licensed psychologists in the United States and Canada, New York, NY: Professional Examination Service.

Greenberg, S., Muenzen, P. & Smith, I.L. (2007). The study of practice: Validating synergy. In M. Curley, (Ed.). Synergy: The unique relationship between nurses and patients (pp. 183-209). Indianapolis, IN: Sigma Theta Tau International.

Greenberg, S. & Smith, I.L. (2008) Methods to evaluate competency and enhance quality assurance internationally and across professions. In Hall, J. and Altmaier, E., (Eds.). Global Promise: Quality assurance and accountability in professional psychology (pp. 51-72). New York, NY: Oxford University Press.

Greenberg, S., Smith, I.L. & Muenzen, P. (1996). Study of the practice of psychology in the United States and Canada. New York, NY: Professional Examination Service.

Harrison, D., McPeak, D. & Greenberg, S., (2004). Validating competencies underlying a professional accounting credential. Paper presented at the ABER 3rd Global Conference on Business and Economics, Amsterdam, Netherlands.


Hatcher, R. L. & Lassiter, K. D. (2007). Initial training in professional psychology: The practicum competencies outline. Training and Education in Professional Psychology, 1, 49-63.

Kasch, M., Greenberg, S., & Muenzen, P. (2003) Competencies in hand therapy, Journal of Hand Therapy, 16, 49-58.

Kaslow, N. J. (2004). Competencies in professional psychology. American Psychologist, 59, 774-781.

Kaslow, N. J., Rubin, N.J., Bebeau, M.J., Leigh, I.W., Lichtenberg, J.W., Nelson, P.D., Portnoy, S.M. & Smith, I.L. (2007). Guiding principles and recommendations for the assessment of competence. Professional Psychology: Research and Practice, 38, 441–451.

Kaslow, N., Forrest, L., Van Horne, B., Huprich, S., Pantesco, V., Grus. C., Shen Miller, D., Mintz, L., Schwartz-Mette, R., Rubin, N., Elman. N., Jacobs, S., Benton, S., Dollinger, S., Behnke, S., Shealy, C., Van Sickle, K. & Thorn, B (2007). Recognizing, assessing, and intervening with problems of professional competence. Professional Psychology: Research and Practice, 5, 479-492.

Kaslow, N.J., Borden, K.A., Collins, F.L., Jr., Forrest, L., Illfelder-Kaye, J., Nelson, P. & Rallo, J.S. (2004). Competencies conference: Future directions in education and credentialing in professional psychology. Journal of Clinical Psychology, 60, 699-712.

Kruger, J. & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own self incompetence lead to inflated self-assessment. Journal of Personality and Social Psychology, 77, 1121-1134.

Leigh, I.W., Smith, I.L., Bebeau, M.J., Lichtenberg, J.W., Nelson, P.D., Portnoy, S.M., Rubin, N.J. & Kaslow, N.J. (2007). Competency assessment models. Professional Psychology: Research and Practice, 38, 463–473

Lichtenberg, J.W., Portnoy, S.M., Bebeau, M.J., Leigh, I.W., Nelson, P.D., Rubin, N.J., Smith, I.L. & Kaslow, N.J. (2007). Challenges to the assessment of competence and competencies, Professional Psychology: Research and Practice, 38, 474–478.

Parsons, T., (2009) Virtual Patients for Assessing the Pre-licensure Competencies of Psychologists. Grant proposal submitted to ASPPB Foundation. Roberts, M., Borden, K, & Christiansen, M. & Lopez, S. (2005). Fostering a culture shift: Assessment of competence in the education and careers of professional psychologists. Professional Psychology: Research and Practice, 36, 355-361.

Rodolfa, E. R., Bent, R. J., Eisman, E., Nelson, P. D., Rehm, L., & Ritchie, P. (2005). A cube model for competency development: Implications for psychology educators and regulators. Professional Psychology: Research and Practice, 36, 347-354.


Roth, J., Somma, R., Greenberg, S. & Demos, A. (2006). Developing and implementing a certification program to drive change in the pharmaceutical industry. Pharmaceutical Engineering, May-June 2006 (Suppl), 1-112.

Rubin, N.J., Bebeau, M.J., Leigh, I.W., Lichtenberg, J.W., Nelson, P.D., Portnoy, S.M., Smith, I.L. & Kaslow, N. (2007). The competency movement within psychology: An historical perspective. Professional Psychology: Research and Practice, 38, 452–462.

Schoon, C.G. (1985). Methods for defining and assessing professional competence. Professional Practice of Psychology, 6(1), 144 – 155.

Tamblyn, R, Abrahamowicz, M., Brailovsky, C., Grand’Maison, P., Lescop, J., Norcini, J., Girard, N. & Haggerty, J. (1998). Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA, 280, 989-996.

Tamblyn, R., Abrahamowicz, M., Dauphinee, D., Wenhofer, E., Jacques, A., Klass, D., Smee, S., Lackmore, D., Winslade, N., Girard, N., Du Berger, R., Bartmen, I., Buckeridge, D. & Hanley, J. (2007). Physician Scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA, 298, 993-1001.


Appendix 1 Practice Analysis Advisory Committee and their Professional Affiliations

and ASPPB Personnel Monitoring Study

Appendix 2 Members of the Practice Analysis Task Force and their Professional Affiliations

Appendix 3

Final Report of Independent Review, Round 1

Appendix 4 Final Report of Focus Panels, Round 1

Appendix 5

Final Report of Independent Review, Round 2

Appendix 6 Materials Related to the Conduct of the Pilot Survey

Appendix 7 Screen Shots of Large-Scale Validation Survey

Appendix 8

Materials Related to the Conduct of the Large-Scale Validation Survey

Appendix 9 Final Report of Focus Panels, Round 2

Appendix 10

Areas of Formal Post-doctoral Level Training

Appendix 11 Knowledge Acquisition, Criticality, and Usage ratings of U.S. and Canadian Respondents


Appendix 12 Knowledge Acquisition, Criticality, and Usage ratings of Recently and Less-recently

Licensed/Registered Respondents

Appendix 13 Verbatim Responses to Open-ended Question

Appendix 14

Competency Frequency, Criticality, and Importance Ratings of U.S. and Canadian Respondents

Appendix 15 Competency Frequency, Criticality, and Importance Ratings of Recently and Less-recently

Licensed/Registered Respondents

Documents

S PRACTICE OF LICENSED PSYCHOLOGISTS I …c.ymcdn.com/sites/ Final Report An Update Study of the Practice of Licensed Psychologists iv Table of Appendices Appendix 1 Practice Analysis