Standard Setting Report - Recertification · Standard Setting Report – Certified Personal Trainer – August 2019 2 Introduction This report describes the standard setting study

Standard Setting Report PTA Global Certified Personal Trainer

August 2019

Submitted to:

to:

Where people meet potential

Standard Setting Report – Certified Personal Trainer – August 2019 1

Contents Introduction 2

Methodology 3

Definition of Minimum Competence 3

Rating of Examination Items 3

Consideration of Empirical Data 4

Results 5

Summary of Ratings 5

Established Passing Standard 7

Appendix A 8

Panel of Standard Setting Judges 8

Appendix B 9

Standard Setting Presentation 9

Appendix C 14

Definition of Minimum Competence 14

Appendix D 15

Form 3 15

Individual Round 1 Ratings 15


Form 4 2




Introduction

This report describes the standard setting study conducted for the PTA Global (PTAG) Certified Personal Trainer (CPT) Certification Examination. The study was conducted using methods that objectively facilitated the criterion-referenced nature of credentialing decisions, based on a method described by Angoff (1971)1. The study results were used to guide selection of a passing standard.

As with any professional practice standard setting process, some type of judgment is required. However, it is essential that the judgments involved in determining the passing point be made by qualified experts who are well informed regarding the intended use of the examination and possess the requisite knowledge and experience in the content domain to know what level of competence should be reasonably expected. Additionally, the judgments should be rendered in a meaningful way that accounts for the format and purpose of the test.

This determination must be made with recognition of the effects of potential error on classification decisions and the negative consequences of possible misclassification for examinees and the public. This report documents the appropriateness of the established passing point (cut score) used to determine examinee decision outcomes.

The underlying philosophy of the Angoff procedure is that the standard set relates to expectations of performance for those who are minimally competent (i.e., those with the requisite capabilities commensurate with the eligibility requirements). Therefore, this procedure requires judges to render an expected performance rating for each test item that reflects their expectation of performance for those who are minimally competent.

1 Angoff, W.H. (1971). Scales, Norms, and Equivalent Scores. In R.L. Thorndike (ed.), Educational Measurement. Washington, D.C., American Council on Education, 1971.


Methodology

The judges serving on the standard setting study panel were selected by the PTAG, all of whom are subject matter experts. They were selected to provide for an appropriate balance on potentially relevant professional characteristics, such as area of special expertise and geographical distribution. See Appendix A for a listing of judges and their qualifications

During web meetings on August 5, 2019 and August 7, 2019, the judges participated in a standard setting study that consists of the following three major steps:

1. Defintion of Minumum Competence 2. Rating of Examination Items 3. Consideration of Empirical Data

Judges engaged in an introductory presentation on the process that describes the standard setting activity and explains their role in the process. See Appendix B to see the presentation used.

Definition of Minimum Competence In preparation of the rating process, a discussion regarding the definition of a minimally competent practitioner (MCP) was facilitated. An MCP is described as an individual who has enough knowledge to practice safely and competently but does not demonstrate the knowledge level to be considered an expert.

The judges discussed specific behaviors, or performance-level descriptors (PLDs), that someone with the requisite education and training would do to demonstrate his or her competence. Judges were instructed on the following four principles to help guide their identification of PLDs: 1) not something that they know, 2) not something that they don’t do, 3) not what an expert would do, and 4) specific actions performed that demonstrate competence. The judges were asked to identify at least three or four PLDs per content domain to ensure an adequate representation of the role. Judges suggested ideas for PLDs and the panel deliberated and refined the wording until there was consensus. The PLDs developed are listed in Appendix C.

Rating of Examination Items Judges were then trained on the rating process. Central to this process is the notion that each rating is provided inidividually by each rater and reflect the answer to this question: What percentage of MCPs do you expect will answer this item correctly?

The judges were shown each item one at a time and instructed to provide a rating for each item (round 1 rating) after reading the stem and the response options. Judges then recorded their ratings in a spreadsheet.


Consideration of Empirical Data Following the initial rating, the answer key and p value (if available) were presented so that the panelists could re-evaluate their thought process and revise their ratings, as necessary (round 2 rating). For example, the judges were specifically advised to consider the possibility that their ratings might be too high on items that they answered incorrectly when the initial ratings were recorded or if their expectations of performance for MCPs were significantly different from the p value, which represents the performance of a sample of examinees. Judges were provided access to the items to provide independent ratings on two test forms (PTAPT03A and PTAPT04A) during August 5, 2019 to August 13, 2019.


Results Summary of Ratings The tables below present the results of the analysis of the judges’ ratings from Round 2. The judges' individual cut score estimates (round 3 ratings) from 67.60 to 71.45 for Form 3 and 67.40 to 72.15 for Form 4. The mean of the judges' estimates yields a cut point of 69.93 and 70.25 for Form 3 and Form 4, respectively. These cut points represent a raw score passing point of 70 out of 100 items for both forms. The standard deviation of the ratings represents the expected amount of variability in ratings if the judges were to repeat the process and can be interpreted as a confidence interval around the judges’ final estimate. The individual judges’ ratings can be found in Appendix D.

Summary of Judges’ Ratings for Form 3 (PTAPT03A)

JUDGE # MEAN

RATINGS SD OF

RATINGS

1 67.60 11.63

2 71.45 11.73

3 70.10 11.18

4 70.56 10.69

Summary of Judges’ Ratings for Form 4 (PTAPT04A)

JUDGE # MEAN

RATINGS SD OF

RATINGS

1 67.40 12.52

2 72.15 10.68

3 70.70 13.06

4 70.75 12.13


Summary of Passing Point Options

Exam Form Form 3 Form 4

Average of R3 Angoff Ratings 69.93 70.25

Exam Length (Scored Items Only) 100 100

Number of Raters 4 4

Standard Error of R3 Angoff Ratings

5.70 6.13

Plus Three SE of R3 Angoff Ratings 87.03 88.63

Plus Two SE of R3 Angoff Ratings 81.33 82.51

Plus One SE of R3 Angoff Ratings 75.63 76.38

Panel-Recommended Raw Cut 69.93 70.25

Minus One SE of R3 Angoff Ratings 64.23 64.12

Minus Two SE of R3 Angoff Ratings 58.52 57.99

Minus Three SE of R3 Angoff Ratings 52.82 51.87


Established Passing Standard The results of the study were presented to the PTAG on August 15, 2019. The PTAG was advised to establish a cut score within the range of one standard error below or above the panel-estimated cut score (64 to 76). Following discussion, the PTAG established 70 out of 100 as the raw score passing point for the CPT examination.

The two forms used for standard setting will be used as the base forms for the creation of subsequent forms of the CPT examination, to be made parallel and equivalent through the use of statistical pre-equating and adherence to the content allocation requirements of the exam content outline.


Appendix A Panel of Standard Setting Judges

NAME JOB TITLE EMPLOYER LOCATION YEARS EXP.

Assistant Wellness Director

California 10

Personal Trainer

London, England

3

Director of Education

California 16

Fitness Director

California 30


Appendix B Standard Setting Presentation






Appendix C Definition of Minimum Competence

What does a Minimally Competent Practitioner do to demonstrate competence?

I. Program DesignA. Deliver a coherent client needs analysis assessment during initial and ongoing client

consultations, without supervisionB. Conduct a physical activity readiness questionnaire and client screening (e.g., health

history, resting heart rate, blood pressure, cardiorespiratory, muscular endurance,strength, and flexibility)

C. Develop safe and effective personalized client programming strategies using theprinciples and acute variables of exercise for different goals and ability levels

D. Safely Apply various forms of exercise and equipment, including modificationstrategies, for different clients’ goals and abilities

II. Human BehaviorA. Use communication and motivational interviewing techniques effectively (e.g., open-

ended questions, active listening, empathy)B. Deliver practical strategies for facilitating client behavior change

III. Exercise SciencesA. Effectively Measure and monitor exercise intensity including heart rate and perceived

level of exertion to ensure safety for different populationsB. Instruct client on proper breathing techniques while performing cardiovascular,

strength, and flexibility trainingC. Recognize signs and symptoms of various health conditions (e.g., stroke, heart

attack, low blood pressure, insulin shock, diabetic coma)IV. Workplace Practices

A. Detail strategies to gain clientele (e.g., marketing, prospecting, customer serviceskills)

B. Detail strategies for referring clients to other professionals (e.g., MDs, dietitians,physical therapists, massage therapists)

V. NutritionA. Empathize with and accommodating for various eating strategies for different types of

groups (e.g., vegetarians, diabetics, religious practices)B. Recognize and adhere to the scope of practice when making dietary

recommendations (e.g., referral, appropriate resources)C. Recognize symptoms of low blood-sugar and dehydration, and how to address them

VI. Special PopulationsA. Recognize special conditions based on medical and health history and obtain

physician release forms as necessary (e.g., elderly, youth, pregnancy, overweight orobese, chronic injuries and disease)

B. Recognize and adhere to boundaries of scope of practice with regard to medical andhealth concerns

C. Modify exercises to deliver safe and effective training programs to the specialpopulations


Appendix D Form 3

Individual Round 1 Ratings

ITEM ID JUDGE 1 JUDGE 2 JUDGE 3 JUDGE 4

1 80 80 75 75

2 70 75 50 60

3 70 85 55 80

4 75 85 60 75

5 80 90 90 80

6 65 75 40 80

7 65 65 80 75

8 80 85 90 90

9 70 85 75 85

10 80 75 65 40

11 80 90 80 75

12 85 80 70 80

13 75 70 75 75

14 50 70 80 60

15 85 75 60 90

16 60 65 60 60

17 80 90 80 85

18 70 75 75 65

19 60 75 60 60

20 75 75 50 70

21 75 70 70 70

22 65 65 70 75

23 70 80 60 75

24 85 75 75 70

25 55 80 65 55

26 70 65 75 75

27 75 70 60 70

28 60 60 65 70

29 50 60 60 75

30 70 75 75 75

31 70 75 65 65

32 80 75 50 80

33 55 65 65 70

34 75 70 80 75

35 85 80 70 80

36 55 70 50 45

37 75 70 85 65

38 65 80 75 70

39 70 75 90 90

40 70 70 75 70

41 50 60 50 50

42 65 70 60 60

43 80 85 60 75

44 50 65 80 65

45 85 85 60 85

46 70 75 65 80

47 60 65 60 55

48 40 85 60 50

49 60 45 50 70

50 80 50 80 60


51 75 65 50 75

52 80 70 75 60

53 55 45 65 70

54 50 65 65 65

55 60 50 70 50

56 65 70 80 70

57 70 70 75 65

58 75 70 70 70

59 55 70 50 60

60 75 75 70 50

61 50 55 65 50

62 80 75 75 75

63 65 75 70 60

64 60 70 45 70

65 50 65 75 55

66 65 70 80 70

67 80 70 80 70

68 50 70 50 55

69 70 60 50 50

70 70 65 70 55

71 70 80 80 80

72 50 50 50 70

73 75 70 85 70

74 75 75 75 70

75 75 70 70 70

76 60 65 75 70

77 70 70 70 70

78 70 80 90 70

79 80 75 90 85

80 70 80 60 65

81 80 85 90 80

82 45 70 50 60

83 50 45 50 60

84 80 65 85 80

85 80 70 80 90

86 80 80 50 60

87 65 65 65 70

89 70 80 65 70

90 85 85 90 90

91 70 75 80 50

92 80 90 65 75

93 75 75 80 80

94 60 85 90 75

95 70 85 70 90

96 60 50 75 70

97 70 65 75 70

98 40 80 65 75

99 60 70 65 70

100 55 75 75 55


Individual Round 2 Ratings ITEM ID JUDGE 1 JUDGE 2 JUDGE 3 JUDGE 4

1 80 80 75 75

2 70 75 55 60

3 70 85 75 80

4 75 85 70 75

5 80 90 90 85

6 65 75 60 75

7 65 65 80 75

8 85 85 90 90

9 75 85 80 85

10 80 75 65 40

11 80 90 80 75

12 85 80 70 80

13 75 70 75 75

14 50 70 80 60

15 85 75 60 90

16 60 65 60 60

17 80 90 80 85

18 65 75 75 65

19 60 75 60 60

20 75 75 50 70

21 75 70 70 70

22 65 65 70 75

23 70 0 60 75

24 85 75 75 70

25 55 80 65 55

26 70 65 75 75

27 75 70 60 75

28 60 60 65 70

29 50 60 60 75

30 70 75 75 75

31 70 75 65 65

32 80 75 50 80

33 55 65 65 70

34 75 75 80 80

35 75 80 75 80

36 45 60 50 45

37 75 70 85 65

38 70 75 75 70

39 80 85 85 86

40 70 65 75 70

41 40 50 45 50

42 65 70 60 60

43 80 85 60 75

44 50 65 80 65

45 85 85 60 85

46 70 75 75 80

47 60 60 50 55

48 40 75 55 50

49 60 55 70 70

50 70 55 75 65

51 75 70 70 75

52 80 75 75 75

53 55 55 75 70

54 55 65 60 60

55 65 60 75 60

56 65 70 75 75

57 70 70 70 70


58 75 75 80 80

59 55 70 60 65

60 75 75 70 60

61 50 55 60 55

62 75 75 75 75

63 65 75 70 70

64 65 70 70 75

65 50 65 60 55

66 55 65 80 70

67 80 75 85 85

68 50 60 50 55

69 60 60 60 55

70 65 65 70 55

71 60 70 60 75

72 50 50 60 70

73 65 70 70 70

74 75 75 75 70

75 80 70 70 70

76 65 70 75 75

77 65 70 65 70

78 70 75 75 70

79 80 85 90 90

80 70 80 60 60

81 80 85 85 80

82 45 65 50 60

83 50 45 50 55

84 80 75 85 85

85 80 80 90 90

86 80 80 40 50

87 70 70 70 70

89 70 80 75 70

90 85 85 90 90

91 70 75 75 60

92 80 85 70 75

93 75 80 85 80

94 60 80 90 75

95 70 85 75 85

96 65 60 85 80

97 70 70 75 75

98 30 70 65 75

99 60 70 65 70

100 55 75 75 55


Form 4


1 70 80 60 70

2 70 75 75 75

3 70 85 55 80

4 75 85 60 75

5 80 90 90 80

6 65 75 40 80

7 80 85 80 80

8 80 85 90 90

9 70 85 75 85

10 80 75 65 40

11 80 90 80 75

12 85 80 70 80

13 75 70 75 75

14 50 70 80 60

15 85 75 60 90

16 60 65 60 60

17 80 90 80 85

18 70 75 75 65

19 60 75 60 60

20 75 75 50 70

21 75 70 70 70

22 65 65 70 75

23 85 75 75 70

24 55 80 65 55

25 70 65 75 75

26 70 75 80 80

27 65 65 80 75

28 75 70 60 70

29 60 60 65 70

30 50 60 60 75

31 70 75 65 65

32 80 75 50 80

33 75 70 85 65

34 45 35 60 55

35 85 80 90 85

36 50 45 50 60

37 70 70 75 70

38 70 65 70 75

39 65 60 80 70

40 50 70 80 60

41 50 60 50 50

42 65 70 60 60

43 80 85 60 75

44 50 65 80 65

45 85 85 60 85

46 40 50 40 70

47 70 80 70 65

48 60 70 75 80

49 60 60 75 50

50 80 65 85 65

51 50 50 55 40

52 60 75 80 65

53 70 70 70 65


54 60 75 50 70

55 70 80 75 80

56 80 65 90 80

57 40 65 50 35

58 45 65 65 70

59 55 80 65 75

60 65 60 60 55

61 45 65 60 50

62 60 60 80 65

63 80 75 75 85

64 80 90 80 80

65 65 75 70 60

66 60 70 45 70

67 40 65 50 50

68 65 70 80 70

69 65 70 80 70

70 70 65 70 55

71 70 50 85 70

72 40 65 60 45

73 70 70 80 85

74 80 70 90 80

75 75 70 70 70

76 80 90 95 90

77 70 65 90 70

78 90 85 95 75

79 80 75 90 85

80 70 80 60 65

81 50 50 50 60

82 50 75 85 90

83 85 90 95 90

84 70 75 75 70

85 70 75 75 70

86 80 75 75 75

87 80 85 80 90

89 65 60 50 60

90 90 90 90 85

91 70 75 80 50

92 80 75 85 65

93 50 65 50 60

94 70 70 65 50

95 50 70 60 45

96 60 85 90 75

97 70 85 70 90

98 70 65 65 75

99 60 70 65 70

100 55 75 75 55



1 75 80 65 75

2 70 75 75 75

3 70 85 75 80

4 75 85 70 75

5 80 90 90 85

6 65 75 60 75

7 80 85 85 85

8 85 85 90 90

9 75 85 80 85

10 80 75 65 40

11 80 90 80 75

12 85 80 70 80

13 75 70 75 75

14 50 70 80 60

15 85 75 60 90

16 60 65 60 60

17 80 90 80 85

18 65 75 75 65

19 60 75 60 60

20 75 75 50 70

21 75 70 70 70

22 65 65 70 75

23 85 75 75 70

24 55 80 65 55

25 70 65 75 75

26 70 75 80 80

27 65 65 80 75

28 75 70 60 75

29 60 60 65 70

30 50 60 60 75

31 70 75 65 65

32 80 75 50 80

33 75 70 85 65

34 45 40 50 55

35 85 85 90 85

36 50 50 60 60

37 70 65 75 70

38 70 70 75 75

39 65 65 75 70

40 50 60 70 60

41 40 50 45 50

42 65 70 60 60

43 80 85 60 75

44 50 65 80 65

45 85 85 60 85

46 50 55 40 70

47 70 80 70 70

48 70 70 80 80

49 60 70 80 70

50 75 65 80 70

51 50 50 50 40

52 60 75 75 70

53 65 70 65 65

54 60 75 60 70

55 75 80 85 80

56 80 75 85 80

57 40 55 45 40


58 45 65 60 70

59 60 75 65 70

60 65 60 60 60

61 45 55 50 50

62 60 60 70 70

63 80 80 80 85

64 80 90 80 80

65 65 75 70 70

66 65 70 70 75

67 40 55 40 50

68 65 70 75 75

69 55 65 80 70

70 65 65 70 55

71 75 60 85 75

72 40 55 45 40

73 75 70 80 85

74 80 75 85 85

75 80 70 70 70

76 80 90 85 85

77 70 75 80 75

78 85 85 90 80

79 80 85 90 90

80 70 80 60 60

81 50 50 50 55

82 50 70 90 90

83 85 90 90 90

84 70 75 70 70

85 70 75 75 70

86 75 75 75 75

87 85 85 90 90

89 65 65 50 70

90 90 90 90 90

91 70 75 75 60

92 85 85 90 70

93 50 60 50 60

94 60 65 60 50

95 50 65 50 45

96 60 80 90 75

97 70 85 75 85

98 70 70 75 75

99 60 70 65 70

100 55 75 75 55

PSI Services

psionline.com

@PSIServicesLLC

PSI Services LLC

© 2019 PTAG. All rights reserved.

No portion of this publication may be translated or reproduced in whole or in part, stored in a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise) without the prior written permission of the copyright owner. This publication may not be resold, rented, lent, leased, exchanged, given or otherwise disposed of to third parties. Neither the purchaser nor any individual test user employed by or otherwise contracted to the purchaser may act as agent, distribution channel or supplier for this publication.

PSI and the PSI logo are registered trademarks of PSI Services LLC.

Documents

Standard Setting Report - Recertification · Standard Setting Report – Certified Personal Trainer – August 2019 2 Introduction This report describes the standard setting study