View
92
Download
7
Category
Preview:
DESCRIPTION
42nd Apmc Convention Nmat Ppt
Citation preview
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
1
National Medical Admission Test (NMAT)
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
2
NMAT
• Objective: to upgrade the selection of applicants to medical schools
• Description: an instrument that measures several abilities deemed necessary for success in medical training and practice
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
3
NMAT
• Abilities measured:– verbal comprehension– inductive reasoning– quantitative ability– perceptual acuity– knowledge and analytical proficiency in the sciences
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
4
Test Scores
• 8 subtest scores
• APT Composite
• SA Composite
• General Performance Score (GPS)
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
5
Test Development Concerns
1. Relevance of the areas measured by the test to the curricula of premedical courses and the medical course itself
2. Comparability of test scores from different test forms across the years
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
6
Test Development Procedure
1. Content specification
2. Item writing and review
3. Pretesting and item analysis
4. Test assembly
5. Test administration
6. Score scales generation & equating
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
7
Bases for a Two-Part Test
1. PACT key aptitudes for the medical course
2. Academic requirements of medical schools
3. Possibility of test bias
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
8
Test Form Equating
• The process of making the score scales of different test forms equivalent to be able to directly compare the scores from the different test forms
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
9
Test Form Equating
• NMAT subtest standard scores are comparable across subtests and across test forms
• APT and SA composite scores are comparable with each other and across test forms
• GPSs are comparable across test forms
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
10
Test ResearchConcurrent Validity Study
Sample Multiple Correlations
APT subtest scores & GPA
SA subtest scores & GPA
College freshmen .61 .54
College seniors .36 .33
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
11
Test ResearchNMAT Highlights 1986–1997
• NMAT: Overall performance remained relatively stable across time.
• PLE: Overall performance declined across the years.
• NMAT & PLE relationship: Students who were selected on the basis of the NMAT had a better chance of passing the PLE.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
12
Test ResearchNMAT Scores of 2006 PLE Passers• PLE passers w/ NMAT taken once (N = 1,176):
GPS Mean = 584; SD = 87
GPS Distribution on Quality Indices
GPS Range QI Range %
524–800 HA–E 76
476–523 A 14
200–475 VP–LA 10
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
13
Test ResearchNMAT Scores of 2006 PLE Passers
• PLE passers w/ NMAT retaken (N = 182):GPS on retake not significantly different from initial GPS
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
14
Future Direction
• Development of shorter test forms that produce better ability estimates and efficiently predict academic performance in the medical course
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
15
• The minimum requirement for NMAT to be useful in decision-making is an empirical demonstration of its predictive validity.
• Predictive validity refers to the efficiency of NMAT in forecasting any measure of academic performance in medical school.
Predictive Validity Study
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
16
• Technically, predictive validity of NMAT is expressed as a bivariate or multivariate correlation coefficient of NMAT scores and college grades.
• The higher this correlation coefficient, the more confidence can be placed on the NMAT as a valid predictor of academic performance.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
17
The variables• Predictor variable
1. NMAT General Performance Score (NGPS)
• Dependent or Criterion variables
2. First year college grade point average (FGPA)based on first year academic subjects
3. Overall college grade point average (OGPA)based on ALL academic subjects
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
18
The validation samples• Medical school applicants who were accepted from SY1997-98 to
SY2002-03
School A (Luzon) 744
School B (Visayas) 205
School C (Mindanao) 232
• Three (3) medical schools A, B, and C; located in Luzon, Visayas, and Mindanao; respectively, participated in this validity study
• The final sample sizes include only students with complete data on all the three variables under study:
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
19
Data Analysis• Computation of weighted GPAs using the prescribed
number of hours per academic subject as weights
• Computation of correlation coefficients between NMAT test scores and a criterion of medical school performance
The accuracy of prediction is usually represented by the correlation coefficients.
That is, the extent to which scores on the NMAT are correlated to success in medical college is the extent to which the test fulfills its predictive function.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
20
• Adjustment of computed correlation coefficients due to restriction of NGPS within the range above cut-off scores used by schools (which tends to severely decrease the correlation of NGPS with other variables.)
Adjusted or otherwise, the values of correlation coefficients are always interpreted within the context in which it was computed. Coefficients as low as .30 are generally considered appropriate and valid for predictive purposes (Bobko, 1995).
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
21
• Computation of the coefficient of determination, d, which indicates the proportion of variance in the FGPA or in the OGPA that is predicted by NGPS
• Derivation of prediction equation which estimates the FGPA or OGPA for a given NGPS
• Generating expectancy table which estimates the probability of a medical school applicant to attain certain FGPA or OGPA based on his/her NGPS
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
22
ResultsTABLE 2. Pearson correlation coefficients
School A School B School C Variable pair Restricted Adjusted Restricted Adjusted Restricted Adjusted
NGPS x FGPA r12 -0.375 -0.545 -0.347 -0.560 -0.380 -0.604 NGPS x OGPA r13 -0.373 -0.542 -0.346 -0.558 -0.296 -0.496
Performance in NMAT is moderately correlated with academic performance in the first year of medical education as well as with performance over three or four years of academic subjects.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
23
Table 3 shows that around 30% of the total variation in academic performance can be explained by performance in the NMAT. In other words, the use of the NMAT GPS as predictor makes forecasting 30% to 36% better than mere random guessing.
TABLE 3. Coefficients of determination Variable pair School A School B School C
NGPS x FGPA d12 0.2966 0.3131 0.3648 NGPS x OGPA d13 0.2940 0.3112 0.2461
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
24
School A: Predicted FGPA = -0.0025 (NGPS) + 3.8076Predicted OGPA = -0.0020 (NGPS) + 3.5354
School B: Predicted FGPA = -0.0026 (NGPS) + 3.8048Predicted OGPA = -0.0021 (NGPS) + 3.6110
School C: Predicted FGPA = -0.0023 (NGPS) + 3.7591Predicted OGPA = -0.0011 (NGPS) + 2.9049
The prediction equations can be used to estimate the first year GPA and the overall academic performance of a medical student applying for admission to medical School A, B, or C.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
25
TABLE 5. Index of predictive efficiency Predicted variable School A School B School C
FGPA 16.13% 17.12% 20.30% OGPA 15.98% 17.00% 13.17%
Table 5 shows that the prediction of academic performance in the three medical schools based on the derived prediction equations is about 13% to 20% better than prediction based on knowledge of the mean of the obtained FGPA or OGPA.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
26
Students who obtain an NMAT GPS between 651 and 700 have a 27 percent probability of attaining a first year GPA of 2.00 or higher in School A.
TABLE 7.2. Expectancy or probability of obtaining a particular FGPA at different ranges of NGPS in School A FGPA
NGPS 1.75 & Below
1.76 -2.00
2.01 -2.25
2.26 -2.50
2.51 -2.75
2.76 -3.00
Below 501 ……… 4% 10% 34% 46% 6% 501-550 ……… 4% 19% 34% 35% 9% 551-600 1% 10% 20% 35% 27% 7% 601-650 2% 15% 29% 27% 22% 5% 651-700 1% 26% 40% 20% 9% 4% Above 700 11% 44% 22% 11% 11% ………
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
27
TABLE 7.3. Expectancy or probability of obtaining a particular OGPA at different ranges of NGPS in School A FGPA
NGPS 1.75 & Below
1.76 -2.00
2.01 -2.25
2.26 -2.50
2.51 -2.75
2.76 -3.00
Below 501 2% 4% 36% 54% 4% 501-550 1% 16% 38% 43% 2% 551-600 1% 5% 17% 46% 28% 3% 601-650 2% 8% 26% 36% 26% 2% 651-700 12% 42% 35% 11% Above 700 5% 35% 50% 30% 15%
Students who obtain an NMAT GPS between 651 and 700 have also 12 percent probability of attaining an overall GPA of 2.00 or higher in School A.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
28
NMAT subtests that are predictive of FGPA and OGPA:
School A: FGPA PHYS, BIOL, CHEM, SSCI, VCOMOGPA CHEM, SSCI, PHYS, QUAN
School B: FGPA BIOL, PHYS, VCOMOGPA BIOL, VCOM, PHYS
School C: FGPA SSCI, CHEM, PHYSOGPA SSCI, CHEM
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
29
Summary and Conclusion
• The use of the NMAT GPS as predictor makes forecasting 30% to 36% better than mere random guessing
• Predicting future academic performance based on NMAT GPS and using the derived prediction equations is about 13% to 20% more accurate than predicting based on knowledge of the mean of the obtained FGPA or OGPA of the validation sample used.
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
30
Recommendations• It should be noted that test scores are always subject to
measurement errors. Hence, it is advised that other measures that assess other student characteristics (e.g., socio-demographics, motivation levels, etc.) may be considered before coming up with a final decision whether to accept or reject a candidate for admission to School A, B, or C.
• Test components should be reviewed and new test specifications should be considered to reflect subtests that highly correlate with academic performance
Center for
Educational
Measurement, Inc.
T h e t e s t o f e x c e l l e n c e
31
• Explore the option of restoring a national cut-off score for quality assurance
• Address the need for more research studies on the implications of NMAT on quality assurance. This will entail full cooperation among medical schools and other agencies in providing the necessary database.
Recommended