Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
RESEARCHREPORT
March 2001RR-01-01
Statistics & Research DivisionPrinceton, NJ 08541
Examinees’ Attitudes Aboutthe TOEFL-CBT, PossibleDeterminants, and RelationshipsWith Test Performance
Lawrence J. StrickerGita Z. Wilder
Examinees’ Attitudes About the TOEFL-CBT,Possible Determinants, and Relationships
with Test Performance
Lawrence J. Stricker and Gita Z. Wilder
March 2001
Research Reports provide preliminary and limiteddissemination of ETS research prior to publication. They areavailable without charge from the
Research Publications OfficeMail Stop 07-REducational Testing ServicePrinceton, NJ 08541
Examinees� Attitudes About the TOEFL-CBT,Possible Determinants, and Relationships
with Test Performance
Lawrence J. Stricker and Gita Z. Wilder
March 2001
Abstract
The principal aim of this study was to assess examinees� acceptance of the TOEFL-CBT,
and its associations with possible determinants of this acceptance and with test
performance. A secondary goal was to evaluate differences in the pattern of results for
examinees from different countries. A questionnaire concerning attitudes about the test,
familiarity with computers, and other relevant variables was administered to TOEFL-
CBT examinees at large testing centers in Buenos Aires, Cairo, and Frankfurt. Attitudes
about the TOEFL-CBT were moderately positive and had similar patterns of relationships
in the three countries: slight or moderate with test performance, moderate with attitudes
about admissions tests in general, slight with computer anxiety and test anxiety, and
minimal with other variables. (Key words: TOEFL, computer-adaptive testing, attitudes,
test anxiety, computer anxiety, computer familiarity, test preparation)
The introduction of the TOEFL-CBT inevitably raises questions about examinees�
reactions to it. This issue is important not only because of customer service concerns but
also because the test�s validity may be affected insofar as these reactions represent an
irrelevant source of variance in test performance.
Kirsch, Jamieson, Taylor, and Eignor (1998) have already demonstrated that
computer familiarity is related to performance on the paper-and-pencil TOEFL (high
scorers are more familiar with computers) and to examinees� gender (men are more
familiar) and their native language (e.g., Spanish speakers are more familiar, Japanese
speakers are less familiar).
Other research with college students and adults has established, in turn, that
computer familiarity is related to acceptance and other attitudes about computers (Burke,
Normand, & Raju, 1987; Powers & O�Neill, 1993; Wilder, Mackie & Cooper, 1985) and
to anxiety about computers (Kernan & Howard, 1990; Powers & O�Neill, 1993). Less is
known about the relationship of familiarity and computer anxiety with performance on
computerized tests. Familiarity was related to performance in one study (Lee, 1986) but
not in three others (Powers & O�Neill, 1993; Taylor, Jamieson, Eignor, & Kirsch, 1998;
Wise, Barnes, Harvey, & Plake, 1989), and anxiety and performance were unrelated in
three studies (Mazzeo, Druesne, Raffeld, Checketts, & Muhlstein, 1992; Powers &
O�Neill, 1993; Wise et al., 1989). The Taylor et al. (1998) study is especially relevant,
for it concerns examinees� performance on a prototype TOEFL-CBT. Finally, several
studies have established high acceptance of computerized tests (Powers & O�Neill, 1993;
Schmitt, Gilliand, Landis, & Devine, 1993; Schmidt, Urry, & Gugel, 1978). And
Jamieson, Taylor, Kirsch, and Eignor (1999) found that providing TOEFL examinees
with a computer-administered tutorial on taking a prototype TOEFL-CBT increased their
acceptance of that test, particularly among examinees who were less familiar with
computers.
A number of key issues remain unresolved. The bulk of this research, with the
notable exceptions of the Kirsch et al. (1998), Taylor et al. (1998), and Jamieson et al.
work (1999), has been done with Americans and may not be applicable to TOEFL
examinees. Nothing is known about the acceptance of the operational version of the
2
TOEFL-CBT. And nothing is known about the implications of acceptance for
computerized test performance in general or on TOEFL-CBT performance in particular.
Moreover, both the determinants and generality of CBT acceptance remain unclear. Is
CBT acceptance distinct from acceptance of tests in general, paper-and-pencil or
otherwise? Is CBT acceptance linked to anxiety about computers or anxiety about tests?
Does CBT acceptance and its determinants vary from culture to culture? Answers to
these questions are needed for a clear understanding of the nature of acceptance of the
TOEFL-CBT.
Accordingly, the main purpose of this study was to assess examinees� acceptance
of the TOEFL-CBT, and its associations with possible determinants and with test
performance. A secondary goal was to evaluate differences in the pattern of results for
examinees from different language groups and different countries.Method
Sample
The sample was drawn from examinees taking the TOEFL-CBT in the Spring and
Summer of 1999 at a testing center operated by or affiliated with Sylvan Learning
Systems in each of three cities: Buenos Aires, Cairo, and Frankfurt. These three centers
were chosen because of their large testing volumes; diversity, spanning the world and
major language groups; and relative within-center homogeneity in native language and
country of origin of the examinees.
The sample consisted of everyone who agreed to participate: 144 examinees in
Buenos Aires tested between June 12 and August 11, 142 examinees in Cairo tested
between June 14 and June 28, and 403 examinees in Frankfurt tested between June 7 and
August 12, (one examinee with unusable data was excluded), a total of 689 examinees1.
Examinees tested during these periods but not participating in the study numbered 152 in
Buenos Aires, 75 in Cairo, and 182 in Frankfurt, a total of 409 examinees. The
participation rates were 48.6% in Buenos Aires, 65.4% in Cairo, and 68.9% in Frankfurt.
Measures
Questionnaire. A questionnaire (in English) made up of a set of scales and two
single-item measures was employed. (The questionnaire appears in the Appendix.) Most
3
of the scales were derived from existing measures to maximize the scales� validity and to
ensure comparability with previous studies.
1. TOEFL-CBT Acceptance. This is a 12-item scale (e.g., �The TOEFL tells
how well people can use English in school�) with Agree, Do Not Agree, and Do Not
Know options. (To obviate language problems, the instructions were to use the Do Not
Know option if �you do not know whether you agree with the statement or do not
understand the statement�.) Most of the items were adapted from measures of attitudes
towards admissions tests and tests used in industrial selection (Arvey, Strickland,
Drauden, & Martin, 1990; Baird, 1977; Rowan & Blewitt Incorporated, 1998; Schmitt et
al., 1993; Smither, Reilly, Millsap, Pearlman, & Stoffey, 1993).
2. Acceptance of Admissions Tests. This is a five-item scale (e.g., �People who
receive high scores on university admissions tests will be successful in school�) with
Agree, Do Not Agree, and Do Not Know options. All of the items were adapted from
measures of attitudes towards admissions tests, standardized tests, or tests used in
industrial selection (Fiske, 1967; Gilliand, 1994; Schmitt et al., 1993; Smither et al.,
1993).
3. Computer Liking. This is a five-item scale adapted from the Computer Liking
factor subscale (Bandalos & Benson, 1990) on the Computer Attitude Scale (Loyd &
Gressard, 1984 (e.g., �I want to use computers�), a measure of computer anxiety,
with Agree, Do Not Agree, and Do Not Know options. Note that a high score represents
low anxiety.
4. Computer Confidence. This is a seven-item scale adapted from the Computer
Confidence factor subscale (Bandalos & Benson, 1990) on the Computer Attitude Scale
(e.g., �I feel I know what I am doing when I use computers�) with Agree, Do Not Agree,
and Do Not Know options. Again note that a high score represents low anxiety.
5. Total Computer Attitude. This is the total score for the Computer Liking and
Computer Confidence scales. Note that a high score represents low anxiety.
6. Worry. This is a five-item scale adapted from the Worry subscale on the
Revised Test Anxiety Scale (Benson & El-Zahhar, 1994) (e.g., �When I am taking a test,
4
I often think how difficult the test is�) with Agree, Do Not Agree, and Do Not Know
options.
7. Tension. This is a five-item scale adapted from the Tension subscale of the
Revised Test Anxiety Scale and, to secure sufficient items, from the Test Anxiety
Inventory (Spielberger, 1980) and Reactions to Tests (Sarason, 1984) (e.g.., �I am
nervous about tests�) with Agree, Do Not Agree, and Do Not Know options.
8. Test-Irrelevant Thinking. This is a four-item scale adapted from the Test-
Irrelevant Thinking subscale on the Revised Text Anxiety Scale (e.g., �When I am taking
a test, I think about things I will do after the test�) with Agree, Do Not Agree, and Do
Not Know options.
9. Bodily Symptoms. This is a five-item scale adapted from the Bodily
Symptoms subscale on the Revised Test Anxiety Scale (e.g., �My mouth becomes dry
during a test�) with Agree, Disagree, and Do Not Know options.
10. Total Test Anxiety Scale. This is the total score for the Worry, Tension,
Test-Irrelevant Thinking, and Bodily Symptoms subscales.
11. Computer Familiarity. This is a six-item scale adapted from the Computer
Familiarity Questionnaire (Eignor, Taylor, Kirsch, & Jamieson, 1998) (e.g., �How often
do you use a computer?�) with Never, Once a Week or Less, More than Once a Week,
and Do Not Know options.
12. Preparation for TOEFL. This is a single-item measure, �About how many
hours did you prepare for the computer-based TOEFL?,� with a seven-point scale (0
Hours to More than 40 Hours) plus a Do Not Know option.
13. Admissions Tests Taken. This is a single-item measure, �About how many
different university admissions tests have you taken?� with a five-point scale (1 test to 5
or More Tests) plus a Do Not Know option.
The items on the questionnaire were written or revised (1) to be clear to ESL
students and easy for them to answer and (2) to balance the direction of keying of items
on the parent scales to minimize acquiescence response style. Versions of the
questionnaire were successively revised on the basis of reviews by an ESL expert and
5
pilot tests with three groups of Arabic-, German-, and Spanish-speaking ESL students
from the same geographical regions or even the same countries where the testing centers
were located: 13 students from Hunter College, City University of New York; 12 students
from Rutgers University--Newark, and 6 students from Middlesex Community College
(New Jersey). The students were at the intermediate level in ESL programs at these
universities, and were not matriculated at the institutions.
Each scale was item analyzed for the total sample, using two indexes: (1) the
item-total score correlation (corrected for item overlap in the total score) and (2) the item-
subgroup (i.e., testing center) partial correlation, controlling for the total score. The latter
is an index of differential item functioning (Stricker, 1982).
In scoring items and obtaining total scores for all scales except Computer
Familiarity, items were scored 1 for the keyed response (Agree or Do Not Agree), -1 for
the unkeyed response (Agree or Do Not Agree), and 0 for the Do Not Know response.
For Computer Familiarity, items were scored 1 for the Never response to 4 for the More
Than Once a Week response, and Do Not Know responses were not scored.
A total score was not obtained on a scale if an examinee had an excessive number
of omitted or not reached responses on the scale: more than two on TOEFL-CBT
Acceptance, and more than one on Acceptance of Admissions Tests, Total Test Anxiety
Scale (Worry, Tension, Test-Irrelevant Thinking, and Bodily Symptoms combined),
Total Computer Attitudes (Computer Liking and Computer Confidence combined), and
Computer Familiarity. A total score for Computer Familiarity was also not obtained if an
examinee had any Do Not Know responses on this scale. If an examinee had fewer than
the critical number of omitted or not reached responses on a scale, the center�s modal
responses to the items were substituted for the missing responses in obtaining the total
score and in item and reliability analyses.
Items with item-total score correlations below .15 or item-subgroup partial
correlations at or above ± .15 were eliminated: three items on Acceptance of TOEFL-
CBT (leaving a nine-item scale), two items on Acceptance of Admissions Tests (leaving
a three-item scale), and one item on Worry (leaving a four-item scale).
6
In scoring the single-item measures, responses on Preparation for TOEFL were
dichotomized: 0 hours and 1 to 5 Hours = 0, 6 to 10 Hours to More than 40 Hours = 1,
and Do Not Know responses were not scored. And Admissions Tests Taken was scored:
1 Test = 1, 2 Tests = 2, 3 Tests = 3, 4 Tests = 4, and 5 or More Tests = 5; Do Not Know
responses were not scored.
Other Variables. Additional variables were obtained from other information
provided by the examinees during the TOEFL-CBT administration.
1. Age (age in years at the time of testing).
2. Sex.
3. Native language.
4. Native country.
5. Reason for taking TOEFL.
6. Planning study in United States or Canada.
7. TOEFL Listening score. This is a scaled score, ranging from 0 to 30.
8. TOEFL Structure/Writing score. This is a scaled score, ranging from 0 to 30.
9. TOEFL Reading score. This is a scaled score, ranging from 0 to 30.
10. TOEFL Total score. This is a scaled score, ranging from 0 to 300.
Procedure
The test center staff was instructed to recruit everyone who took the TOEFL
during the data collection period. After an examinee completed the test and logged off, a
staff member explained the purpose of the study and asked him or her to stay and
complete the questionnaire. Participants were given a Parker ballpoint pen with a retail
value of $19. The questionnaire took approximately 15 minutes to complete.
Analysis
Chi Square tests of the categorical background variables for participants and
nonparticipants were carried out for each center. Parallel one-way analyses of variance
7
of the means for age and the TOEFL scores for participants and nonparticipants were also
carried out.
Chi Square tests for individual items on TOEFL-CBT Acceptance and
Acceptance of Admissions Tests in the three centers were carried out.
One-way analyses of variance of the means for the questionnaire scales and
Admissions Tests Taken, age, and the TOEFL scores in the three centers were carried
out. Parallel Chi Square tests of Preparation for TOEFL and the categorical background
variables were also carried out.
The product-moment intercorrelations of the questionnaire measures, TOEFL
scores, and background variables were computed separately for each center, using a pair-
wise missing data program. In these analyses, dummy codes were used for Sex (male =
0, female = 1), Native Language (same as the country where the center is located = 1, all
others = 0), Native Country (same as the country where the center is located = 1, all
others = 0), Reason for Taking TOEFL,2 and Planning to Study in United States or
Canada (yes = 1, no = 0).
The difference between corresponding correlations for pairs of centers was
evaluated by a Z test of the transformed correlations.
The internal-consistency reliability of the questionnaire scales for each center was
computed by Coefficient Alpha.
Both statistical and practical significance were considered in evaluating the
results. For statistical significance, an .05 alpha level was used in all analyses. For
practical significance, indexes that reflect a �small� effect size, accounting for 1% of the
variance, were used: an η of .10 in the analyses of variance, an r of .10 and a difference
in r of .10 in the correlation analyses, and a W of .10 in the Chi Square analyses
(Cohen, 1988). Because of the large number of variables, the focus was on consistent
results within and between centers.
8
Results
Comparisons of Background Variables and TOEFL Scores for Participants andNonparticipants
The frequency distributions of the background variables for the participants and
nonparticipants in each center and Chi Square tests are summarized in Table 1. The
means for age and TOEFL scores, and analysis of variance statistics are summarized in
Table 2.
In Buenos Aires, the differences in background characteristics and TOEFL scores
for participants and nonparticipants were not statistically or practically significant. In
Cairo, the differences in background characteristics were not significant, but the
differences in TOEFL Reading and TOEFL Total were significant (participants had
higher scores than nonparticipants on both variables). In Frankfurt, the differences in two
background characteristics were significant: Native Country and Native Language (more
participants were German-born and native speakers of German); differences in all
TOEFL scores were also significant (participants had higher subtest and total scores).
Summary. In brief, study participants and nonparticipants did not differ in
Buenos Aires, but did differ in Cairo and Frankfurt, with the participants performing
better on the TOEFL in both cities, and being more likely to be German-born and native
speakers of German in Frankfurt.
______________________________
Insert Tables 1 and 2 about here.______________________________
Individual Items on Acceptance of TOEFL-CBT and Acceptance of Admissions Tests
The responses to the individual items on TOEFL-CBT Acceptance in the three
centers and Chi Square analyses are summarized in Table 3; the corresponding data for
the items on Acceptance of Admissions Tests are summarized in Table 4.
TOEFL-CBT Acceptance. In general, most examinees gave favorable responses
to the items on TOEFL-CBT Acceptance. In only four out of 27 instances (nine items by
three centers) did most examinees choose an unfavorable response, and three of the
instances involved the same item. Examinees in every center predominantly choose the
9
unfavorable response (Agree) to this item, �My English is better than my TOEFL scores
demonstrate.�
The differences among the centers were statistically and practically significant for
six items. For these items, the most favorable (Agree) responses were made by Cairo to
�A school or company can learn a lot about a person�s English ability from their TOEFL
scores� and to �The TOEFL tells how well people can use English in school�, and the
most favorable (Do Not Agree) responses were made by Frankfurt to �The TOEFL
examines English skills I will never need�, by Cairo to �My English skills are better than
my TOEFL scores demonstrate�, by Frankfurt to �The TOEFL did not give me a good
opportunity to demonstrate my English ability�, and by Frankfurt to �The English skills
tested by the TOEFL are different from the English needed in school�.
Acceptance of Admissions Tests. The results were more mixed for examinees�
responses to the items on Acceptance of Admission Tests. In five out of nine instances
(three items by three centers), most examinees chose an unfavorable response. Four of
these instances involved the same two items. Examinees in Buenos Aires and Frankfurt
predominantly chose the unfavorable response (Do Not Agree) to these items, �People
who receive high scores on university admissions tests will be successful in school� and
�My scores on university admissions tests accurately demonstrate my skills and abilities.�
The differences among the centers were statistically and practically significant for
all items. The most favorable (Agree) responses were made by Cairo to �People who
receive high scores on university admissions tests will be successful in school�, �My
scores on university admissions tests accurately demonstrate my skills and abilities�, and
�University admissions tests are a good way to learn about a person�s skills and abilities�.
Summary. In short, examinees� attitudes were generally positive towards theTOEFL-CBT and more mixed towards admissions tests in general. The centers differedinconsistently on individual attitude items concerning the TOEFL-CBT, but Cairo haduniformity more favorable responses than other centers on items concerning admissionstests in general.
______________________________
Insert Tables 3 and 4 about here.______________________________
10
Comparisons of Questionnaire and Background Variables, and TOEFL Scores forCenters
The mean scores on all of the questionnaire variables (except Preparation for
TOEFL) for the three centers and analysis of variance statistics are summarized in Table
5; the corresponding data for age and TOEFL scores are summarized in Table 2. The
frequency distributions for the remaining variables and the Chi Square tests are
summarized in Tables 1 and 6.
Questionnaire Variables. The differences among the three centers were not
statistically and practically significant for TOEFL-CBT Acceptance, but they were
significant for Acceptance of Admissions Tests (Cairo was highest) and most of the other
questionnaire variables: Computer Liking (Cairo was highest), Total Computer Attitude
(Cairo was highest), Worry (Cairo was highest), Test Irrelevant Thinking (Cairo was
highest), Bodily Symptoms (Frankfurt was lowest), Total Test Anxiety Scale (Cairo was
highest), Computer Familiarity (Buenos Aires was highest), and Preparation for TOEFL
(Buenos Aires was highest).
It is noteworthy that the mean scores for TOEFL-CBT Acceptance and
Acceptance of Admissions Tests were consistent with the item-level analyses. The
scores were moderately positive for TOEFL-CBT Acceptance (item means of .25 to .30;
the theoretical score range is 1 to �1) and moderately positive to moderately negative for
Acceptance of Admissions Tests (item means of -.20 to .21; the theoretical score range is
1 to �1).
Background Variables. The differences among the centers were significant for
several background variables: Age (Buenos Aires was highest), Sex (Cairo had the most
males), and Reason for Taking TOEFL--Undergraduate (Buenos Aires had the fewest
undergraduates).
TOEFL Scores. The differences among the centers were significant for all of the
TOEFL scores: Listening, Structure/Writing, Reading, and Total (Frankfurt was highest
on every score).
11
Summary. In sum, the three centers did not differ in attitudes about the TOEFL-
CBT but did differ in attitudes about admissions tests, TOEFL performance, and most
other questionnaire and background variables.
______________________________
Insert Tables 5 and 6 about here.______________________________
Correlations of TOEFL-CBT, Acceptance of Admissions Tests, and TOEFL Scores WithQuestionnaire and Background Variables.
The correlations of TOEFL-CBT Acceptance, Acceptance of Admissions Tests,
and TOEFL with the other variables in the three centers are reported in Tables 8 to 10.
The internal-consistency reliability of the questionnaire scales are reported in Table 7.
Reliability of Questionnaire Scales. The reliability of TOEFL-CBT Acceptance
was modest, ranging from .60 to .68 over the three cities; the reliability of Acceptance of
Admissions Tests was similar, ranging from .50 to .55. The reliability of the other
questionnaire scales was also generally moderate.
TOEFL-CBT Acceptance Correlates. TOEFL-CBT Acceptance had statistically
and practically significant correlations in every center with several variables: Acceptance
of Admissions Tests, Computer Liking, Total Computer Attitude, TOEFL
Structure/Writing, and TOEFL Total. All of these correlations were positive. In
addition, in Cairo, TOEFL-CBT Acceptance correlated significantly and negatively with
Test-Irrelevant Thinking, Total Test Anxiety, and Preparation for TOEFL. And in
Frankfurt, TOEFL-CBT Acceptance also correlated significantly and positively with
Computer Confidence, Age, and Reason for Taking TOEFL�Graduate student, and
negatively with Worry, Test-Irrelevant Thinking, Total Test Anxiety, Admissions Tests
Taken, and Reason for Taking TOEFL�Undergraduate.
Acceptance of Admissions Tests Correlates. Acceptance of Admissions Tests,
besides its consistent significant correlations with TOEFL-CBT Acceptance across
centers, had sparse and inconsistent correlations with other variables. In Buenos Aires,
Acceptance of Admissions Tests correlated negatively with all TOEFL scores, except
TOEFL Structure/Writing. In Cairo, this variable correlated negatively with TOEFL
12
Reading. And in Frankfurt, it correlated positively with Computer Liking and negatively
with Sex.
TOEFL Total Correlates. TOEFL Total correlated significantly but inconsistently
with a variety of other variables, besides its consistent positive correlations with TOEFL-
CBT Acceptance in the three centers, which was noted previously. (Only the correlations
for TOEFL Total will be discussed for simplicity). TOEFL Total also correlated
consistently and negatively with Bodily Symptoms. In addition, in Buenos Aires TOEFL
Total correlated significantly and positively with Computer Liking, Computer
Confidence, Total Computer Attitude, Reason for Taking TOEFL�Graduate student,
and negatively with Total Test Anxiety and Reason for Taking TOEFL�Other. In
Cairo, TOEFL Total correlated negatively with Preparation for TOEFL. And in
Frankfurt, TOEFL Total correlated positively with Computer Liking, Computer
Confidence, Total Computer Attitude, Native Country, Native Language, and Reason for
Taking TOEFL�Graduate student, and negatively with Worry, Preparation for the
TOEFL, and Reason for Taking TOEFL�Undergraduate.
Center Differences in Correlates. It should be pointed out that many of the
differences in the patterns of correlations for the three centers may be attributable to
sampling error, and the smaller level of correlation required for significance in Frankfurt
(r = .10) relative to Buenos Aires and Cairo (r = .16), given differences in sample sizes
for these centers (403 vs. 142 and 144). However, some of the differences in correlations
were statistically and practically significant: TOEFL-CBT Acceptance�s negative
correlation with Preparation for TOEFL in Cairo and its positive correlations with Reason
for Taking TOEFL�Graduate student and Age in Frankfurt, and TOEFL Total�s
minimal correlations with Preparation for TOEFL in Buenos Aires and Reason for
Taking TOEFL�Graduate student in Cairo, and its positive correlations with Native
Country and Native Language in Frankfurt.
Summary. In brief, favorable attitudes about the TOEFL-CBT were generally
related to favorable attitudes about admissions tests, high performance on the TOEFL,
and low computer anxiety and test anxiety. These favorable attitudes were related to
minimal preparation in Cairo and to age and better education in Frankfurt. Favorable
13
attitudes about admissions tests in general were essentially unrelated to any variable
besides favorable attitudes about the TOEFL-CBT. High TOEFL performance, besides
related to favorable attitudes about the TOEFL-CBT, was also generally related to low
computer anxiety and test anxiety, minimal preparation for the TOEFL, and, in Frankfurt,
better education and native origin.
______________________________
Insert Tables 7 to 10 about here.______________________________
Discussion
Level and Correlates of TOEFL-CBT Acceptance
A central finding was that attitudes about the TOEFL-CBT appeared to be
moderately positive in the countries studied, though attitudes towards admission tests in
general seemed to be mixed in two of the countries (the relatively positive attitudes in
Egypt was the exception). This similarity in attitudes about the TOEFL-CBT in the three
countries is especially remarkable, given the differences among the countries on a wide
array of relevant variables, including attitudes about admissions tests, personality traits,
background characteristics, and TOEFL-CBT performance. Although attitudes towards
the TOEFL-CBT could be more favorable, they are still more favorable than attitudes
towards admissions tests in general. These positive attitudes about the TOEFL-CBT are
consistent with previous studies that found positive attitudes about other computer-
administered tests (Powers & O�Neill, 1993; Schmitt et al., 1993; Schmidt et al., 1978).
In all of these studies, including the present one, the positive attitudes were found after
the examinees had taken the tests. One of these studies (Powers & O�Neill, 1993) also
observed that the attitudes were less favorable before examinees took the test, suggesting
that the actual test-taking experience may have been better than the examinees
anticipated.
An equally important result was that attitudes about the TOEFL-CBT were no
more than moderately related to performance on the test in the three countries. This
outcome suggests that these attitudes do not represent an important source of irrelevant
variance on the test. These modest correlations are especially striking because the
14
TOEFL-CBT routinely reported the test scores to the examinees before they described
their attitudes about the test on the questionnaire, making it possible for the score
feedback to influence the reported attitudes.
Apart from the TOEFL scores, the associations were sparse between attitudes
about the TOEFL-CBT and their potential determinants across the three countries, and
the variables found to be related were not surprising: general attitudes towards
admissions tests, computer anxiety, and test anxiety. One of the few differences among
countries in the relationships between these attitudes and other variables stands out: the
inverse relationship with the amount of test preparation in Egypt. This result is intriguing
and deserves to be followed up. At first glance, this outcome suggests that preparing for
the TOEFL leads to unfavorable attitudes about the test. However, it is equally plausible
that examinees who prepare most do so because their ESL skills are most deficient (e.g.,
Powers & Hecht, 1982), and these poor skills or the low scores the examinees earn on the
TOEFL, despite the preparation, colors their attitudes about the test.
The moderate relationship between attitudes about the TOEFL and attitudes about
admissions tests in general�even corrected for attenuation in both variables, the
correlations ranged only from .44 to .64�along with the different patterns of
relationships between these two kinds of attitudes and the other variables in this study
make it clear that the two, though related, are distinguishable.
Level and Correlates of TOEFL Performance
Although not the focus of the study, TOEFL performance and its correlates are of
interest. The level of performance varied appreciably from country to country, consistent
with other findings (e.g., Computer-Based TOEFL Score User Guide, 1998), and within
countries was weakly related to a few variables, besides attitudes about the TOEFL-CBT.
The inverse relationship between test preparation and test performance in two
countries is noteworthy. Again, the explanation may simply be that examinees with poor
ESL skills prepare the most but this skill deficiency is still reflected in poor scores on the
test. This counterintuitive relationship between test preparation and test performance has
been found in studies of test preparation for admissions tests, such as the
15
Graduate Record Examination (GRE) General Test (Powers, 1985) and the Graduate
Management Admission Test (Powers & Hecht, 1982).
The negative association between test anxiety and test performance is consistent
with other studies of admissions tests, such as the Scholastic Assessment Test (e.g.,
Alpert & Haber, 1960; Sarason, 1957; Sarason & Mandler, 1952) and the GRE General
Test (Powers, 1988).
The negative relationship between computer anxiety and test performance is not
congruent with previous studies of computer-administered tests that found no association
(Mazzeo et al., 1992; Powers & O�Neill , 1993; Wise et al., 1989). One possible
explanation for the divergent results is that these investigations assessed the relationship
between anxiety and performance on the computer-administered test after controlling for
performance on a paper-and-pencil version of the test, but the present study did not use
this control.
The absence of an association between computer familiarity and test performance
is consistent with three of four previous studies of computer-administered tests (Powers
& O�Neill, 1993; Taylor et al., 1998; Wise et al., 1989; the exception was Lee, 1986).
One potentially important difference between all of these studies and the present one is
that, like the computer anxiety studies already discussed, they controlled for performance
on a paper-and-pencil version of the test.
Methodological Issues
In view of the frequent absence of significant results in this study, it is worth
pointing out that the statistical power was adequate (Cohen, 1988). For example, the
power to detect a medium effect size in the correlational analysis, an r of .30, using an
alpha level of .05 and a two-tail test, was .95 for both the Buenos Aires and Cairo
analyses, and .99 for the Frankfurt analysis. And the power to detect a medium effect
size in the analyses of variance that compare the three cities, an η of .24, using an alpha
level of .05, was .95.
The questionnaire used in this study also deserves comment. Because most
TOEFL test takers have sufficient grasp of English that they plan to study in North
16
America, a single English-language questionnaire was feasible. Using an English-
language questionnaire avoided the need to develop separate versions in Arabic, German,
and Spanish and then establish that they were linguistically and psychometrically
comparable, a complex, lengthy, and expensive endeavor (e.g., Hambleton, 1994).
In devising the questionnaire for this study, special attention was given to
ensuring that it would be comprehensible to examinees with limited English and function
similarly for examinees from the three countries. In addition, to deal with the possibility
that examinees might still be unable to understand particular items, Do Not Know
alternatives were provided for use in this case as well as when examinees did not have
opinions or could not report on their characteristics or behavior.3
The success of the entire questionnaire development process�writing, revising,
and pilot testing items; item analyses; and the use of Do Not Know alternatives�is
demonstrated by the relatively small proportion of Do Not Know responses (an average
of 6.9% to 10.4% in the three centers), the small proportion of scores for scales and
single-item measure that were unavailable because of missing data (a maximum of 1.3%
to 3.5% in the centers for all but three variables),4 the adequate reliability of the scales,
and the coherent pattern of correlations among the scales and single-item measures.
Conclusion
The main lesson to be drawn from this study is that the TOEFL-CBT possesses a
reasonable degree of acceptance by examinees, but there is clearly room for
improvement. However, it is probably unrealistic to expect extremely favorable attitudes
towards any high-stakes test. Steps could be taken to increase acceptance of the test,
such as improving test preparation materials. These efforts might be worthwhile from a
customer service standpoint, but there seems to be no compelling reason to do them in an
attempt to enhance the test�s construct validity, given the modest relationship between
acceptance and test performance.
17
References
Alpert, R., & Haber, R.N. (1960). Anxiety in academic achievement situations. Journalof Abnormal and Social Psychology, 61, 207-215.
Arvey, R.D., Strickland, W., Drauden, G., & Martin, C. (1990). Motivationalcomponents of test taking. Personnel Psychology, 43, 695-716.
Baird, L.L. (1977). What graduate and professional school students think aboutadmissions tests. NCME Measurement in Education, 7 (3), 1-7.
Bandalos, D., & Benson, J. (1990). Testing the factor structure invariance of a computerattitude scale over two grouping conditions. Educational and PsychologicalMeasurement, 50, 49-60.
Benson, J., & El-Zahhar, N. (1994). Further refinement and validation of the RevisedTest Anxiety Scale. Structural Equation Modeling, 1, 203-221.
Burke, M.J., Normand, J., & Raju, N. S. (1987). Examinee attitudes toward computer-administered ability tests. Computers in Human Behavior, 3, 95-107.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).Hillsdale, NJ: Erlbaum.
Computer-based TOEFL score user guide, 1998-1999 (1998). Princeton, NJ:Educational Testing Service.
Eignor, D., Taylor, C., Kirsch, I., & Jamieson, J. (1998). Development of a scale forassessing the level of computer familiarity of TOEFL examinees (TOEFLResearch Report 60; ETS Research Report 98-7). Princeton, NJ: EducationalTesting Service.
Fiske, D.W. (1967). The subject reacts to tests. American Psychologist, 22, 287-296.
Gilliand, S.W. (1994). Effects of procedural and distributive justice on reactions to aselection system. Journal of Applied Psychology, 79, 691-701.
Hambleton, R.K. (1994). Guidelines for adapting educational and psychological tests: Aprogress report. European Journal of Psychological Assessment, 10, 229-244.
Jamieson, J., Taylor, C., Kirsch, I., & Eignor, D. (1999). Design and evaluation of acomputer-based TOEFL tutorial (TOEFL Research Report 62; ETS ResearchReport 99-01). Princeton, NJ: Educational Testing Service.
18
Kernan, M.C., & Howard, G. S. (1990). Computer anxiety and computer attitudes: Aninvestigation of construct and predictive validity issues. Educational andPsychological Measurement, 50, 681-690.
Kirsch, I., Jamieson, J., Taylor, C., & Eignor, D. (1998). Computer familiarity amongTOEFL examinees (TOEFL Research Report 59; ETS Research Report 98-6).Princeton, NJ: Educational Testing Service.
Lee, J.A. (1986). The effects of past computer experience on computerized aptitude testperformance. Educational and Psychological Measurement, 46, 727-733.
Loyd, B.H., & Gressard, C. (1984). Reliability and factorial validity of computer attitudescales. Educational and Psychological Measurement, 44, 501-505.
Mazzeo, J., Druesne, B., Raffeld, P., Checketts, K., & Muhlstein, A. (1992).Comparability of computer and paper-and-pencil scores for two CLEP generalexaminations (College Board Report 91-5; ETS Research Report 92-14). NewYork: College Entrance Examination Board.
Powers, D.E. (1985). Effects of coaching on GRE Aptitude Test scores. Journal ofEducational Measurement, 22, 121-136.
Powers, D.E. (1988). Incidence, correlates, and possible causes of test anxiety ingraduate admissions testing. Advances in Personality Assessment, 7, 49-75.
Powers, D.E., & Hecht, L.W. (1982). Preparing for the Graduate ManagementAdmission Test: A descriptive study of candidates� use of alternative methods oftest preparation (GMAC Research Report 82-2; ETS Research Report 82-36).Princeton, NJ: Educational Testing Service.
Powers, D.E., & O�Neill, K. (1993). Inexperienced and anxious computer users: Copingwith a computer-administered test of academic skills. Educational Assessment, 1,153-173.
Robinson, J.P., Shaver, P.R., & Wrightsman, L.S., (Eds.). (1991). Measures ofpersonality and social psychological attitudes. San Diego: Academic Press.
Rowan & Blewitt Incorporated. (1998). Educational Testing Service nationwide opinionsurvey final report. Washington, DC: Author.
Sarason, I.G. (1957). Test anxiety, general anxiety, and intellectual performance.Journal of Consulting Psychology, 21, 485-490.
Sarason, I.G. (1984). Stress, anxiety, and cognitive interference: Reactions to tests.Journal of Personality and Social Psychology, 46, 929-938.
19
Sarason, S.B., & Mandler, G. (1952). Some correlates of test anxiety. Journal ofAbnormal and Social Psychology, 47, 810-817.
Schmidt, F.L., Urry, V. W., & Gugel, J. F. (1978). Computer assisted tailored testing:Examinee reactions and evaluations. Educational and PsychologicalMeasurement, 38, 265-273.
Schmitt, N., Gilliand, S. W., Landis, R. S., & Devine, D. (1993). Computer-based testingapplied to selection of secretarial applicants. Personnel Psychology, 46, 149-165.
Smither, J.W., Reilly, R. R., Millsap, R. E., Pearlman, K., & Stoffey, R. W. (1993).Applicant reactions to selection procedures. Personnel Psychology, 46, 49-76.
Spielberger, C.D. (1980). Preliminary professional manual for the Test AnxietyInventory. Palo Alto, CA: Consulting Psychologists Press.
Stricker, L.J. (1982). Identifying test items that perform differentially in populationsubgroups: A partial correlation index. Applied Psychological Measurement, 6,261-273.
Taylor, C., Jamieson, J., Eignor, D., & Kirsch, I. (1998). The relationship betweencomputer familiarity and performance on computer-based TOEFL test tasks(TOEFL Research Report 61; ETS Research Report 98-8). Princeton, NJ:Educational Testing Service.
Wilder, G., Mackie, D., & Cooper, J. (1985). Gender and computers: Two surveys ofcomputer-related attitudes. Sex Roles, 13, 215-228.
Wise, S.L., Barnes, L. B., Harvey, A. L., & Plake, B. S. (1989). Effects of computeranxiety and computer experience on the computer-based achievement testperformance of college students. Applied Measurement in Education, 2, 235-241.
20
Footnotes
1Data collection in Buenos Aires and Cairo extended beyond these dates but was
unsystematic and unusable.
2The dummy codes for Reasons for Taking TOEFL were: to be an undergraduate
= 1, all others = 0; to be a graduate student = 1, all others = 0; some other reason = 1, all
others = 0.
3Not comprehending an item because of its language and not having an opinion or
being able to report on a personal characteristic or behavior obviously have different
psychological meaning, but providing separate alternatives would have unduly
complicated the examinee�s task in completing the questionnaire.
In order to preserve as much usable data as possible, Do Not Know responses
were retained in scoring items on scales and assigned a neutral score (0, with keyed
alternatives scored +1 and �1), a common practice in scoring don�t know, uncertain, and
neutral responses on personality and attitude scales (e.g., Robinson, Shaver, &
Wrightsman, 1991).
Retaining these responses but assigning them the mean score for the sample, as
was done with the omitted items, would be inappropriate when they signify that
examinees simply do not have opinions or cannot report on their characteristics or
behavior. Indeed, this seems to be the predominant meaning of these responses, not lack
of comprehension because of language difficulties, judging from the weak or minimal
correlations of Do Not Know responses with the TOEFL scores. For example, the total
number of Do Not Know responses correlated �.08, -.15, and -.18 with TOEFL Total in
the three centers (only the -.18 correlation, in Buenos Aires, was statistically and
practically significant).
Excluding Do Not Know responses and not obtaining a score for a scale when an
examinee had such responses would result in an excessive amount of missing data. For
instance, the percentage of examinees with one or more Do Not Know responses on the
original 12-item TOEFL-CBT Acceptance scale ranged from 60.6% to 71.4% in the
centers.
21
4For the three exceptions, Computer Familiarity, Preparation for TOEFL, and
Admissions Tests Taken, the maximum percentage of missing data was 4.5% to 24.6% in
the three centers.
22
Author Note
Thanks are due to Kurt F. Geisinger, Theresa Hoff Macan, Neal W. Schmitt, and
Thomas E. Scruggs for providing measures of attitudes about tests; Pamela McPartland-
Fairman for reviewing the questionnaire; Eileen Hansen, Mary Mora, and Allison Rice
for assisting in the pilot testing of the questionnaire; Baker Evans, M. Hashem, Roger
Hermodsson, Edwin Kolen, Selma Laporte, Bea Lichtensteiger, Mark Porzuc, and Linda
A. Silverstein for supervising the data collection in the test centers; Edward J. Kelly,
Sandra Lee, and Edward S. Paul for providing logistical advice and support; Debra E.
Friedman for coordinating the data collection; Patricia E. Carey, Karen A. Copper, Hai
Jiang, and Cindy Nguyen for providing TOEFL program data; Min Hwei Wang for doing
the data analysis; and Brent Bridgeman and Howard Wainer for reviewing a draft of this
report.
Table 1
Percentage Distributions of Background Variables for Participants and Nonparticipants in Each Testing Center
_______________________________________________________________________________________________________________________________________________________ Participants in
Buenos Aires Cairo Frankfurt Testing Centers
Variable Participants Nonparticipants χ2 W Participants Nonparticipants χ 2 W Participants Nonparticipants χ 2 W χ 2 W_______________________________________________________________________________________________________________________________________________________
Sex (N = 143) (N = 152) 3.12 .10 (N = 138) (N = 75) .06 .02 (N = 402) (N = 182) .00 .00 17.29** .16
Male 49.0 59.2 73.2 74.7 60.0 59.9
Female 51.0 40.8 26.8 25.3 40.0 40.1
Native Country (N = 138) (N = 150) .05 .01 (N = 131) (N = 67) .61 .06 (N = 389) (N = 176) 19.84** .19 2.71 .06
Same 86.2 85.3 90.8 94.0 91.0 77.3
Different 13.8 14.7 9.2 6.0 9.0 22.7
Native Language (N = 134) (N = 148) .88 .06 (N = 127) (N = 65) .26 .04 (N = 389) (N = 170) 19.54** .19 4.40 .08
Same 91.8 94.6 96.9 95.4 91.3 77.6
Different 8.2 5.4 3.1 4.6 8.7 22.4
Planning Study inUnited States or Canada (N = 143) (N = 151) .29 .03 (N = 139) (N = 74) .01 .01 (N = 402) (N = 182) .06 .01 5.74 .09
Yes 68.5 65.6 65.5 66.2 58.2 57.1
No 31.5 34.4 34.5 33.8 41.8 42.9
Table 1 (continued.)
_______________________________________________________________________________________________________________________________________________________ Participants in
Buenos Aires Cairo Frankfurt Testing Centers
Variable Participants Nonparticipants χ2 W Participants Nonparticipants χ 2 W Participants Nonparticipants χ 2 W χ 2 W_______________________________________________________________________________________________________________________________________________________
Reason for takingTOEFL (N = 144) (N = 152) .93 .06 (N = 138) (N = 74) 4.00 .14 (N = 402) (N = 182) .04 .01 23.82** .19
To be undergraduate 18.1 20.4 28.3 31.1 28.9 28.0
To be graduate Student 49.3 52.0 44.9 54.1 55.5 56.0
Other reasona 32.6 27.6 26.8 14.9 15.7 15.9_______________________________________________________________________________________________________________________________________________________
aTo enter a school other than a college or university, to become licensed to practice my profession in the USA or Canada, to demonstrate my proficiency in English to the
company for which I work or plan to work, other than the above.
*p < .05; ** p < .01.
Table 2
Means of Age and TOEFL Scores for Participants and Nonparticipants in Each Testing Center
_______________________________________________________________________________________________________________________________________________________Participants in
Buenos Aires Cairo Frankfurt Testing CentersVariable Participants Nonparticipants F η Participants Nonparticipants F η Participants Nonparticipants F η F η
Mean S. D. Mean S. D. Mean S. D. Mean S. D. Mean S. D. Mean S. D._______________________________________________________________________________________________________________________________________________________
Age 25.95 5.66 26.86 6.08 1.75 .08 24.83 6.32 24.27 6.18 .40 .04 23.96 4.33 24.14 4.80 .20 .02 8.37** .15
TOEFL ListeningScore 22.03 4.72 22.26 5.14 .16 .02 20.25 5.46 18.09 6.43 6.77**.17 25.28 3.04 23.41 4.11 37.62** .25 94.82** .47
TOEFL Structure/ Writing Score 23.96 3.47 23.37 4.15 1.14 .06 20.65 4.37 21.03 4.86 1.74 .09 25.41 3.42 23.58 4.19 26.04** .21 45.62** .34
TOEFL ReadingScore 22.83 4.48 24.46 3.57 1.51 .07 21.93 4.75 19.11 5.08 5.44* .16 25.27 3.47 23.89 4.27 21.06** .19 89.79** .46
TOEFL TotalScore 229.44 37.81 233.64 38.58 .90 .06 209.44 42.43 194.12 48.42 5.80* .16 253.19 29.00 236.24 37.14 35.74** .24 94.02** .46
_______________________________________________________________________________________________________________________________________________________
Note. The Ns for the participants and nonparticipants are 144 and 152 in Buenos Aires, 142 and 75 in Cairo, and 403 and 182 in Frankfurt.
* p < .05; ** p < .01.
Table 3
Percentage Distributions of Responses to Items on TOEFL-CBT Acceptance in Each Testing Center
___________________________________________________________________________________________________________________________________________________ Participants in
Buenos Aires Cairo Frankfurt Testing CentersDo Not Do Not Do Not Do Not Do Not Do Not
Item N Agree Agree Know N Agree Agree Know N Agree Agree Know χ2 W______________________________________________________________________________________________________________________________________
2. The different types of questions in theTOEFL are good for testing the Englishskills needed in school. 144 80.6 8.3 11.1 140 78.6 14.3 7.1 400 76.8 14.0 9.3 4.34 .08
3. The TOEFL examines English skills Iwill never need. 144 13.9 72.2 13.9 141 19.9 69.5 10.6 399 8.5 80.5 11.0 14.66** .15
4. A school or company can learn a lotabout a person�s English ability fromthe TOEFL scores. 142 47.9 35.9 16.2 141 61.7 26.2 12.1 401 42.6 33.2 24.2 19.20** .17
5. A low score on the TOEFL showsthat your English ability is limited. 144 47.2 40.3 12.5 140 50.0 41.4 8.6 400 49.5 36.0 14.5 4.04 .08
6. The TOEFL was difficult becauseI had to use a computer. 143 11.2 87.4 1.4 142 12.7 85.9 1.4 402 9.0 87.8 3.2 3.90 .08
7. My English skills are better than myTOEFL scores demonstrate. 142 49.3 18.3 32.4 140 55.7 21.4 22.9 402 31.3 16.4 52.2 46.00** .26
Table 3 (continued)
___________________________________________________________________________________________________________________________________________________ Participants in
Buenos Aires Cairo Frankfurt Testing CentersDo Not Do Not Do Not Do Not Do Not Do Not
Item N Agree Agree Know N Agree Agree Know N Agree Agree Know χ2 W______________________________________________________________________________________________________________________________________
8. The TOEFL did not give me a goodopportunity to demonstrate my Englishability. 144 35.4 38.9 25.7 142 50.0 38.7 11.3 400 32.0 44.3 23.8 19.55** .17
9. The TOEFL tells how well people canuse English in school. 144 36.8 28.5 34.7 141 61.7 27.0 11.3 402 40.0 37.6 22.4 35.27** .23
11. The English skills tested by the TOEFLare different from the English neededin school. 142 16.9 46.5 36.6 142 42.3 48.6 9.2 402 33.1 49.3 17.7 44.55** .25
___________________________________________________________________________________________________________________________________
Note. The favorable response is underlined.
* p < .05; ** p < .01.
Table 4
Percentage Distributions of Responses to Items on Acceptance of Admissions Tests in Each Testing Center
_______________________________________________________________________________________________________________________________________________________ Participants in
Buenos Aires Cairo Frankfurt Testing CenterDo Not Do Not Do Not Do Not Do Not Do Not
Item N Agree Agree Know N Agree Agree Know N Agree Agree Know χ2 W_________________________________________________________________________________________________________________________________________
3. People who receive high scores onuniversity admissions tests will besuccessful in school. 138 17.4 51.4 31.2 139 40.3 30.2 29.5 400 15.5 53.5 31.0 43.30** .25
4. My scores on university admissionstests accurately demonstrate my skillsand abilities 141 24.8 37.6 37.6 142 40.1 32.4 27.5 399 17.8 37.8 44.4 30.57** .21
5. University admissions tests are a goodway to learn about a person�s skills andabilities. 140 45.0 26.4 28.6 142 62.0 18.3 19.7 399 34.3 34.8 30.8 34.11** .22
________________________________________________________________________________________________________________________________________
Note. The favorable response is underlined.
* p < .05; ** p < .01.
Table 5
Means of Questionnaire Variables in Each Testing Center
_______________________________________________________________________________________
Buenos Aires Cairo Frankfurt Variable N Mean S. D. N Mean S. D. N Mean S. D. F η
_______________________________________________________________________________________
TOEFL-CBT Acceptance 144 2.38 3.45 142 2.29 3.92 403 2.55 3.77 .31 .03
Acceptance of Admissions Tests 140 -.29 1.75 142 .63 1.75 401 -.59 1.66 27.26** .27
Computer Liking 144 2.84 2.54 140 3.34 2.06 401 2.36 2.63 8.50** .16
Computer Confidence 144 4.56 2.86 140 4.81 2.75 399 4.45 3.08 .73 .05
Total Computer Attitude 144 7.40 4.87 140 8.15 4.20 399 6.81 5.20 3.92* .11
Worry 142 -.31 2.29 142 -.04 2.21 400 -.60 2.27 3.41* .10
Tension 142 1.48 2.89 140 1.06 2.72 398 .89 3.21 1.96 .08
Test-Irrelevant Thinking 141 -2.66 2.07 140 -1.56 2.45 398 -2.26 2.41 7.94** .15
Bodily Symptoms 141 -2.57 2.36 141 -2.05 2.53 400 -3.36 2.07 20.11** .24
Total Test Anxiety Scale 139 -4.10 6.14 140 -2.60 6.64 398 -5.33 6.12 9.38** .16
Computer Familiarity 125 15.60 2.13 107 14.34 3.28 357 14.68 2.27 8.99** .17
Admissions Tests Taken 121 1.35 .68 115 1.49 .88 385 1.32 .70 2.17 .08_______________________________________________________________________________________
* p < .05; ** p < .01.
Table 6
Percentage Distributions of Preparation for TOEFL in Each Testing Center
______________________________________________________________________________
Variable Buenos Aires Cairo Frankfurt χ2 W(N = 121) (N = 119) (N = 388)
______________________________________________________________________________
0 to 5 hours 25.6 48.7 53.1 28.13** .21
6 hours or more 74.4 51.3 46.9______________________________________________________________________________
*p < .05; ** p < .01
Table 7
Reliability of Questionnaire Scales in Each Testing Center
_____________________________________________________________________________________________
Scale Buenos Aires Cairo Frankfurt_____________________________________________________________________________
TOEFL-CBT Acceptance .60 .65 .68
Acceptance of Admissions Tests .58 .50 .55
Computer Liking .69 .50 .66
Computer Confidence .68 .64 .74
Worry .50 .36 .46
Tension .64 .49 .70
Test-Irrelevant Thinking .71 .65 .75
Bodily Symptoms .50 .50 .54
Computer Familiarity .36 .69 .60______________________________________________________________________________
Note. The Ns appear in Table 5.
Table 8
Correlations of TOEFL-CBT Acceptance, Acceptance of Admissions Tests, and TOEFL Scores with Other Questionnaire and Background Variables
in Buenos Aires
_______________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
TOEFL-CBT Acceptance - .35** .14 .19* .11 .16*
Acceptance of Admissions Tests .35** - -.18* -.12 -.17* -.18*
Computer Liking .16* -.08 .22** .18* .11 .19*
Computer Confidence .13 -.14 .30** .30** .23** .31**
Total Computer Attitude .16* -.12 .29** .27** .20* .28**
Worry .02 -.05 -.07 -.09 -.03 -.07
Tension -.03 -.03 -.08 -.07 -.03 -.07
Test-Irrelevant Thinking -.04 -.01 -.11 -.02 -.05 -.07
Bodily Symptoms -.03 -.02 -.18* -.26** -.22** -.24**
Total Test Anxiety Scale -.03 -.04 -.18* -.18* -.13 -.18*
Computer Familiarity .06 -.09 .16 .22** .10 .18*
Table 8 (continued)
________________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
Preparation for TOEFL -.14 .03 -.09 -.04 -.12 -.10
Admissions Tests Taken -.08 .07 -.04 -.09 .05 -.02
Age -.02 -.05 -.13 .01 -.18* -.12
Sex -.07 -.06 -.13 -.20* -.06 -.14
Native Country .11 .07 -.07 -.13 -.10 -.11
Native Language .06 .00 -.02 .07 -.07 -.01
Planning to Study in United States or Canada -.09 .12 -.07 -.10 -.03 -.07
Reason for Taking TOEFL-Undergraduate -.09 -.03 -.08 -.25** -.07 -.14
Reason for Taking TOEFL-Graduate Student .04 -.11 .16 .37** .19* .26**
Reason for Taking TOEFL-Other .03 -.14 -.10 -.20* -.15 -.16*
Table 8 (continued)
________________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
TOEFL Listening Score .14 -.18* - .68** .70** .90**
TOEFL Structure/Writing Score .19* -.12 .68** - .72** .88**
TOEFL Reading Score .11 -.17* .70** .72** - .91**
TOEFL Total Score .16* -.18* .90** .88** .91** -________________________________________________________________________________________________________________________
Note. Ns range from 118 to 144.
* p < .05 (two-tail); ** p < .01 (two-tail).
Table 9
Correlations of TOEFL-CBT Acceptance, Acceptance of Admissions Tests, and TOEFL Scores with Other Questionnaire and Background Variables
in Cairo
_______________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
TOEFL-CBT Acceptance - .25** .40** .32** .17* .34**
Acceptance of Admissions Tests .25** - -.02 -.11 -.20* -.12
Computer Liking .21** .10 .13 .13 .00 .10
Computer Confidence .11 .15 .12 .17* -.04 .09
Total Computer Attitude .18* .15 .14 .18* -.03 .11
Worry -.05 -.05 .10 -.07 -.01 .02
Tension -.09 -.16 -.01 .00 .05 .01
Test-Irrelevant Thinking -.20* .05 -.06 -.21** -.08 -.13
Bodily Symptoms -.13 .09 -.19* -.20* -.11 -.19**
Total Test Anxiety Scale -.18* -.03 -.07 -.18* -.05 -.11
Computer Familiarity .16 .12 .14 .05 -.09 .05
Table 9 (continued)
________________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
Preparation for TOEFL -.21* .05 -.36** -.17 -.30** -.33**
Admissions Tests Taken -.13 -.04 .01 -.14 .04 -.03
Age -.08 .00 -.22** .13 -.08 -.08
Sex -.02 .03 .01 -.10 .15 .02
Native Country -.10 -.13 -.19* -.02 -.09 -.13
Native Language -.02 -.14 .02 .15 .03 .07
Planning to Study in United States or Canada .09 -.08 .09 .00 .10 .08
Reason for Taking TOEFL--Undergraduate .14 .08 .09 -.15 .08 .05
Reason for Taking TOEFL--Graduate Student -.10 -.14 .16 .01 -.16 -.12
Reason for Taking TOEFL--Other .03 .08 -.16 .13 .10 .09
Table 9 (continued)
________________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
TOEFL Listening Score .40** -.01 - .67** .59** .88**
TOEFL Structure/Writing Score .32** -.11 .67** - .67** .88**
TOEFL Reading Score .17* -.20* .59** .67** - .86**
TOEFL Total Score .34** -.12 .88** .88** .86** -________________________________________________________________________________________________________________________
Note. Ns range from 107 to 142.
* p < .05 (two-tail); ** p < .01 (two-tail).
Table 10
Correlations of TOEFL-CBT Acceptance, Acceptance of Admissions Tests, and TOEFL Scores with Other Questionnaire and Background Variables
in Frankfurt
_______________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
TOEFL-CBT Acceptance - .39** .21** .25** .15** .23**
Acceptance of Admissions Tests .39** - -.07 -.09 -.09 -.09
Computer Liking .14** .11* .14** .13** .07 .13**
Computer Confidence .18** .06 .23** .19** .15** .21**
Total Computer Attitude .17** .09 .21** .18** .13** .19**
Worry -.13** .05 -.11* -.17** -.10* -.14*
Tension -.01 -.02 .08 .02 .11* .08
Test-Irrelevant Thinking -.14** -.01 -.09 -.09 -.07 -.10
Bodily Symptoms -.10 .08 -.12* -.16** -.04 -.12*
Total Test Anxiety Scale -.13** .03 -.07 -.13** -.02 -.08
Computer Familiarity .08 .07 .06 .07 .03 .06
Table 10 (continued)
________________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
Preparation for TOEFL .01 .09 -.19** -.09 -.13** -.15**
Admissions Tests Taken -.14** -.04 .06 -.03 .03 -.02
Age .15** .04 -.03 .10* -.09 -.01
Sex -.14** -.10* -.07 -.08 .04 -.04
Native Country -.08 -.09 .25** .17** .19** .23**
Native Language -.07 -.07 .26** .16** .18** .23**
Planning to Study in United States or Canada .00 .05 -.02 .02 -.06 -.02
Reason for Taking TOEFL--Undergraduate -.15** -.04 -.13** -.20** -.10* -.16**
Reason for Taking TOEFL--Graduate Student .13** -.02 .17** .21** -.14** .20**
Reason for Taking TOEFL--Other .01 .08 -.08 -.03 -.07 -.07
Table 10 (continued)
________________________________________________________________________________________________________________________
TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total
Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________
TOEFL Listening Score .21** -.07 - .66** .64** .87**
TOEFL Structure/Writing Score .25** -.09 .64** - .64** .88**
TOEFL Reading Score .15** -.09 .66** .64** - .88**
TOEFL Total Score .23** -.09 .87** .88** .88** -________________________________________________________________________________________________________________________
Note. Ns range from 357 to 403.
* p < .05 (two-tail); ** p < .01 (two-tail).
Appendix