Kember-Development of a questionnaire for assessing students’ perceptions of the teaching and learning environment and its use in quality assurance

7/27/2019 Kember-Development of a questionnaire for assessing students’ perceptions of the teaching and learning environ…

http://slidepdf.com/reader/full/kember-development-of-a-questionnaire-for-assessing-students-perceptions 1/15



While evaluation at the level of the instructor or course must be almost universal,

evaluating the teaching involved in degrees, programs, majors or departments is far less

prevalent and has received a great deal less attention in the literature. This situation is

somewhat surprising in that the curriculum for a degree is more than the sum of the

teaching in the constituent courses. Universities have procedures for approving the plansfor overall degrees and list degree awards in their calendars. Surely, then, the entity itself

needs evaluation at least as much as the parts which make it up.

If evaluation is confined to individual courses, it could be argued that the important

teaching and learning environment becomes neglected. Fraser (1998) has conceptualised a

learning environment as:

‘Learning environment’ refers to the social, psychological and pedagogical contexts

in which learning occurs and which affect student achievement and attitudes. (Fraser

1998, p. 3)

Standard course evaluation questionnaires do not normally encompass this broad con-

struct. Indeed many have been criticised for being based on teacher-centred models of

education (Centra 1993; d’Appollonia and Abrami 1997; Kolitch and Dean 1999;

McKeachie 1997). Yet the teaching and learning environment has been shown to be

important in the development of the generic competencies that are so vital to today’s

graduates (Kember and Leung 2005a; Leung and Kember 2006).

One reasonably common form of evaluation of a learning environment involves the

focus on the student experience. Tinto (1975, 1987) theorised that integrating students into

the social and academic fabric of college life was an important determinant of student

persistence. Consequently, surveys have been developed to examine the degree to whichstudents enjoy a range of campus experiences of the type which might foster academic and

social integration.

The most commonly used survey of this type is probably the College Student Experience

Questionnaire (Pace and Kuh 2007) which has been widely administered (Kuh 1999; Kuh

et al. 2000). The instrument asks about the degree to which students have experienced a

wide range of activities, such as using the library, interaction with staff, cultural activities,

making use of campus facilities, attending societies and making student acquaintances.

Surveys of this type are usually administered at the college or campus level. Programs

to promote integration, such as first-year orientation, are commonly run by school or

college offices. Many of the experiences surveyed are not program-specific. While theinstruments have proved valuable for the purpose for which they were designed, they

would not give useful feedback to specific departments for evaluation at the degree or

program level.

The Course Experience Questionnaire (CEQ; Ramsden 1991) appears to be the most

widely used instrument designed to assess teaching effectiveness at the level of the whole

degree or program in higher education institutes. The CEQ has been employed in the

annual national graduate survey in Australian higher education since 1993 and a similar

national survey in the UK is currently being developed.

The CEQ was primarily designed as a performance indicator for teaching effectivenesswhich assesses a range of student perceptions related to teaching and learning. The older

version of the CEQ consisted of 30 items with the five scales of Good Teaching, Generic

Skill, Clear Goal and Standard, Appropriate Workload and Appropriate Assessment,

together with an item for Overall Satisfaction. In an extensive study by Wilson and Lizzio

(1997) with three large samples of Australian graduates, a shortened version of the CEQ

with 23 items measuring the same six dimensions was developed. The 23-item CEQ was

16 Learning Environ Res (2009) 12:15–29

123



shown to have moderate to high reliability for the five scales, as well as acceptable

construct validity. The CEQ also produced expected correlations with student’s learning

outcomes which established its external validity.

The obvious limitation of the CEQ is that it consists of just five scales and an item for

overall satisfaction. This implies that the teaching and learning environment of a degreecan be characterised by just five factors, or that feedback to departments would be

restricted to five variables. This is possibly why the CEQ has been used to produce ranking

lists, because such a restricted number of scales can give little diagnostic feedback capable

of identifying strengths or weaknesses which can be remedied.

The CEQ has one overall satisfaction item and one scale for all generic skills, which

leaves just four scales for teaching and learning. Even at the instructor level, teaching is

recognised as a multidimensional skill (Marsh 1987) and well-designed teaching evalua-

tion instruments have multiple scales (see Marsh 1987 for examples). Therefore, the CEQ

must be limited in its ability to adequately characterise even the pedagogical context of a

learning environment. It is not clear that it makes any provision at all for the social and

psychological contexts included in Fraser’s (1998) definition. A further limitation is the

inclusion of all generic skills within one scale, which implies that all are of the same

nature.

Development of a teaching and learning environment questionnaire

The university at which we previously worked recognised the need for an evaluation

instrument which was functional at the degree or program level. The design of this wasguided by two principles.

1. It needed to be sufficiently diagnostic to identify strengths and weaknesses, making it

possible for feedback to lead to an action plan for improvement.

2. The instrument needed to be consistent with research into teaching and learning

environments and recognise the breadth of constructs encompassed within an

environment.

Both principles necessitated the inclusion of a range of scales that represented the

breadth of constructs implicit in a learning environment and the ability to provide diag-

nostic feedback on teaching and curriculum design. A constraint is that students arereluctant to complete questionnaires which they consider overly long. The resultant

questionnaire, therefore, needed to include vital constructs without appearing daunting to

complete.

Generic capabilities

Another design decision was associated with the desire to make the questionnaire out-

comes-based. For it to be used across all disciplines, it could not evaluate discipline-

specific outcomes. However, there is general agreement that a university education should

result in graduates developing a range of generic capabilities, which equip them for life-

long learning.

To determine the generic capabilities to include in the instrument, each of the faculties

was asked to determine the capabilities important for their graduates. The resulting lists

were examined and a set of common capabilities emerged. While the terminology was not

necessarily consistent, there was broad agreement that all graduates needed to develop the

Learning Environ Res (2009) 12:15–29 17

123



capabilities in the list below. As a result, the initial version of the questionnaire contained a

scale seeking perceptions of the development of each of the following generic capabilities:

• Critical Thinking

• Creative Thinking

• Ability to Pursue Lifelong Learning

• Adaptability

• Problem Solving

• Communication Skills

• Interpersonal Skills

• Discipline Knowledge

• Career Relevance.

Learning environment

The remainder of the questionnaire needed to be characteristic of a teaching and learning

environment. The initial version concentrated on variables found in the literature on the

student experience. The two reasons for this were that, firstly, the development of the

questionnaire was part of a project known as Evaluating the Part-Time Student Experience.

Secondly, teaching was already heavily evaluated through regular course and teaching

evaluation. Accordingly, there were just four scales specific to teaching:

• Active Learning

• Teaching for Understanding

• Teacher–Student Interaction• Assistance from Teaching Staff.

Field-testing of initial version

The initial version of the questionnaire was used as a graduate survey. It was administered

to all those who graduated from taught courses from a university in Hong Kong in 2001.

The university offered discrete programs for full- and part-time students.

Some rather surprising results were obtained, which have been reported in detail(Kember and Leung 2005a). The part-time students perceived higher levels of capability

development, than full-time students, on all but one of the nine generic capabilities scales.

The result was unexpected in that, while the literature was not definitive about the

mechanism for capability development, there was speculation that generic capabilities are

nurtured through immersion in a stimulating campus environment. However, part-time

students spent markedly less time on campus than their full-time counterparts.

Having eliminated a number of possible explanations, only one seemed to remain. The

principal mechanism for capability development actually was through active learning

experiences which required students to practice the capability in question. The part-time

courses were mostly taught through 3-h evening classes. Even the most teacher-centred

lecturers tended to eschew purely didactic teaching to keep students awake. The part-time

students, therefore, experienced a more interactive teaching and learning environment than

their full-time counterparts.

As a test of the initial version of the questionnaire, the generic capabilities part had

proved successful. The scales for the capabilities had acceptable psychometric properties


123



(Kember and Leung 2005a) and gave useful feedback on their development, which were

accepted as important learning outcomes across the wide range of degrees. The results

seemed to suggest that there needed to be greater emphasis on aspects of teaching and

learning. To avoid lengthening the instrument, new scales could replace ones concerned

with the campus environment, which had not correlated strongly with capabilitydevelopment.

Final version

Subsequent development of the questionnaire took place at another university in Hong

Kong, in conjunction with a quality assurance initiative known as the Student Engagement

Project. The project aims to give feedback to departments about their undergraduate

degrees in alternate years. First- and third-year students are sampled. The students have the

option of completing the questionnaire on paper or through the internet, and English and

Chinese versions are available.

There have been two versions of the instrument. The one used for the first 2 years was

modified in the light of feedback. Only the final version is described here. The instrument

still retains the same two parts included in the original version.

1. A section seeking feedback on students’ perceptions of the development, during their

degree, of a set of generic capabilities.

2. A section seeking feedback on perceptions of the quality of elements of the teaching

and learning environment.

The set of capabilities has not changed greatly from that in the original version. The

minor changes resulted from the need to gain approval for the instrument in a compre-

hensive university. There now seems to be virtually universal acceptance of the need for

graduates to develop generic competencies and the set listed below has proved acceptable

across the university. There are differences between programs as to the relative importance

of the competencies but, as the feedback is considered indicative, it can be interpreted with

these weightings in mind. The capabilities included in the final version are:

•

Critical Thinking• Creative Thinking

• Self-managed Learning

• Adaptability

• Problem Solving

• Communication Skills

• Interpersonal Skills and Groupwork

• Computer Literacy.

The elements included in the teaching and learning environment have been adapted to

include those shown to have a major impact on the development of capabilities (Kember

and Leung 2005a, b; Leung and Kember 2005a, b, 2006) and others which give useful

feedback to departments about teaching and curriculum design. By including this range of

scales, the questionnaire encompasses a wide range of facets of a teaching and learning

environment, so as to make it consistent with Fraser’s (1998) holistic definition. The scales

now included are:


123



• Active Learning

• Teaching for Understanding

• Feedback to Assist Learning

• Assessment

• Relationship between Teachers and Students• Workload

• Relationship with Other Students

• Cooperative Learning

• Coherence of Curriculum.

A listing of scales and their items is given in the Appendix. All but one of the scales has

a minimum of two items. The remaining scale has three items. This was a deliberate design

decision. It permits a range of constructs to be included in the instrument; so reflecting the

breadth of the construct of a teaching and learning environment. The shortness of the

scales, though, keeps the questionnaire to a reasonable length. The paper version fits ontwo A4 sides. The trade-off with short scales is the difficulty of constructing reliable scales

which adequately represent the construct. Evidence of satisfactory reliability and validity

of the scales is given in the following sections.

The questionnaire uses a five-point Likert scale with responses ranging from Strongly

Agree to Strongly Disagree. The instrument also includes two open-ended questions:

• What are the best aspects of your program?

• Which aspects are most in need of improvement?

Reliability

Data for the tests of reliability and validity were obtained from a sample of 2,879 students

from the 21 programs surveyed in 2005. The return rate was 53.2% and 1,530 useable

questionnaires were available for analysis.

The reliabilities of the scales in the questionnaire were checked using Cronbach

a values (Table 1).

Validity

Validity of scales or instruments is harder to establish than reliability. The procedure

involved in deriving the set of capabilities to be included in the original instrument pro-

vided a measure of construct validity in establishing them as a viable set of generic

attributes which graduates could be expected to have developed. Further confirmation of

construct validity comes from the observation that a similar set of attributes was adopted

by a second university which is a comprehensive research-intensive university, whereas the

original was a former polytechnic concentrating on applied professional courses.

The content validity of the items in all scales, as measures of the intended constructs,

was ensured by their examination by expert panels. The formulations were then checked

through focus-group discussions with small groups of typical students. The construct

validity of the instrument was also established through a series of psychometric tests

(Leung 2005).

There are two strands of evidence that indicate that the scales included in the teaching

and learning environment part of the instrument are a valid characterisation of an


123



environment. Firstly, the results from the surveys have been found to provide readily

interpretable feedback for diagnosing strengths and weaknesses, as discussed in the next

main section. Because evidence from all of the scales has resulted in recommendations for

curriculum revisions, they must assess relevant elements of a teaching environment.

Model of capability development

A further demonstration of construct validity comes from using structural equation mod-

elling (SEM) to establish that the elements included in the teaching and learningenvironment make a significant contribution to nurturing the set of generic capabilities

included in the questionnaire. Anderson (1987) suggested that external validity of a con-

struct can be established through examining the set of relationships of the construct with

other constructs in a nomological network.

SEM has the ability to test hypothesised models of the causal relationship between

measured variables. In this case, the hypothesised model is of the teaching and learning

environment influencing the development of a set of generic capabilities. The model is

shown in Fig. 1. The diagram of the model includes the standardised coefficients obtained

from the testing of the model. Although these are not part of the hypothesised model,

showing them avoids duplication of the diagram.The scales from the questionnaire appear in the model as indicators, which are repre-

sented in SEM models as rectangles. It will be noted that two scales from the questionnaire

do not appear as variables in the model. Because computer literacy is somewhat different

in nature to the other generic capabilities, it did not correlate well with them. Students’

perceptions of workload relate only weakly to hours worked but are influenced by their

Table 1 Cronbach a coefficient for scales in the questionnaire

Scale No. of items a

Capability

Critical thinking 2 0.79Creative thinking 2 0.58

Self-managed learning 2 0.73

Adaptability 2 0.65

Problem solving 2 0.73

Communication skills 2 0.64

Interpersonal skills and groupwork 2 0.64

Computer literacy 2 0.70

Teaching and learning environment

Active learning 2 0.65Teaching for understanding 2 0.77

Feedback to assist learning 2 0.68

Assessment 3 0.71

Relationship between teachers and students 2 0.70

Workload 2 0.82

Relationship with other students 2 0.65

Cooperative learning 2 0.74

Coherence of curriculum 2 0.79


123



perceptions of other elements of their learning environment (Kember 2004). Therefore,

workload can be modelled as being weakly influenced by most of the variables in this

characterisation of a teaching and learning environment (Kember and Leung 2006), which

makes its inclusion in the model problematic.

In addition to the variables corresponding to the questionnaire scales, the model

includes five latent variables shown in ovals. On the teaching and learning environment

side of the model, the three latent variables of teaching, teacher–student relationship, and

student–student relationship can be envisaged as higher-order factors that group distinct

facets of the environment. Similarly, on the capabilities side of the model, the two latent

variables are intellectual and working together. Again, these represent logical groupings of the capabilities.

SEM models are tested by examining the goodness of fit of the model against the data

collected to measure the variables included in the model. Assessment of model fit was

based on multiple criteria including both absolute misfit and relative fit indexes. The

absolute misfit indexes included the Root Mean Square Error of Approximation (RMSEA;

Browne and Cudeck 1993) and the Standardised Root Mean Squared Residual (SRMR;

Bentler 1995). The relative goodness-of-fit index used in the study was the Comparative Fit

Index (CFI; Bentler 1990). A model with RMSEA\ 0.08, SRMR\ 0.08 and CFI[ 0.95

is considered an excellent fit to the data.

The goodness-of-fit indexes for this model were RMSEA = 0.057 (90% CI = 0.052,

0.062), SRMR = 0.042 and CFI = 0.943, which indicates an excellent fit to the observed

data. The standardised parameter estimates of the model are shown in Fig. 1. All of the

paths between the latent variables were statistically significant, except the one from

teaching to working together. Moreover, the directions of all the paths were positive as

anticipated.

Fig. 1 The SEM model showing how the teaching and learning environment influences capability

development. Latent variables are shown as ovals and measured variables are shown as rectangles. All paths

are statistically significant at the 5% level. Error terms are omitted in the figure for simplicity


123



The excellent fit of the model to the data from the questionnaire provides further

evidence of the validity to the questionnaire. Its design is based on a hypothesis that the

teaching and learning environment, as characterised by the variables included in the

questionnaire, has a significant impact on the development of the set of generic capabilities

that were selected as being important for graduates. Data collected with the questionnaire,from a large sample of students, shows an excellent fit to the data confirming that the

questionnaire is based upon a valid overall model.

Diagnostic power for quality assurance

It has been fairly common to rely solely on psychometric data to establish the credentials

of questionnaires. Where an instrument is used for quality assurance or enhancement

purposes, though, we feel that its diagnostic power is also important. If an instrument is to

provide help in improving the quality of teaching and learning, it must be able to identify

strengths and weaknesses.

To do this, it must have a factor structure that is representative of the multidimensional

nature of a teaching and learning environment and includes elements that are important for

effective environments. Marsh (1987) has established that the same principle applies to

teaching evaluation questionnaires, which is why well-designed instruments have multiple

factors, based on a model of effective teaching. Unfortunately, there are many poorly

designed instruments, which have just one overall factor and, therefore, are only capable of

ranking teachers rather than providing diagnostic feedback.

Use of the questionnaire in a quality assurance initiative

For 4 years, the results from the surveys have been used as part of a quality enhancement

initiative. Each undergraduate program is surveyed in alternate years. Feedback is supplied

to departments by using a profile of the form shown in Fig. 2. The departments receive a

profile in this form for responses in year 1 and year 3. They are also provided with a

complete set of responses to the two open-ended questions.

The feedback profile includes frequencies and means for each scale for the current andprevious administrations of the instrument. Each scale score for the program is compared

to the overall university mean for all surveyed programs. The comparison is shown

graphically in the column at the right of the feedback sheet. This gives z-scores, which are

the number of standard deviations from the mean. The feedback profile shown in Fig. 2 is

illustrative of the diagnostic power of the instrument. The z-scores indicate that the

department has a generally good record in teaching, with means for all but three scales

being higher than the university mean. The marked variations in the z-scores, though,

indicate that it is possible to identify relative strengths and weaknesses.

Shortly after the feedback profiles are released to departments, an appointment is made

for one of the two professors in the Centre for Learning Enhancement and Research

(CLEAR) to meet with the department chair and other members of the department with

responsibilities for teaching coordination. At this meeting, there is a discussion of feedback

profiles, the listings of qualitative comments and other available feedback. Data are treated

as being indicative, rather than absolute, and the need for intelligent interpretation in the

particular context of the program is recognised. The meetings, therefore, take the form of


123



interactive dialogues. The aims are to identify strengths to be built upon and challenges for

which remedial action plans can be suggested. In the initial stages, reactions to the

initiative varied. In some quarters, there was suspicion because CLEAR was a newly

formed unit and departments were not used to receiving feedback at the program level.

Acceptance has grown as the feedback has been found to be useful. An indication of thelevel of acceptance is that a growing number of departments are seeking additional

information, including annual surveys, additional years besides year 1 and year 3, and

additional competencies specific to disciplines or professions.

Case study

A case study is now presented which shows how the collection of diagnostic feedback

through evaluation can inform and facilitate the enhancement of teaching and learning

quality. The case is the program with the third year feedback profile shown in Fig. 2.

Below is an extract from the meeting notes from the 2003 round of the survey, showing

how some of the recommendations resulted in an improved teaching and learning

environment.

= 35 = 23

3.87 3.63 0.45

4.20 4.10 0.16

4.13 3.91 0.21

3.84 3.87 0.20

3.89 3.41 0.54

3.46 3.35 0.29

3.61 3.69 0.54

2.49 2.62 0.40

=35 =23

3.20 2.92 0.37

3.93 3.63 0.26

3.43 3.47 0.29

3.62 3.31 0.32

3.14 3.38 0.42

3.60 3.54 0.31

2.60 3.04 0.55

2.47 3.04 0.51

3.39 3.51 0.22

3.46 3.20 0.37

Strongly Agree Agree Neutral

Development of capabilities

St rongl y Agree Agree Neutral

0% 25% 50% 75% 100% -2 -1 0 1 2

0% 25% 50% 75% 100% -2 -1 0 1 2

Teaching and learning environment

Fig. 2 Feedback profile for the case-study program


123



Extract from meeting summary 2003

Areas which could need further attention:

• Inconsistency appears in students’ perceptions of low interpersonal and groupwork

skills despite them having been involved in a large number of group projects;

• A perception of a low level of problem solving and career preparation;

• Heavy workload .

Suggestions for ways forward were made during the meeting:

1. The compulsory first year courses, such as X and Y, could be examined to see if

practical applications could be stressed more in order for them to become a

means of developing critical thinking and problem solving ability.

2. The new module which introduces Z with guest speakers sharing their experience in

their profession seems likely to improve problem-solving and career preparation.3. The nature of group projects could be examined to see why they appear to be having

negative effects on student relationships and cooperative learning. Possibly the

introduction of peer assessment of contributions to the group might help.

Comparison of mean scores between 2003 and 2005 enabled the department to monitor

changes. In this case, it can be seen that issues highlighted in the 2003 meeting notes have

been acted upon in the way suggested and that significant improvements have resulted.

This has been noted in the extract from the 2005 meeting notes.

Extract from meeting summary 2005

Compared to the 2003 data, there were significant increases in the ratings for

problem solving for both first and third years. This was attributed to teachers

stressing applications to solve typical problems. This improvement is worth

communicating to the Department for wider application.

There were also major improvements in student–student relationships and

cooperative learning. This was attributed to better handling of group projects and the

wider adoption of the assessment of individual contributions to group projects. It could

be worthwhile to explain to students each time the purpose and anticipated outcomes of

projects as they are a time-consuming, but very worthwhile, form of learning.

The improvements in the identified aspects of teaching and learning are shown

graphically in Fig. 3 which compares z-scores for 2003 and 2005 for the scales corre-

sponding to the issues identified in the 2003 summary. It can be seen that there have been

significant improvements.

Conclusion

There are numerous questionnaires for evaluating teaching and courses, but instruments

suitable for evaluating teaching and learning in programs, degrees, majors or departments

are comparatively scarce. At this level, a major influence on the achievement of learning

outcomes is the teaching and learning environment. An instrument suitable for program-

level quality assurance or enhancement purposes, therefore, needs to provide measures of a

well characterised teaching and learning environment. The questionnaire described in this


123



article has nine scales for assessing a teaching and learning environment. Four are sub-

sumed under a higher-order teaching variable, two are concerned with teacher–student

relationships, and two focus on student–student relationships. The remaining variable is

workload. Together, the resulting conceptualisation of the teaching and learning

environment corresponds closely to the holistic definition by Fraser (1998).

The instrument also measures student perceptions of the development of eight generic

capabilities. These were included as there is wide agreement that they are important

learning outcomes for graduates. Reliabilities of the scales in the instrument were estab-

lished using the Cronbach a coefficient. The validity of construct selection and scale and

item design was initially addressed through consultation with panels with appropriate

expertise. The validity of the whole instrument was then verified by testing a model

inherent in the design of the questionnaire with SEM. There was a very good fit to

collected data for a model of the teaching and learning environment being influential in the

development of the set of generic capabilities.

The instrument also has good diagnostic properties, evidenced by its ability to identify

relative strengths and weaknesses in a program. This is an important characteristic of any

questionnaire which is to be used to guide quality enhancement. The instrument, therefore,is recommended for wider application, as programs are not evaluated as frequently as

teaching or courses, and rarely by instruments designed to be diagnostic of elements which

constitute a teaching and learning environment.

Acknowledgments This research was partially funded by grants from the University Grants Committee of

Hong Kong.

Change in Z -scores for five scales for Year 3 students:

2003 vs 2005

-1.5

-1

-0.5

0

0.5

1

1.5

z - s c o r e

Workload

Cooperativelearning

Relationshipwith otherstudents

Interpersonalskills andgroupwork

Problemsolving

Change in Z -scores for five scales for Year 1 students:

2003 vs 2005

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2003 Year 3 2005 Year 3

2003 Year 1 2005 Year 1

z - s c o r e

Workload

Cooperativelearning

Relationshipwith otherstudents

Interpersonalskills andgroupwork

Problemsolving

Fig. 3 Change in z-scores for

five scales


123





References

Anderson, J. C. (1987). An approach for confirmatory measurement and structural equation modeling of

organizational properties. Management Science, 33, 525–541. doi:10.1287/mnsc.33.4.525.

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107 , 238–246.doi:10.1037/0033-2909.107.2.238.

Bentler, P. M. (1995). EQS structural equations program manual. Encino, CA: Multivariate Software.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long

(Eds.), Testing structural equation models (pp. 136–161). Newbury Park, CA: Sage.

Centra, J. (1993). Reflective faculty evaluation. San Francisco: Jossey Bass.

d’Appollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist,

52, 1198–1208.

Fraser, B. J. (1998). The birth of a new journal: Editor’s introduction. Learning Environments Research,

1, 1–5.

Kember, D. (2004). Interpreting student workload and the factors which shape students’ perceptions of their

workload. Studies in Higher Education, 29, 165–184.

Kember, D., & Leung, D. Y. P. (2005a). The influence of active learning experiences on the development of graduate capabilities. Studies in Higher Education, 30, 157–172.

Kember, D., & Leung, D. Y. P. (2005b). The influence of the teaching and learning environment on the

development of generic capabilities needed for a knowledge-based society. Learning Environments

Research, 8, 245–266.

Kember, D., & Leung, D. Y. P. (2006). Characterising a teaching and learning environment conducive to

making demands on students while not making their workload excessive. Studies in Higher Education,

31, 185–198.

Kolitch, E., & Dean, A. V. (1999). Student ratings of instruction in the USA: Hidden assumptions and

missing conceptions about ‘good’ teaching. Studies in Higher Education, 24, 27–42.

Kuh, G. D. (1999). How are we doing? Tracking the quality of the undergraduate experience, 1960s to the

present. Review of Higher Education, 22(2), 99–119.

Kuh, G. D., Hu, S., & Vesper, N. (2000). ‘‘They shall be known by what they do’’: An activities-basedtypology of college students. Journal of College Student Development, 41(2), 228–244.

Leung, D. Y. P. (2005). Development of a questionnaire for assessing students’ perception of teaching and

learning environment. In T. Hudson, M. Abramson, T. Acland, J. Braham, M. Hill, A. Lines,

M. Mclinden, D. Saunders, & J. Storan (Eds.), Towards a global understanding of lifelong learning:

Making a difference (pp. 284–293). London: Forum for Access and Continuing Education.

Leung, D. Y. P., & Kember, D. (2005a). Comparability of data gathered from evaluation questionnaires on

paper and through the internet. Research in Higher Education, 46 , 571–591.

Assessment

23. The program uses a variety of assessment methods.

24. To do well in assessment in this program you need to have good analytical skills.

25. The assessment tested our understanding of key concepts in this program.

Relationship between teachers and students

26. The communication between teaching staff and students is good.

27. I find teaching staff helpful when asked questions.

Workload

28. I manage to complete the requirements of the program without feeling unduly stressed.

29. The amount of work we are expected to do is quite reasonable.

Relationship with other students

30. I feel a strong sense of belonging to my class group.

31. I frequently work together with others in my classes.

Cooperative learning

32. I have frequently discussed ideas from courses with other students out-of-class.

33. I have found that discussing course material with other students outside classes has helped me toreach a better understanding of the material.

Coherence of curriculum

34. I can see how courses fitted together to make a coherent program of study for my major.

35. The program of study for my major was well integrated.

Appendix continued


123

http://dx.doi.org/10.1287/mnsc.33.4.525

http://dx.doi.org/10.1037/0033-2909.107.2.238

http://dx.doi.org/10.1037/0033-2909.107.2.238

http://dx.doi.org/10.1287/mnsc.33.4.525



Leung, D. Y. P., & Kember, D. (2005b). The influence of the part-time study experience on the development

of generic capabilities. Journal of Further and Higher Education, 29, 91–101.

Leung, D. Y. P., & Kember, D. (2006). The influence of teaching approach and teacher–student interaction

on the development of graduate capabilities. Structural Equation Modeling, 13, 164–186.

Marsh, H. W. (1987). Students’ evaluations of university teaching: Research findings, methodological

issues, and directions for future research. International Journal of Educational Research, 11, 255–388.McKeachie, W. (1997). Student ratings: The validity of use. American Psychologist, 52, 1218–1225.

Pace, C. R., & Kuh, G. D. (2007). The college student experiences questionnaire assessment program.

Retrieved October 29, 2008, from http://cseq.iub.edu/cseq_generalinfo.cfm.

Ramsden, P. (1991). A performance indicator of teaching quality in higher education: The Course Expe-

rience Questionnaire. Studies in Higher Education, 16 , 129–150.

Seldin, P. (1993, July 21). The use and abuse of student ratings of instruction. The Chronicle of Higher

Education, p. A-40.

Tinto, V. (1975). Drop-out from higher education: A theoretical synthesis of recent research. Review of

Educational Research, 45(1), 89–125.

Tinto, V. (1987). Leaving college. Chicago: University of Chicago Press.

Wilson, K. L., & Lizzio, A. (1997). The development, validation and application of the Course Experience

Questionnaire. Studies in Higher Education, 22, 33–53.


13

http://cseq.iub.edu/cseq_generalinfo.cfm

http://cseq.iub.edu/cseq_generalinfo.cfm

Documents

Kember-Development of a questionnaire for assessing students’ perceptions of the teaching and learning environment and its use in quality assurance