Embed Size (px)
Citation preview
This article was downloaded by: [Pennsylvania State University]On: 28 May 2014, At: 05:57Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: MortimerHouse, 37-41 Mortimer Street, London W1T 3JH, UK
Total Quality Management & Business ExcellencePublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/ctqm20
Excellence in Portuguese higher educationinstitutionsMaria João Pires Da Rosa a , Pedro Manuel Saraiva b & Henrique Diz aa DEGEI , University of Aveiro , Campus Universitário de Santiago, Aveiro, Portugalb Department of Chemical Engineering, School of Science and Technology , University ofCoimbra , Coimbra, 3030-290, PortugalPublished online: 25 Aug 2010.
To cite this article: Maria João Pires Da Rosa , Pedro Manuel Saraiva & Henrique Diz (2003) Excellence inPortuguese higher education institutions, Total Quality Management & Business Excellence, 14:2, 189-197, DOI:10.1080/1478336032000051377
To link to this article: http://dx.doi.org/10.1080/1478336032000051377
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose ofthe Content. Any opinions and views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be reliedupon and should be independently verified with primary sources of information. Taylor and Francis shallnot be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and otherliabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
TOTAL QUALITY MANAGEMENT, VOL. 14, NO. 2, 2003, 189–197
Excellence in Portuguese higher educationinstitutions
M J P R1, P M S2 &H D1
1DEGEI—University of Aveiro, Campus Universitario de Santiago, Aveiro, Portugal &2Department of Chemical Engineering, School of Science and Technology, University of Coimbra,3030–290 Coimbra, Portugal
Based upon an empirical study made with the goal of getting a better understanding ofPortuguese higher education institutions—in particular, regarding strategic and quality management,innovation practices and tools being adopted—and on the study of several quality assessment modelsdeveloped both for higher education institutions (CNAVES, 2000a, 2000b, CNE, 2001; CRE,2000) and organizations in general (EFQM, 1999a; Kanji, 1998), we have developed an excellencemodel for the Portuguese higher education institutions, based upon nine criteria (Rosa et al., 2001).It was intended that the relationships between the different model criteria could be empirically tested.Therefore, after having clearly defined each of the model’s nine criteria and the areas to addresswithin each of them, we tried to validate the model underlying the structure through a questionnaireaimed at assessing how each Portuguese higher education institution is dealing with a series ofpractices and factors considered to be critical for its quality improvement. In this paper we will presentthe main results obtained, through a detailed statistical analysis. Structural equation modelling isused to validate the proposed excellence model structure and relationships among its nine underlyingcriteria through PLS (Partial Least Squares) estimation techniques.
Introduction
The Portuguese higher education system is, at present, undergoing periodic assessments ofits quality, namely regarding teaching (CNAVES, 2000a, 2000b) and research (MCT, 2000)activities. Nevertheless, in most cases there is no overall institutional assessment mechanismleading to a careful analysis of each institution as an organizational entity.
We believe that such an analysis, of an institution as a whole, is very important in orderto improve its quality and the quality of each one of its activities. Therefore, we tried initiallyto determine the present management situation in Portuguese higher education institutions,particularly concerning the implementation of strategic management and TQM principles,concepts and tools. The information collected and analysed allowed us to understand whichare the critical success factors for these institutions and how are they facing the abovechallenges. Based upon these findings and the study of several quality assessment modelsdeveloped both for higher education institutions (CNAVES, 2000a, 2000b; CNE, 2001;
Correspondence: Maria Joao Pires da Rosa, DEGEI—University of Aveiro, Campus Universitario de Santiago,3810 Aveiro, Portugal. E-mail: [email protected]
ISSN 1478-3363 print/ISSN 1478-3371 online/03/020189-09 © 2003 Taylor & Francis LtdDOI: 10.1080/1478336032000051377
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
190 M. J. P. DA ROSA ET AL.
CRE, 2000) and organizations in general (EFQM, 1999a; Kanji, 1998; BNQP, 2001), wedid then develop an excellence model for the Portuguese higher education institutions (Rosaet al., 2001).
This model, which will be presented in the next section, is built around nine criteria thatare assumed to be related to each other. It intends to provide a management and assessmenttool for each higher education institution, supporting its self-analysis (including the identifica-tion of strong points and areas for improvement), and simultaneously providing a source forquality improvement opportunities. In order to validate the model and estimate the relation-ships between the criteria, an empirical study was conducted based on a questionnaire sentto all Portuguese higher education institutions. The answers obtained were statistically treatedusing structural equation modelling.
The excellence model for Portuguese higher education institutions
The suggested excellence model (Fig. 1) is built around nine criteria and clearly assumesthat the excellence of an higher education institution will depend primarily upon the processesthat take place within its scope (namely teaching/learning, research and services provided)and the results that, through the processes, it is able to achieve. Actors, resources, and, inmost cases, partnerships, are also important factors in a higher education institution thatwants to assure and/or improve its organizational quality. However, actors, partnerships,resources, processes and results cannot solely represent a school. It also depends on, and ischaracterized by, its internal structure and organization, leadership, policy, strategy andculture, and is influenced quite strongly by other entities and external regulation mechanisms.Efforts made by the institution to assess—periodically and systematically—each one of thenine criteria will lead to its continual and sustained improvement towards excellence.
This model presents a number of differences when compared with other previous models.As opposed to other models used just in the higher education context, this model covers aset of criteria and their relationships in an illustrative graphical perspective, allowing each
Figure 1. A model of excellence for Portuguese higher education institutions.
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
PORTUGUESE HIGHER EDUCATION INSTITUTIONS 191
institution to have a clear idea about the areas under analysis, reflection, assessment andimprovement. Rather than a list of indicators, our model tries to capture the inherentcomplexity of a higher education institution and, simultaneously, presents a coherent pictureof its performance, where different sub-pictures can also be made in a consistent way,according to a particular perspective (teaching, research, services provided); furthermore, itconveys a structure and relationships amongst criteria that can be statistically test forsignificance and provide a semi-quantitative basis for analysis and priority setting.
Our model development included the definition of each one of the criteria and identifica-tion of the areas to address when an institution analyses itself according to each of thecriteria. Each one of the model criteria is divided into sub-criteria, which leads to a betterunderstanding of the points to be studied.
Validation and application of the excellence model
After having conceived our excellence model, we then tried to validate and apply it in thecontext of the Portuguese higher education system. In order to do this, as in other similarapproaches (EFQM, 1999b; Tambi, 2000), a written survey was conducted covering all ofthe Portuguese higher education institutions, leading to a total of 129 valid answers (approxi-mately a 30% response rate).
This survey covered nine sets of questions, corresponding to the model criteria, resultingin a total of 66 questions, which cover practices and factors considered to be fundamentalfor quality improvement and self-evaluation. These measurement items were derived from acomprehensive analysis of quality assessment models that are—and were—applied in thecontext of higher education institutions (CNAVES, 2000a, 2000b; CNE, 2001; CRE, 2000;EFQM, 1999a; Kanji, 1998; BNQP, 2001), as well as from fieldwork conducted that includessome prototypes and a number of in-depth interviews. Therefore the model structure andcriteria were conceived as a result of considerable previous research efforts, aimed atsupporting the choices made and approaches suggested. Thus, our written survey was aimedat validating the suggested underlying structure. Each respondent provided a closed answerto the questions raised, using a scale that ranges from 1 (Very Little) to 10 (Very Much),which enhances the criteria measurement reliability (Kanji & Wallace, 2000). From the datathus obtained, a detailed statistical analysis was performed, including structural equationmodelling (SEM) multivariate techniques in order to validate the model proposed, as will beshown in the forthcoming sections. Such an approach allows one to examine multiplerelationships simultaneously, while also incorporating measurement errors into the estimationprocess (Curkovic et al., 2000). It is a powerful statistical technique that combines ameasurement model (confirmatory factor analysis) and a structural model (regression or pathanalysis) (Garver & Mentzer (1999).
Development of a structural equation model
Our proposed model reflects an underlying structure, which can be graphically represented,and covers relationships between its nine criteria. Therefore, within this framework, wedeveloped a set of causal relationships, as portrayed in the path diagram presented in Fig. 2.
The suggested causal relationships result both from theoretical assumptions and empiricalresults obtained in our previous study. External regulation is still a factor that exerts a stronginfluence over the kind of leadership institutions have, as well as over the policy, strategy andculture they adopt for themselves, or about their own structure and organization. Externalregulation also influences the kind of partnerships that are more suiTable for a higher
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
192 M. J. P. DA ROSA ET AL.
Figure 2. Path diagram for relationships between the nine criteria.
education institution and defines the roles and careers of its main actors. Additionally, in thecase of public institutions, most financial resources are allocated by government.
As concerns other causal relationships established, we assumed that leadership, policy,strategy and culture, and the structure and organization work as an internal catalyst insideeach institution and are responsible for its global management, acting upon each one of itsprocesses and contributing significantly to improvement. This catalyst also has direct influenceover the institution actors, partnerships, and resources management.
The other causal relationships illustrated in the path diagram result directly from ourmodel underlying structure, based upon processes as a key component, as is also the casewith the EFQM Model of Excellence (EFQM, 1999a): processes are driven by actors thatintervene upon them, and supported by financial, physical and material resources, partner-ships, etc., leading to the desired results. Connections between actors/resources and actors/partnerships derive from the fact that collaborators (actors), through their efforts anddedications to the school, they contribute desisively for the capabilities of a gives highereducation institution to get additional resources (e.g. research contracts, private funding,patents, etc.) and establish significant partnerships (with companies, often schools orlaboratories).
From Fig. 2, it is possible to see that our structural equation model does contain ninelatent variables, or constructs—’external regulation’ being the only exogenous one. Sincelatent variables are not directly measured (Kanji, 1998), to estimate the strength of therelationships between these variables it is necessary to operationalize each of the latentvariables in terms of the manifest (directly measured) variables. In our written survey,mentioned in the previous section, each of the 66 questions raised provides results associatedwith a particular manifest variable, and all of these together allow us to operationalize thenine structural equation model latent variables.
Thus, the structural equation model proposed does combine both latent and measure-ment components. It can be stated under a more formal basis as a set of related equations,some of which are related to the causal relationships between constructs (structural equa-tions), and others to the relationships established between each construct and the manifestvariables that operationalize it. Figure 3 presents a schematic representation of the measure-ment model for the latent variable ‘processes’.
The structural parameters (or path coefficients) of the entire structural equation model
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
PORTUGUESE HIGHER EDUCATION INSTITUTIONS 193
Figure 3. Schematic representation of the latent endogenous variable ‘processes’ and the manifest variables thatoperationalize it.
were estimated using partial least squares (PLS), which is a rather robust estimation approachthat deals with ‘multicollinearity, manifest skewness and misspecification of inner and outerstructures in a reasonable robust way’ (Hackl & Westlund, 2000). This is the estimationprocedure adopted by the ECSI (European Customer Satisfaction Index) and ACSI (Amer-ican Customer Satisfaction Index) projects as well as in the estimation of the parameters ofKanji Business Excellence Model (Kanji, 1998).
Structural equation model validation
The 129 answers obtained for our survey were used as input data for validation of our model,including PLS parameter estimation (for both outer and inner path coefficients).
In what concerns the measurement model, an assessment of the quality of the scalesdeveloped was made. According to Garver and Mentzer (1999), content validity refers to‘the degree that the construct is represented by items that cover the domain of meaning forthe construct’. Since the measurement items, as was already mentioned, were derived froma comprehensive analysis of quality assessment models that were and are already applied inthe context of higher education institutions (CNAVES, 2000a; CNAVES, 2000b; CNE,2001; CRE, 2000; EFQMa, 1999; Kanji, 1998; BNQP, 2001), as well as from the results ofour empirical study, content validity derives from such a previous work.
The quality of the scales was also analysed by the careful inspection of construct validityand reliability. In Table 1 we present the main results obtained from the correlation analysisperformed over the manifest variables, as well as the Cronbach coefficient (a) computed forthe nine constructs (criteria) under analysis. According to Van de Ven and Ferry (1979)(cited in Curkovic et al., 2000), Cronbach’s coefficient (a) should be of 0.7 or higher fornarrow constructs, and 0.55 or higher for moderately broad constructs. Looking at the valuespresented in Table 1, it is possible to say that the constructs proposed have an acceptablereliability. In what concerns the discriminant validity of the scales, the two approachessuggested by Ghiselli et al (1981, cited in Flynn and Saladin, 2001) were used, with positiveresults. In fact, the comparison of scale internal consistency values (Cronbach’s alpha values)with average inter-scale correlations indicates that the reliability for each scale is higher thanits correlation with the other scales. Also the analysis of the Table 1 last two columns clearlyshows that the average correlation between the scale and non-scale items is lower thanbetween the scale and scale items, indicating that the items selected for each scale do indeedoperationalize the latent variable they are suppose to address.
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
194 M. J. P. DA ROSA ET AL.
Table 1. Measurement model analysis
Average item correlationsAverage
Cronbach inter-scale Non-scaleCriteria Mean S.D. alpha correlation Scale items items
External regulation 8.12 0.38 0.77 0.45 0.34 0.18Leadership 7.88 0.30 0.83 0.66 0.64 0.42Policy, strategy and culture 7.65 0.50 0.83 0.67 0.59 0.40Structure and organisation 7.35 0.53 0.71 0.59 0.47 0.33Partnerships 7.61 0.40 0.85 0.54 0.74 0.35Actors 7.37 0.72 0.83 0.59 0.48 0.33Resources 6.86 1.15 0.69 0.56 0.34 0.27Processes 6.77 0.54 0.92 0.64 0.63 0.44Results 7.11 0.71 0.87 0.56 0.48 0.30
In what concerns the structural model under analysis, the first results obtained showedthat some of the relationships proposed turned out to be not significant. As such, severalalternative models were analysed, differing in the type and number of the relationshipsconsidered. In Figure 4 we present a simplified version of the structural model initiallyproposed that includes only the relationships between criteria that were found out to bestatistically significant. The values presented are the structural parameters of the model,showing the strength of the relationships between independent and dependent variables ofthe model.
As it is possible to see from Figure 4, all structural parameters have positive values,indicating that causal connections in the model have a symmetrical relationship (Kanji andWallace, 2000). The results show that ‘external regulation’ has a strong causal connectionwith the institution ‘leadership’, ‘structure and organisation’ and ‘policy, strategy and culture’.Those in turn affect ‘resources’, ‘actors’, ‘partnerships’ and ‘processes’. It is worthwhile tonotice that the three criteria that constitute the internal catalyst do not exert the sameinfluence over the other four. Apparently, and according to the data collected, it seems moreplausible to admit that each one of the catalyst criteria do have strong causal connections
Figure 4. Structural equation model with estimated causal connections between criteria and the percentage of variancein each of the endogenous constructs accounted for by the others with which they are related (R2).
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
PORTUGUESE HIGHER EDUCATION INSTITUTIONS 195
with only some of the other four remaining criteria. That does not invalidate that the catalystas a whole is in fact responsible for each institution global management, acting over processesand contributing significantly for improvement. In what concerns ‘processes’, we clearly seethat this criterion is strongly causal connected to ‘results’, as predicted. Finally a remarkmust be made about ‘partnerships’, a criterion that ended up as having a statisticallysignificant direct causal connection with the ‘results’ criteria, which was not initially predicted(it was though that this was an indirect relationship, through the ‘processes’ criteria).
To assess the latent variable model, the percentage of variation in the endogenousconstructs accounted for by the exogenous constructs (R2) was computed for each structuralequation (Baumgartner and Homburg, 1996), leading to the results also presented inFigure 4. As it is possible to see, there are 3 values which are not so good (around 25%), butthe remaining ones contribute to an overall average value of 43% (a reasonable value whencompared to other studies reported in the literature, e.g. Flynn and Saladin (2001); Bartet al. (2001). Given such results and the fact that individual structural parameters aremeaningful and statistically significant, survey results allow us to consider the structuralequation model proposed as a plausible representation for the data.
Based upon the input data and the structural model it is possible to compute a score foreach one of the 9 model criteria, according to a mathematical expression derived by Fornell(1994) (cited in Kanji and Wallace, 2000):
Scoreó;wi, xiñ;wi
(nñ1);wi
î100
where,
n represents the number of manifest variables (questionnaire items) for each criterionwi represents the outer coefficient of each manifest variable (estimated by PLS)xi is the mean for each manifest variable.
Figure 5 presents, in a graphical format, the scores obtained for each one of the nine criteriaincluded in our structural equation model. From these values we can get a general idea aboutthe criteria where Portuguese higher education institutions have better performance, as wellas the ones where further improvement seems to be needed. One can see that overallinstitutions did get quite good scores for all the criteria, the lowest value corresponding to‘processes’.
Conclusions
The need to have a holistic model, capable of serving as a quality tool for self-assessmentand continuous improvement of Portuguese higher education institutions, resulted in thedevelopment of an excellence model tailored to these institutions (Rosa et al., 2001).
In order to validate its underlying structure and relationships, a structural equationmodel was proposed, with its parameters estimated using PLS. A first model included almostall possible relationships between the nine criteria, but the survey data collected revealed thatapproximately half of the criteria turned out not to be statistically significant. Therefore,alternative models were tested, resulting in a final version with 15 relationships, all statisticallyand theoretically significant. This reduced model consists of a system of equations thatdescribe existing relationships amongst the nine criteria.
We believe that such a model could be applied by higher education institutions as a self-assessment tool to support continuous quality improvement.
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
196 M. J. P. DA ROSA ET AL.
Figure 5. Scores obtained for each one of the nine criteria regarding Portuguese higher education institutions.
At present, we are applying the developed structural equation model to a particularhigher education institution, in order to check if the same kind of results and statisticalsignificant relationships between criteria are obtained.
Acknowledgements
Maria Joao Pires da Rosa would like to acknowledge financial support provided by thePRAXIS XXI programme, of FCT (Portuguese Foundation for Science and Technology).In addition, the authors would like to acknowledge Professor Gopal Kanji, William Wallaceand Patricia Moura Sa for providing us with PLS software and for all the help provided indata treatment.
References
B, C., B, N. & T, S. (2001) A model of the impact of mission statements on firm performance,Management Decisions, 39(1), pp. 19–35.
B, H. & H, C. (1996) Applications of structural equation modelling in marketing andconsumer research: a review, International Journal of Research in Marketing, 13, pp. 139–161.
BNQP—B N Q P (2001) Education Criteria for Performance Excellence, NIST.CNAVES, Conselho Nacional de Avaliacao do Ensino Superior (2000a) Processo de Avaliacao do Ensino
Universitario—Guiao de Auto-Avaliacao.CNAVES, Conselho Nacional de Avaliacao do Ensino Superior (2000b) Processo de Avaliacao do Ensino
Politecnico—Guiao de Auto-Avaliacao.CNE, Comite National d’Evaluation (2001) Guide de l’Evaluation des Universites (Paris, Autorite Administrative
Independante).CRE, Association of European Universities (2000) CRE Institutional Review Program: Guidelines for Self-
Evaluation and Preparation for Review Visits.C, S., M, S., C, R. & H, R. (2000) Validating the Malcolm Baldrige National
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
PORTUGUESE HIGHER EDUCATION INSTITUTIONS 197
Quality Award framework through structural equation modelling, International Journal of ProductionResearch, 38(4), pp. 765–791.
EFQM—European Foundation for Quality Management (1999a) The EFQM Excellence Model 1999—Publicand Voluntary Sector (EFQM).
EFQM—European Foundation for Quality Management (1999a) Assessing for Excellence—A Practical Guide forSelf-Assessment (EFQM).
F, B. & S, B. (2001) Further evidence on the validity of the theoretical models underlying theBaldrige criteria, Journal of Operations Management, 19, pp. 617–652.
G, M.S. & M, J. T. (1999) Logistics research methods: employing structural equation modellingto test for construct validity, Journal of Business Logistics, 20(1), pp. 33–57.
H, P. & W, A. (2000) Customer satisfaction measurement: PLS and alternative estimationmethods, communication presented in the 44th EOQ Congress, Budapest.
MCT, Ministerio da Ciencia e Tecnologia (2000) Descricao Sumaria do Modelo de Financiamento Plurianual ede Avaliacao das Unidades de Investigacao, document available at http://www.fct.mct.pt/unidades/sumaria/index2.htm.
K, G. (1998) Measurement of business excellence, Total Quality Management, 9(7), pp. 633–644.K, G. & W, W. (2000) Business excellence through customer satisfaction, Total Quality Management,
11(7), pp. 979–998.R, M., S, P. & D, H. (2001) The development of an excellence model for Portuguese higher
education institutions, Total Quality Management, 12(7&8), pp. 1010–1017.T, A. (2000) Total quality management in higher education—modelling critical success factors. PhD
Thesis, Sheffield Hallam University.
Dow
nloa
ded
by [
Penn
sylv
ania
Sta
te U
nive
rsity
] at
05:
57 2
8 M
ay 2
014
