Original Articlehttp://mjiri.iums.ac.ir Medical Journal of the Islamic Republic of Iran (MJIRI)

Iran University of Medical Sciences

_______________________________________________________________________________________________________1. Associate Professor of Community Medicine, Center of Educational Research Medical Sciences (CERMS), Iran University of Medical Sci-ences, Tehran, Iran. jkuhpayeh@yahoo.com2. (Corresponding author) MSc in Medical Educating, Medical Education Department , Virtual School of Medicine , Tehran University ofMedical Sciences, Tehran, Iran. Mha325@yahoo.com3. Assistant Professor, English Language Department, School of Health Management and Information Sciences, Iran University of MedicalSciences, Tehran, Iran. afsanehdehnad@gmail.com4. Professor of Internal Medicine, Department of Medical Education, Faculty of Medicine, Iran University of Medical Sciences, Tehran, Iran.Soltarab34@gmail.com5. Assistant Professor, Department of Medical Educational, Faculty of Medicine, Iran University of Medical Sciences, Tehran, Iran.Shb555@yahoo.com6. MD, MPH. Department of Health, Semnan University of Medical Sciences, Semnan, Iran. Soyh_325@yahoo.com

Validity and reliability of activities coaching contextquestionnaire

Jalil Koohpayehzadeh1, Maryam Haji Ahmadi2, Afsaneh Dehnad3, Seyed kamranSoltani Arabshahi4, Shoaleh Bigdeli5, Sohrab Yadollahi6

Received: 19 Jan 2013 Accepted: 17 Feb 2014 Published: 7 June 2014

AbstractBackground: Learning in a clinical environment is an inseparable part of a training program in medical educa-

tion. To evaluate the quality of training in a clinical environment, a comprehensive questionnaire which is ad-justed for local purposes is essential. This study was conducted to determine the validity and reliability of Activ-ities Coaching Context (ACC)-questionnaire from the perspective of residents and students in a clinical learningenvironment.

Methods: This was a cross-sectional study conducted with 65 residents and students of Semnan University ofMedical Sciences. The Content Validity of the questionnaire was confirmed by the judgment of a panel of tenexperts in medical education. The construct validity of the questionnaire was determined by factor analysis. Da-ta were collected and analyzed by SPSS16.

Results: The ratio and index of content validity calculated by experts’ view were quite acceptable (0.31 and0.7, respectively). Construct validity was examined by factor analysis which confirmed seven first order factors.Cronbach’s Alfa coefficient revealed a high degree of internal consistency (0.932). The reliability of the ques-tionnaire was measured by test - retest (0.9).

Conclusions: The ACC questionnaire appeared to be a valid, reliable and also appropriate tool to evaluate theeducational quality of a clinical learning environment.Keywords: Validity, Reliability, Questionnaires, Clinical, Learning.

Cite this article as: Koohpayehzadeh J, Haji Ahmadi M, Dehnad A, Soltani Arabshahi S.K, Bigdeli Sh, Yadollahi S. Validity and Relia-bility of Activities Coaching Context Questionnaire. Med J Islam Repub Iran 2014 (7 June). Vol. 28:41.

IntroductionTraining expert manpower, extending and

promoting knowledge, developing researchand paving the ground for the developmentof a country are the main elements includedin the mission of each university endeavor-ing to have its mission completed. In thisregard, universities should be able to identi-fy problems, provide a plan and administerthis plan appropriately. In the same line,

medical education as a part of higher edu-cation plays an important role in protectingthe public health, thereby showing specialconcerns about the qualitative and quantita-tive aspects of medical education (1,2).Evaluation of learning environment withthe purpose of quality promotion has al-ways been emphasized by curriculum de-signers; however, a tool is needed to pro-vide comprehensive information about dif-

Validity and reliability of activities coaching context questionnaire

2 MJIRI, Vol. 28.41. 7 June 2014http://mjiri.iums.ac.ir

ferent aspects of clinical leaning environ-ment having complex and multi-dimensional characteristics (3). There are anumber of questionnaires designed for thispurpose. Surgical Theatre Educational En-vironmental Measure (STEEM) (4), Anes-thetic Theatre Educational EnvironmentMeasure (ATEEM) (5), postgraduate hospi-tal educational environment measure(PHEEM) (6), Dundee Ready EducationEnvironment Measure (DREEM) (7) andActivities Coaching Context (ACC) aresome of them.

Among these questionnaires, ACC devel-oped by Dectler et al (2011) is related toclinical training activities. It consists of 44questions in seven sections including“Clinical Learning Experiences" (CLE),“Self – Directed Learning (SDL)", “Coach-ing and Teaching" (CT), “Welcoming Pro-cess" (WP), “Work organization " (WO) ,“Multi Profession Collaboration (MPC)"and Climate (C). In addition to these sec-tions, it holds three sections related to in-struction: “Coaching and Teaching”, “Ac-tivity” and “Cooperative Activity”. Alt-hough ACC seems to be an appropriate toolfor quantitative and qualitative evaluationof a clinical learning environment, its psy-chometric characteristics should be con-firmed before applying it. The presentstudy is unique in this regard as no researchstudy has yet evaluated these characteristicneither in Iran nor in other countries (8).

This study was conducted to determinethe psychometric characteristics of ACCquestionnaire and to present a standard toolto evaluate the clinical learning environ-ment in Semnan University of Medical Sci-ences from the perspective of residents andinterns.

MethodsThis was a cross-sectional study in which

the data was collected via census. The par-ticipants of the study were all the obstetric,internal and pediatric residents (N= 42),and obstetric, pediatric and surgery clerk-ships (N= 34) working in the main clinicalwards of teaching and medical centers in

Semnan University of Medical Sciences.Data was collected by ACC questionnaire

developed by Dectlar (2011) whose per-mission for using his questionnaire in thesetting of Iran was acquired through anemail. . The reliability and validity of thisquestionnaire were initially determined byusing it in 150 wards of 160 affiliated hos-pitals in Louver University in Belgium. Thequestionnaire was examined with twogroups of medical students who were in theyear 6 of the program (N1= 531, N2= 402).The purpose of developing this instrumentwas to explicate the quality of learning inclinical wards, as a learning environment,by using a systematic and unique method.

For the purpose of the present study,ACC questionnaire was first translatedfrom English into Persian through a stand-ard back translation procedure. The ques-tionnaire was then reviewed for clarity andsimplicity and no major differences werefound in the meaning of the concepts. Inthe next step, the Persian version of thequestionnaire was sent to 12 medical edu-cators to determine Content Validity Ratio(CVR) and Content Validity Index (CVI).Two options of “suitable” and “not suita-ble” were given to have the reviewers’comments. After collecting the comments,no item was omitted, and there were someminor changes in the items to localize thequestionnaire for clinical setting of Iran asfollows:

“The survey of some pathology caseswas on my duty.”

“My work program enables me to dowhat I expect to do .e.g; I spend one day inthe lab, radiology, central pharmacy and inthe ambulance.”

"Seminars held, was at a high scientificlevel, they were understandable, rich andfull of encouragement.″

“The supervisor informed me, when aninteresting event was about to happen in thefield of pathology and clinical examina-tion.”

″The supervisor gave me a useful feed-back for my files."

"At the end of the period, I had a feed-

J. Koohpayehzadeh, et al.

3MJIRI, Vol. 28.41. 7 June 2014 http://mjiri.iums.ac.ir

back dialogue with the supervisor"."As the hospital had defined tasks, I knew

what I was supposed to do".″There was a balance between consulta-

tion, hardworking, working in operationtheatre (the necessary times), delivery roomand management in the weekly program ".

"There was always a person to answer myquestions or resolve my problems. ″

'During this period, I encountered varioussyndromes and I had enough time to be fa-miliarized with the symptoms of diseases".

"I had enough reference studies."“I feel, I was well acknowledged due to

my activities in the clinical ward ".Consistency of the content and face valid-

ity of the questionnaire were confirmed byfive experts and a test – retest method, withan interval of 10 days, was performed toexamine the reliability of the instrument.The questionnaire was piloted with 10 stu-dents who were randomly selected to de-termine the internal consistency of thequestionnaire. The Kolmogorov–Smirnovwas used to test the normality of the data.The collected data were analyzed by SPSSversion 16 using descriptive statistics pre-sented by central statistics index (mean,variance, standard deviation, and graph).

The ACC questionnaire consisted of twoparts. The first part was for collecting de-mographic information (age, sex, educa-tional level, clinical ward). The second partconsisted of 44 items and seven areas relat-ed to the followings: Clinical Learning Experiences (7 items) Self-directed Learning (7) Coaching and Teaching (16)Welcoming Procedure (3)

Work Organization (3) Multi Professional Collaboration (2) Climate (6)Each item was ranked on the basis of a 5-

point Likert scale of “totally agree” (4points), “agree” (3points), “no comments”(2 points), “disagree” (1point) and “totallydisagree” (0 point). The maximum score inthe questionnaire was 176. To have a betterand clearer description, the scores werecategorized into four categories: unfavora-ble (0-44), semi-favorable (45-88), favora-ble (89-132) and very favorable (133-176).

Statistical analysisData analysis was performed using SPSS

software version 16. The required statisticalmethods include confirmatory factor analy-sis methods and Pearson correlation testsand Cronbach's alpha coefficient determi-nation

ResultsFrom 76 respondents, 65 (including 32

residentsand 33 clerkships) completed thequestionnaire (85.5%). The respondentswere both females (72%) and males (28%).The results of the study showed the CVI of0.7 and the mean of 0.31 for CVR. Thesefindings is most probably due to the highmean of experts’ judgment (1.5 for eachquestion) leading to the acceptance of allquestions in the ACC questionnaire.

The results of test-retest and Pearson Cor-relation Coefficient which was done to de-termine the reliability of ACC, showed thatall items were strongly consistent (r =0.9,p=0.02) indicating a highly reliable ques-tionnaire (Table 1).

Internal consistency of the items was de-

Table 1. Test-retest correlation coefficient related to seven - scale domain in the questionnaireScale Pearson ′s correlation co-efficient No. p

Clinical learning experiences 0/95 10 0/012Self-directed learning 0/86 10 0/023Coaching teaching 0/79 10 0/021Welcoming procedure 0/88 10 0/001Work organization 0/92 10 0/04Multi-professional collaboration 0/9 10 0/03Climate 0/93 10 0/04Total score 0/9 10 0/02

Validity and reliability of activities coaching context questionnaire

4 MJIRI, Vol. 28.41. 7 June 2014http://mjiri.iums.ac.ir

termined by Cronbach Alpha (0.932) (Ta-ble 2).

Construct validity was examined via fac-tor analysis. Bartlett Test with an index of2003.958 showed that the sample size wassufficient for factor analysis (df=949,p<0.001) (Table 3).

The Scree graph showed a rapid decreasefrom seven factors reaching to the zero var-iance. Each factor with the Eigen value ofmore than one was known as a factor orscale. Using this method, seven factors withthe greatest role in describing the variancewere then extracted (Fig. 1).

The result of the confirmatory factoranalysis showed the appropriateness of theexploration model. Seven factors were de-termined by values of more than 1.6 andthe variance of 57.5. Rotation was done byVarimax and Normalization methods. Theitems of ACC questionnaire were supportedby the seven – order factors pertaining toClinical Learning Experiences (CLE), Self-Directed Learning (SDL), Welcoming Pro-cedure (WP), Work Organization (WO),Multi-Professional Collaboration (MPC)and Climate (C) (Table 4).

The results of confirmatory factor analy-sis (Table 4) and the correlation coefficientshowed that some of the items belong toother domains. However, moving thoseitems was not conceptually correct and theyremained in the original place.

DiscussionThe primary purpose of this study was to

determine the reliability and validity ofACC among residents and clerkships inSemnan University of Medical Sciences.The results of this study revealed that thequestionnaire was valid and reliable.

To our knowledge, reliability and validityof this tool have not been examined yet inIran and other countries; therefore, datacollected through this study was not com-parable with similar studies.

In evaluating the quality of the clinicaleducational setting, the mean differencebetween males’ and females’ view

Regarding clinical learning experiences,self – learning and the role of supervisorrevealed a significant statistical differencebetween the males’ and females’ view in 2scales of the factors including Activity andRole of Supervisor. These differencesmight be due to the higher population offemales and their intellectual and motivat-ing states of mind in comparison withmales. In the first factor scale, there was a

Table 2. Cronbach’s Alpha coefficient related to the seven-scale domains of the ACC QItems Alpha Cronbach co-efficient standardized Number of questions in each area

Total items 0/932 44Clinical learning experiences 0/791 7Self-directed learning 0/734 7Coaching teaching 0/908 16Welcoming procedure 0/757 3Work organization 0/415 3Multi-professional collaboration 0/714 2Climate 0/809 6

Table 3. KMO and Bartlett's test statistic.376Kaiser-Meyer-Olkin Measure of Sampling

Adequacy.2003.958Bartlett's Test of Approx. Chi-Square

946Sphericity df<0.001p

Graph1. Eigenvalues of the questionnaire

J. Koohpayehzadeh, et al.

5MJIRI, Vol. 28.41. 7 June 2014 http://mjiri.iums.ac.ir

statistical significant difference betweenpeoples’ view about the social environ-ment, clinical learning experiences, andwork organization at pediatric, gynecologi-cal and maternity, internal and surgicalwards. There was also a significant differ-ence between the second factor scale inpeoples’ view in pediatric, gynecological

and maternity, internal and surgical wardsthis seems to be due to the favorable condi-tion of internal wards in comparison withother wards. The high coefficient of occu-pied beds of over 75% could be due to theefficiency of management of internal wardsby capable specialists including respiratory,urology, digestive and liver, blood and on-

Table 4. Results loadings on Factor AnalysisMatrix elements of the spinQuestions

Factors7654321

.66/.72CLE1.72/80CLE2.66/67CLE3.75/74CLE4.88/.88CLE5.70/.72CLE6.61/.63CLE7

.45/.57SDL1

.45/.56SDL2

.58/.55SDL3

.64/.50SDL4

.78/.63SDL5

.96/.57SDL6

.79/.73SDL7.78/.67C&T1.76/.70C&T2.76/.80C&T3.76/.77C&T4.77/.83C&T5.72/.82C&T6.62./60C&T7.66/.66C&T8.72/.55C&T9.67/.70C&T1071/.76C&T11.49/.51C&T12.53/.55C&T13.56/.62C&T1453/.53C&T15.61/.63C&T16

.56/.59WP1

.79/.79WP2

.77/.79WP3.69/.68WO1.66/.63WO2.42/.47WO3

.71/.77MPC1

.80/.88MPC2.73/.74C1.87/.90C2.85/.85C3.87/.85C4.82/.82C5.87/.90C6

CLE: clinical Learning / SDL: Self-Directed Learning /C&T: Coaching and Teaching / WP: Welcoming Procedure /WO:Work Organization /MPCS: Multi-Professional Collaboration

Validity and reliability of activities coaching context questionnaire

6 MJIRI, Vol. 28.41. 7 June 2014http://mjiri.iums.ac.ir

cology, and rheumatology fields. Other fac-tors such as physical condition of patients,food served free of charge, suitable workschedule and provision of rest breaks atwork might be responsible for the betterperformance of internal ward.

ConclusionThe Persian version of ACC for evaluat-

ing the quality of clinical learning envi-ronment appeared to be a valid and reliableinstrument in Iran and can also be used toevaluate the learning environments of otherfields of medical education. It is suggestedto use this instrument and to repeat factoranalysis with a larger population to developand remove the trivial problems which maybe caused in other settings. A qualitativestudy in major clinical wards is suggestedto extract in depth data for promoting theperformance of educational environment.

AcknowledgementWe should, hereby thank all physicians

and health workers who sincerely cooperat-ed to accomplish the objectives of this pro-ject. We would like to thank Dr. VahidSemnani and Dr. Raheb Ghorbani fromSemnan University of Medical Sciences forthe time and valuable help they gave to im-prove the paper.

References1. Fasihi Harandi T. Clinical educational quality

from view point of medical students:iran unversityof medical sciences. In: soltani arabshahi k, editor:qazvin university of medical sciences journal; 2004.p. 4-9.

2. Soltani Arabshahi K, koohpayeh zadeh J, SobutiB. The main sectors of the clinical learning envi-ronment, teaching hospitals of Iran University ofMedical Sciences learners zero sum modelDREEM.Iranian Journal of Medical Education/spring summer 1387, 8 (1): 43 to 50

3. Malakan Rad E, et al. teaching and assessment:what every clinical teacher must know: ISBN 964-8505-29-2

4. Nagraj S, Wall D, Jones E. Can steam be usedto measure the educational environment within theoperating theatre for undergraduate medical stu-dents. Birmingham Heartlands Hospital,Heart ofEngland NHS Foundation Trust. UK, Med Teach.2006 Nov; 28(7):642-7

5. Holt MC, Roff S. Development and validationof the Anaesthetic Theatre Educational EnvironmentMeasure (ATEEM).Med Teach 26:553, 2004 553-8

6. Roff S, McAleer S, Skinner A. Developmentand validation of an instrument to measure the post-graduate clinical learning and teaching educationalenvironment for hospital-based junior doctors in theUK. Med Teach. 2005;27(4):326-331

7. Dimoliatis I, Vasilaki E, Anastassopoulos P, Io-annidis J, Roff S. Validation of the Greek translationof the Dundee ready education environment meas-ure (DREEM). Education for Health. 2010;23(1):1-16

8. Deketelaere A, Kuppens S, Ceulemans E, LisaD, Agnes D, Paul D. Multi-factorial aspects of aclinical learning environment: rationale, develop-ment and psychometric properties of the ACC-questionnaire.version.2011.