Embed Size (px)
Citation preview
Original Research Article A MS J
Australian Medical Student Journal14
IntroductionChest radiology is important for acute and emergency management, and is therefore an essential learning component of undergraduate radiology teaching. [1] However, studies show that chest radiology competency amongst graduating medical students is poor. [2,3] Poor competency is attributed to lack of formal teaching of radiology in the curriculum. [2,3] Worldwide, radiology teaching is compromised by limited formal teaching in a hectic curriculum, and competing demands on radiologists. [4,5]
Computer aided learning (CAL) has been advocated as a potential tool to alleviate some of the limitations in radiology teaching. [6] CAL is time and cost effective for educators, [7] and especially useful in an image rich specialty such as radiology. To evaluate the effectiveness of CAL for transferring knowledge gain, previous studies have undertaken media comparisons between CAL and traditional learning, such as lectures or tutorials. Individual studies in radiology and non-radiology medical education [8,9] demonstrate that overall, knowledge gain with CAL is comparable to traditional media. [6,7,10] However, critics have repeatedly advocated for a shift of focus – away from the debate of whether CAL is superior, to research using CAL to CAL comparisons, and on how CAL can be used effectively in the curriculum. [11,12]
The majority of media comparison studies in undergraduate medicine have limited CAL usage to a controlled onsite session. In these studies, the study design consisted of a pre-test, CAL or traditional learning, followed immediately by the post-test – therefore, usage of CAL outside a single session was not possible. [6,8-10] This is paradoxical, as CAL is valued by students and educators alike for its flexibility and convenience of access offsite. [7]
Onsite and offsite use of computer aided learning in undergraduate radiology education
Education literature have promoted student-centered learning to be useful for deep understanding, and flexibility to be useful for developing self learning skills in an era where information is rapidly updating. Further, due to the web-based nature of recent CAL programs, access is unlikely to be restricted to a single session. In an attempt to emulate “real world” use of CAL, Devitt, Palmer and Hudson used a different study design whereby students were given free access to a CAL program for a period of two weeks. [13,14] Third year students were permitted to use the CAL program at a time and place of choice. The experience with offsite learning showed access logs of some students failing to access the program at all; yet some of the same students were willing to attend scheduled pre- and post-tests. [14]
Aims of the StudyIn light of prior experiences with offsite CAL use, whether knowledge gain between traditional media and CAL is still comparable when CAL is used offsite becomes a relevant question to explore. Using a CAL radiology courseware, our study aims to compare usage and knowledge gain in onsite and offsite settings for undergraduate students.
MethodsThe programThe “Radiology Courseware” created by the radiology department at the National University of Singapore (NUS) is a CAL program with 102 electronic pages, and covers principals of chest radiology interpretation and common pathology. It was created to supplement undergraduate teaching.
SubjectsAll fourth year clinical students undergoing their one week radiology rotation in October 2009 (n=52) were invited to participate in the study (Figure 1). Traditionally, fourth years received a didactic lecture on chest radiology. Students in this study were given a web-based chest radiology (chest x-ray; CXR) CAL program to complete and a supplementary chest radiology lecture after completing pre- and post-tests.
Prior to the rotation, the radiology department allocated ten clinical groups (n=5 or n=6 per group). For the purposes of this study, each clinical group was evenly allocated into either group A or group B by alternation. Group A completed the chest radiology CAL program onsite in the department library during an allocated two hour session. Group B were given one week to complete the same CAL program offsite, at a time and place of choice. Participating students provided informed consent.
Aim: Computer-aided learning (CAL) is considered comparable to traditional media for undergraduate radiology teaching. Previous studies have often compared the efficacy of traditional media to onsite CAL use, yet real world usage of CAL is likely to occur in offsite settings. This study aims to compare usage and learning outcomes of a chest radiology CAL in onsite and offsite settings. Methods: Participants were fourth year medical students (n=52) at the National University of Singapore (NUS) undertaking one week radiology rotations. Students were randomly allocated to complete a web-based chest radiology CAL onsite, or offsite at a time and place of choice. Pre- and post-tests were taken to measure knowledge gain, and a questionnaire was used to explore student usage and preferences. Results: The onsite CAL group demonstrated significant knowledge gain (+15.8%, p<0.05) whilst the offsite group did not (+5.8%, p>0.05). However, the difference between the groups was not statistically significant (p=0.069). Total time spent and completion of the program was similar between the two groups. Yet, questionnaire results showed that the offsite group multitasked more and appeared to have poorer concentration. A majority of students from both groups preferred the convenience of offsite CAL use over onsite CAL use. Conclusion: A significant difference between the test groups was not observed, although there was a trend toward onsite CAL use being more effective. In planning CAL teaching, particularly for offsite use, educators need to provide sufficient support and integration for an optimal outcome.
Winnie ChenFifth Year Medicine (Undergraduate)University of Melbourne
Winnie completed her research year in July 2010 at the National University of Singapore. She is currently completing her clinical studies in Ballarat and Western clinical school. She is interested in rural health, and hopes to be involved with medical education and medical writing in the future.
Volume 2, Issue 2 | 2011
Australian Medical Student Journal 15
Medical students in radiology rotation
n = 52
Group A n = 26 (100%)
Pre-test
Group B n = 26 (100%)
n = 22 (85%) n = 26 (100%)
Onsite Intervention
Offsite Intervention
Post-testn = 17 (65%) n = 24 (92%)
Lecture
Questionnairen = 21 (81%) n = 21 (81%)
Figure 1. Study design and participants.
Pre- and post-testsPre- and post-tests were completed at the start and end of the rotation week, respectively. The pre- and post-test were derived from the chest radiology CAL program content and consisted of thirteen multiple choice questions. Blueprinting ensured that areas of knowledge tested were proportional to the CAL content and concordant with intended chest radiology learning outcomes for clinical students. For radiograph interpretation questions, only standard teaching images from the program itself were used. The test was reviewed and approved by a senior radiologist.
The pre- and post-tests covered similar topics, but differed in content. Although identical tests would ensure equal difficulty for pre- and post-tests, it may also increase the likelihood of students performing better in the post-test, simply as a result of participating in the pre-test. The pre- and post-tests were piloted on elective medical students and were rated to be of comparable difficulty. The tests were collectively administered on PowerPoint projections to provide good image visibility, and a standardised test setting for the students. Students were required to work individually. Pre- and post-tests were matched by de-identified codes.
QuestionnaireAt the end of the study a questionnaire was given to each participating student. The questionnaire included questions on demographics, onsite and offsite usage patterns, and student preferences for setting. The questionnaire was modelled after similar qualitative evaluations of CAL from previous studies. [6,8] Questions regarding impact of setting on learning and motivation were based upon psychology and education learning theories. [15,16] Computer usage and CAL attitude questions were based on previous computer attitude surveys. [17-19] A combination of open and closed question techniques were used. Most closed questions were in a 5-point Likert scale form with responses from “strongly agree” to “strongly disagree.” Open questions were used for greater exploration of student attitudes and preferences, and the reasoning behind their opinions. The instrument was piloted together with the pre- and post-tests. Poorly phrased questions eliciting ambiguous answers were excluded or revised.
Statistical analysisData was entered into Microsoft Excel and analyses were performed using SPSS 17.0 created by IBM (New York, USA). Descriptive results were presented for the questionnaire, and pre- and post-test results were analysed by within group (Wilcoxon signed rank test) and between group (Mann Whitney U test) comparisons. Raw scores were converted to percentages for easy interpretation. A p-value of 0.05 or less was considered to be statistically significant.
Ethical approvalThis study was exempt from NUS Institutional Review Board as it was an educational settings research without identifiers. The project was approved by the Head of Department at NUS, Department of Diagnostic Radiology, and the Dean for Yong Loo Lin School of Medicine.
ResultsDemographicsOverall participation was high for the pre-test, post-test and questionnaire. For the questionnaire, group A and group B both had a response rate of 81%. For pre- and post-tests, participation was higher amongst group B students (Figure 1). Student demographics for age and gender were similar between the two groups (Table 1).
Table 1. Summary of student demographics.
Pre- and post-test resultsThe scores obtained by students ranged from 38.5 to 100% for both pre-test and post-test scores. Pre-test scores were significantly lower in group A compared to group B (p<0.05). Pre- and post-test differences showed that the knowledge gain in group A was statistically significant, but the knowledge gain in group B was not statistically significant (Table 2). Between-group comparisons show that group A had a 10.0% larger mean improvement than group B, but this was not statistically significant (p>0.05).
Table 2. Summary of pre- and post-test results.
Usage and completionTotal time spent on the chest radiology CAL was measured via student self reporting on the questionnaire, and observation of the onsite group. The two groups reported similar duration of use, with 46 minutes average for Group A, and 47 minutes average for Group B. Self reported completion of the chest radiology CAL was also similar across both groups (Table 3). The entire CAL program was completed by 71.4% of students in group A and 66.7% of students in group B and a proportion of students in both groups A (23.8%) and B (19.0%) completed half, less than half or none of the chest radiology CAL.
Table 3. Total amount of CAL completed.
Gender
Average age (years)Male Female
Group A (onsite) 57.1% 42.9% 22.19
Group B (offsite) 61.9% 38.1% 22.10
Total 59.5% 40.5% 22.14
Pre-test Post-test Improvement
Group A (onsite) 69.7±6.1 85.5±8.0 15.8±9.3*
Group B (offsite) 78.5±3.8 84.3±4.2 5.8±6.8**
Difference - - 10.0±10.9****p=0.005, **p= 0.109, ***p=0.052
None < Half Half > Half All
Group A 0.0% 4.8% 23.8% 0.0% 71.4%
Group B 4.8% 9.5% 9.5% 9.5% 66.7%
Total 2.4% 7.2% 16.7% 4.8% 69.1%
A MS J
Australian Medical Student Journal16
Motivation, distraction and multitaskingSelf reported motivation levels were moderately high and similar between group A and B (Figure 2). Most students selected “agree” (52.4% in group A, 61.9% in group B) or “strongly agree” (19.0% in group A, 14.3% in group B) to the statement that “I was motivated during the CAL session.” Overall distribution of responses to the statement “I was distracted during the CAL session” was similar across all options (Figure 3). Notably, a higher proportion of group A respondents (33.3%) strongly disagreed that they were distracted during the chest radiology CAL compared to in group B (4.8%). Conversely, more respondents in group B (23.8%) than group A (4.8%) strongly agreed that they were distracted during the session.
Finally, multitasking, such as use of instant messengers, and social network websites during CAL, was used as an additional criterion for assessing motivation and distraction. As with the previous question on distractions during CAL, responses to the statement “I was multitasking during the CAL session” also ranged from highly agree to highly disagree (Figure 4). However, a distinct pattern to the question was seen with the majority of group A students strongly disagreeing (52.4%) or disagreeing (33.3%), and the majority of group B students strongly agreeing (38.1%) or agreeing (38.1%) to the statement.
Preference for settingThe most popular chest radiology learning method in both onsite group A (90%) and offsite group B (61.9%) was the “both CAL and lecture” teaching, compared to “CAL only” or “lecture only” teaching (Figure 5). Explanation given for wanting to complete CAL offsite included freedom, flexibility and convenience to complete the task at one’s “own time, own target.” Reasons given for preferring to access CAL during scheduled sessions included being “forced” to complete the task, having a tutor present and the ability to discuss learning with peers.
Discussion
Previously, CAL studies have predominantly occurred in an onsite setting, where access was controlled to a single, scheduled session. This prospective, randomised study aimed to test the hypothesis that onsite and offsite access of CAL differed in usage and learning outcomes.
Pre- and post-test resultsPre- and post-test differences demonstrated a statistically significant (p<0.05) improvement in chest radiology knowledge for students in the onsite CAL group. Offsite CAL students also gained knowledge, but this was not statistically significant. Between-group comparison showed that knowledge gain was greater onsite than offsite, but this was also not statistically significant.
Searches of Pubmed and ERIC databases of articles up to December 2009 did not reveal any previous studies investigating learning outcomes in onsite and offsite CAL settings. Nevertheless we can understand the results of this study better in light of previous studies comparing CAL to CAL learning outcomes. Maleck et al. studied third year medical students (n=225) in a comparable pre- and post-test design. Their results showed post-test improvements of 11.2% and 15.1% (p<0.05 for both), after non-interactive and interactive CAL, respectively. [6] A separate study of similar design in third year medical students (n=100) showed neuroradiology knowledge improvements of 16%, 17% and 21% (p<0.05 for all) for didactic, problem solving and free text CAL, respectively. [14] Considering these results, the onsite improvement of 15.8% (p<0.05) in this study is equal, whereas the offsite improvement of 5.8% (p>0.05) is smaller in magnitude compared to previous studies.
Usage of chest radiology CALUsage results contradicted the hypothesis that there is a difference between onsite and offsite groups in duration of use, and amount
"I was motivated during the CAL session"
Strong
ly dis
agree
Disagre
e
Neutra
l
Agree
Strong
ly ag
ree0
20
40
60
80
100
% P
artic
ipan
ts
Group AGroup B
"I was distracted during the CAL session"
Strong
ly dis
agree
Disagre
e
Neutra
l
Agree
Strong
ly ag
ree0
20
40
60
80
100
% P
artic
ipan
ts
Group AGroup B
"I was multitasking during the CAL session"
Strong
ly dis
agree
Disagre
e
Neutra
l
Agree
Strong
ly ag
ree0
20
40
60
80
100
% P
artic
ipan
ts
Group AGroup B
Preferred method for CXR learning
CAL only
Both C
AL and
lectu
re
Lectu
re on
ly0
20
40
60
80
100
% P
artic
ipan
ts
Group AGroup B
Figure 4. Multitasking during the CAL session (“I was multitasking during the CAL session”).
Figure 2. Motivation during the CAL session (“I was motivated during the CAL session”).
Figure 3. Distraction during the CAL session (“I was distracted during the CAL session”).
Figure 5. Preferred method for CXR learning.
Volume 2, Issue 2 | 2011
Australian Medical Student Journal 17
of CAL completed. Average time spent in the two groups was almost identical. In contrast, Devitt and Palmer’s interactive to non-interactive CAL study showed a difference of up to 50 minutes in average duration of use between different CAL groups. [13] Hudson’s study also showed up to twenty minutes difference in average time spent on CAL between different CAL groups. [14] Previous studies did not directly investigate the amount of CAL completed, but based estimations on login data. Based on login data, Hudson concluded that there were students who failed to use the CAL program at all. [14] Using student reported data, this study also identified individuals from both groups who completed little or none of the chest radiology CAL. Overall, 69% of all students completed the entire CAL, and this was similar between the onsite and offsite groups.
Though time spent on CAL was similar between onsite and offsite groups, the quality of time spent may have contributed to differences in knowledge gain. Student in onsite and offsite settings self reported similar levels of interest and motivation for CAL. However, more students in the offsite group agreed to the statement that “I was distracted during the session.” Moreover, multitasking, such as checking emails and news online, was distinctly more prevalent in the offsite group. Onsite students may be motivated to use CAL through influence of tutors, peers undertaking similar tasks, and by being physically present in a learning environment. [20] On the other hand, offsite CAL is akin to distance learning, which is more difficult to motivate and monitor. [20] Despite apparent difficulties in concentration offsite, the majority of students preferred to complete CAL offsite rather than onsite. Students found the flexibility and “own time, own target” capacities of CAL particularly attractive in a busy medical curriculum.
LimitationsThe initial pre-test results in the offsite group were higher than that of the onsite group, and may have contributed to the smaller post-test improvement in the offsite group. However, the offsite group appears to have had room for measurable improvement in this study, as students achieved up to full marks in both pre- and post-tests.
Sample size was limited by the use of radiology rotation students at a single site, and this limited the possibility of a control group, and the ability of the study to demonstrate statistical significance. Volunteer recruitment from the entire cohort may have provided a larger sample size, but is also likely to introduce selection bias. Recruiting students from other teaching hospitals may confound results as radiology teaching content differs between each hospital. Another limitation in the sample was that pre- and post-test participation was higher in group B than group A, and it is difficult to determine whether selection bias occurred.
Our study only tested for short term chest radiology knowledge gain. Some researchers have used a delay test method to test knowledge retention, and long term effects of CAL interventions. [21] In this study, a post-test was performed in the same week as the CAL intervention because differences in clinical exposure and teaching subsequent to the radiology rotation may confound knowledge gain.
Usage data was mainly drawn from the questionnaire. Consequently, honesty of student feedback is a potential limitation. To encourage honest answers, the questionnaire was anonymous and non-judgmental wording was used. Another limitation of the questionnaire was that no statistical demonstrations of reliability were available.
As with previous studies, this study faced challenges in balancing internal and external validity. [14] Some researchers limited student interaction, requiring students to use CAL individually. [9,13] In many onsite studies, usage outside of allocated sessions was also impossible,
as the posttest was scheduled immediately after CAL intervention. In contrast, this study did not control peer interaction, and free access of CAL was permitted outside of scheduled sessions for the onsite group. Although internal validity is reduced, we used this methodology to allow investigation of student usage and learning outcomes of CAL in “real life” onsite and offsite settings.
CAL setting was categorised into two broad settings for the purposes of the study. We recognise that categories of settings need not be limited to onsite and offsite settings. Learning style and needs also varies between students, between institutions and even within a single cohort. Within the students in this study, a range of preferences and attitudes towards CAL was seen. Clearly, recommendations from this study may not benefit all students; for example, highly self motivated students are less influenced by external motivational factors, [16] and for these students, setting would be unlikely to result in differences in CAL usage and learning.
ConclusionThis study addresses an unanswered question in CAL literature regarding the differences in usage and learning between different CAL settings. According to use of setting in studies to date, CAL use was categorised into onsite and offsite settings. The results showed that the onsite setting discouraged multitasking and may have produced greater knowledge gain. Yet there is great potential for CAL use in medical teaching due to its flexibility and accessibility outside a physical classroom.
Together with previous researchers, [8,14] we recommend that implementation of offsite CAL use needs to be carefully supported and planned into the curriculum. Improvements to integration may include setting deadlines, use of electronic or face to face reminders to complete CAL, and implementation of assessments for evaluation and feedback. In particular, experience with medical students shows that formal assessment is important in motivating effective CAL use. CAL can be used together with traditional learning; as students suggested, basic information can be provided by CAL, followed by a subsequent summary lecture where questions can be addressed. Alternatively, tutors can answer questions by invitation of email questions, and incorporate frequently asked questions for future reference. [8]
Further investigation is needed to understand and optimise CAL use in medical teaching. For example, CAL use and outcomes with and without tutor feedback can be compared. Furthermore, ongoing monitoring of usage and learning outcomes is required as new CAL curriculums and technologies are developed and implemented.
AcknowledgementsThe chest radiology courseware used in this study was developed by staff of the National University Singapore (NUS) Department of Diagnostic Radiology. I thank my supervisor, Dr. Goh Poh Sun for his guidance, and Dr. Chan Yiong Huak for his statistical advice.
Conflict of interestNone declared.
CorrespondenceW Chen: [email protected]
References[3] Jeffrey D, Goddard P, Callaway M, Greenwood R. Chest radiograph interpretation by medical students. Clin Radiol 2003;58(6):478-81.[4] Gunderman R, Siddiqui A, Heitkamp D, Kipfer H. The vital role of radiology in the medical school curriculum. Am J Roentgenol 2003;180(5):1239-42.
[1] Subramaniam R, Gibson R. Radiology teaching: Essentials of a quality teaching programme. Australas Radiol 2007;51(1):42-5.[2] Subramaniam R, Hall T, Chou T, Sheehan D. Radiology knowledge in new medical graduates in New Zealand. N Z Med J 2005;118(1224):U1699.
A MS J
Australian Medical Student Journal18
[14] Hudson J. Computer-aided learning in the real world of medical education: Does the quality of interaction with the computer affect student learning? Med Educ 2004;38(8):887-95.[15] Mayer R. The Cambridge handbook of multimedia learning. Cambridge, UK: Cambridge University Press; 2005.[16] Archer J. From motivation to self-Regulation. In: Chin C, Salili F, Hong Y [Eds]. Multiple Competencies and Self-Regulated Learning. Greenwich, Connecticut: Information Age Publishing; 2001.[17] Dorup J. Experience and attitudes towards information technology among first-year medical students in Denmark: Longitudinal questionnaire survey. J Med Internet Res 2004;6(1):e10.[18] Link T, Marz R. Computer literacy and attitudes towards e-learning among first year medical students. BMC Med Educ 2006;6(1):34.[19] Virtanen J, Nieminen P. Information and communication technology among undergraduate dental students in Finland. Eur J Dent Educ 2002;6(4):147-52.[20] Khan B. Flexible learning in an open and distributed environment. In: Khan B [Ed]. Flexible Learning in an Information Society. London, UK: Information Sciences Publishing; 2007.[21] Erkonen W, D’Alessandro M, Galvin J, Albanese M, Michaelsen V. Longitudinal comparison of multimedia textbook instruction with a lecture in radiology education. Acad Radiol 1994;1(3):287-92.
[5] Cassumbhoy R, Lau W. Challenges in medical imaging teaching. Australas Radiol 2007;51(4):307-8.[6] Maleck M, Fischer M, Kammer B, Zeiler C, Mangel E, Schenk F, et al. Do computers teach better? A media comparison study for case-based teaching in radiology. Radiographics 2001;21(4):1025-32.[7] Ruiz J, Mintzer M, Leipzig R. The impact of E-learning in medical education. Acad Med 2006;81(3):207-12.[8] Smith S, Roberts N, Partridge M. Comparison of a web-based package with tutor-based methods of teaching respiratory medicine: subjective and objective evaluations. BMC Med Educ 2007;7(1):41.[9] Vichitvejpaisal P, Sitthikongsak S, Preechakoon B, Kraiprasit K, Parakkamodom S, Manon C, et al. Does computer-assisted instruction really help to improve the learning process? Med Educ 2001;35(10):983-9.[10] Lieberman G, Abramson R, Volkan K, McArdle P. Tutor versus computer: A prospective comparison of interactive tutorial and computer-assisted instruction in radiology education. Acad Radiol 2002;9(1):40-9.[11] Cook D. The research we still are not doing: An agenda for the study of computer-based learning. Acad Med 2005;80(6):541-8.[12] Adler M, Johnson K. Quantifying the literature of computer-aided instruction in medical education. Acad Med 2000;75(10):1025-8.[13] Devitt P, Palmer E. Computer-aided learning: An overvalued educational resource? Med Educ 1999;33(2):136-9.
