From Expectation to Evaluation of E-Learning - Experience from the Water Education E-Learning

Project -

Hsin-Yu Shan

Dept. of Civil Engineering, National Chiao Tung University, Taiwan

Jen-Gaw Lee

National Center for High-Performance Computing, Taiwan

Frances Su

Microelectronics and Information Systems Research Center, National Chiao Tung University, Taiwan

Abstract

While the rapidly development of information and communication technology constantly provides innovative tools and methods of education, the real world has not been able to catch up. Traditional face to face teaching and training still prevails in schools and in the work place. Although many forms of e-learning courses have been experimented in schools and delivered by global enterprises for on-the-job training, analysis on the difference between expectation of the participants before the courses and satisfaction afterwards is scarce. Questions on what the participants of e-learning programs expect have seldom been asked. Furthermore, little information is available on comparing the evaluation of the e-learning program afterwards and the expectation before it began. This paper presents the results of a study accompanying a large-scale e-learning program for continued education of thousands of school teachers in Taiwan. The expectation before beginning the asynchronous on-line courses and evaluation of the program and their own performance were collected by compulsory questionnaires. The results showed that most of the participants think highly of the e-learning program and considered the program delivered the knowledge and skills as expected. However, the interaction among participants and between instructors and participants was less satisfactory as other aspect of the e-learning program also as the participants expected. Nevertheless, e-learning remains to be the most efficient way to deliver high-quality education programs to a large number of participants. In addition, observation of the participants’ behavior reveals that good management of e-learning programs is labor-intensive.

Keywords: Distance learning, web-based, asynchronous, water environment.

Introduction The rapid development of information and communication technology (ICT) has not only

facilitated classroom learning tremendously but is also increasingly enabling education to move outside of the classrooms and even schools. The most up-to-date “ubiquitous learning

environment” (Shankar et al., 2001) enables learning at anytime and anywhere such that the whole paradigm of learning could be revolutionized.

However, research to date suggests that traditional classroom education results in a more favorable learner response than e-learning. This issue represents a perplexing problem for proponents of e-learning. It also raises the question of whether evaluation of e-learning compared with traditional learning should be the real issue or is it evaluation of e-learning within itself?

On the other hand, the design of e-learning programs have benefited from the use of instructional system design models very early on as many e-learning programs involved educational research. Thus in many cases, e-learning programs are better structured and managed than traditional education programs, which were not designed and implemented with systematic approach.

Instructional design is defined by Berger and Kam (1996) as ‘the systematic development of instructional specifications using learning and instructional theory to ensure the quality of instruction. Instructional system design (ISD) evolved from post-World War II research in the United States military to find a more effective and manageable way to build training programs. Numerous ISD models have been developed over the years, and they are also known as Instructional Systems Design and Development (ISDD), the Systems Approach to Training (SAT), or just Instructional Design (ID). These include ADDIE, Dick and Carey (1978), Hannifen and Peck (1988), Knirk and Gustafson (1986), Kemp (1985), and Gerlach and Ely (1980) to name a few. These models provide a step-by-step system for the evaluation of participants’ needs, the design and development of training materials, and the evaluation of the effectiveness of the training program. Almost all of these models are based on the generic ADDIE model while making modifications to emphasize a more holistic, iterative approach to the development of training. The five generic phases of ADDIE are: 1. Analyze, 2. Design, 3. Develop, 4. Implement, and 5. Evaluate. These phases provide dynamic and flexible guidelines which are used for effective and efficient instruction.

The cycle (Fig. 1) starts with the analysis as the designer develops a clear understanding of the desired outcomes or behaviors, and the participants’ existing knowledge and skills. The design phase documents specific learning objectives, assessment instruments, exercises, and content. The creation of learning materials is completed in the development phase. During implementation, these materials are delivered or distributed to the participants. And finally the effectiveness of the training program is evaluated.

Figure 1 The ADDIE Model

Analyze

Design

Develop

Implement Evaluate

On the other hand, for implementing education programs, Gagne et al. (1992) introduced

the concept of nine events of instruction and their corresponding cognitive processes (Table 1). The nine events are excellent steps that the design of e-learning programs can easily follow.

Table 1: Gagne’s Nine Instructional Events

1. Gaining attention (reception) 2. Informing learners of the objectives (expectancy) 3. Stimulating recall of prior learning (retrieval) 4. Presenting the stimulus (selective perception) 5. Providing learning guidance (semantic encoding) 6. Eliciting performance (responding) 7. Providing feed back (reinforcement) 8. Assessing performance (retrieval) 9. Enhancing retention and transfer (generalization)

Evaluation approaches based on the program goals are fundamentally oriented towards the

verification and quantification of the execution of the program goals. For this purpose, experimental or quasi-experimental research models are used in a 4-step approach: (1) Identification of the program goals, (2) Translation of the program goals into quantitative data, (3) Data collection from individuals related to the program, and (4) Comparison of stated goals with attained goals.

As indicated by Figueira (2003), there are five major clusters of variables that affect the outcome of e-learning: individual learner variables, environmental variables, technology variables contextual variables and pedagogic variables. Each of these can be disaggregated into more precise groups (Table 2).

Table 2 Variables of E-Learning Individual learner variables include:

Physical characteristics (e.g. age, sex, physical abilities) Learning history, (negative/positive experience, level of attainment, duration, recentness etc)

Learner attitude (positive / negative) Learner motivation (high / low) Familiarity with the technology

Learning environment variables include: The immediate (physical) learning environment The organizational or institutional environment The subject environment

Contextual variables include: Socio-economic factors (e.g. Class, gender,) The political context (e.g. Who is funding /paying for the e-learning and for what reason?)

Cultural background (e.g. How highly is learning / e-learning valued?) Geographic location (e.g. Country, language, urban/rural)

Technology variables include: Hardware Software, Connectivity,

The media Mode of delivery

Pedagogic variables include: Level and nature of learner support systems Accessibility issues. Methodologies Flexibility Learner autonomy Selection and recruitment Assessment and examination Accreditation and certification

This paper describes the results of a study conducted on the participants of the Water

Environment Education Project. The project implanted a large-scale e-learning continued education program for school teachers in Taiwan. It is the largest e-learning projects in Taiwan on a single subject and specific group of participants. The design and evaluation of the e-learning program followed the ADDIE model and other principles outlined in previous paragraphs.

The Water Education Project The Water Environment Education Project is sponsored by the Ministry of Education, the

Environmental Protection Agency, and the Water Resources Agency. The objective of the project is to enhance the knowledge and skill of school teachers to integrate water environment education into day-to-day classroom teaching in all subject areas in elementary schools and high schools. As a matter of fact, Taiwan has been promoting environmental education in schools for over a decade. However, the transfer of knowledge and attitude on sustainable development and water environment had not been as successful due to the difficulty to train more than 200,000 school teachers in the entire nation with good instructors and high quality course contents.

Fortunately, Taiwan is not only a country that produces most of the notebook computers and internet networking equipment, but also has a high household broadband internet connection percentage of 65% (Taiwan Network Information Centre, 2006). As a result, Taiwan already has a mature environment for wide application of e-Learning. Thus, the delivery of high quality water education training courses to thousands of teachers all over the island in a short time can rely on no other method than distance learning.

The project was initiated under the leadership of Taiwan International Institute for Water Education (TIIWE) and the work team consists of Graduate Institute of Environmental Engineering of National Taiwan University, ITS EduMedia Studio of Microelectronics and Information Systems Research Center of National Chiao Tung University (NCTU), Chinatimes Cultural and Educational Foundation, National Center for High-Performance Computing (NCHC) and Environment Conservation Center of National Taiwan Normal University. The training program is co-sponsored by the Ministry of Education, the Environmental Protection Agency, and the Water Resources Agency. The target participants of the water education training program described in this paper are the 200,000 teachers from elementary schools, junior and senior high schools, and vocational schools.

The course topics not only covers science, ecology, engineering knowledge on water environment but also include social, political, and economical issues surrounding water resources development and water environment protection. After two years of carrying out the

project, four modules each with 10 1.5-hour asynchronous classes are on-line. During 2004, 2005, 2006, a total of 5,617 people registered to participate and 3,940 participants finished at least one of the modules. The total number of modules taken by the participants was more than 15,000. The participants were awarded certificate for the modules and hours they completed.

The Web-Site and the E-Learning Program

Course Content and Research Process

In the first year, the e-learning program provides 20 1.5-hour courses on fundamental water education. In the second year, 20 advanced courses were added to the program. In addition to the on-line courses, there is an accompanying one-day face-to-face learning session which includes half day of classroom discussion and half day of field trip available for the more active participants to attend. The overall course structure and classes are listed in Table 3.

Table 3 List of On-Line Courses on Water Education Program

Fundamental Water Education Courses Advanced Water Education Courses Module A Module C Theme: Humanistic Aspect of Water Environment Theme: Humanistic Aspect of Water Environment A01 The Earth Environment and Sustainable Development: The Continuation of Civilization

C01 Water Environment and Civilization

A02 The History of Water Resources Development C02 Global Environmental Crisis A03 The Evolution of Environmentalism, Civil Movement, and International Environmental Treaties

C03 Biodiversity and Preservation of Taiwan’s Ocean Environment

A04 Impact of Economical Development on the Environment

C04 Sustainable Development and International Cooperation

A05 Governance of Water Environment – Legal and Economic Control

C05 Environmental Economics

Theme: Ecology of Water Environment C06 Oceanic Heritage and Expeditionary Spirit A06 Ocean and Coastal Ecology C07 Aboriginal Heritage Culture and Environment

Preservation A07 Ecology of Lakes and Reservoirs Theme: Environmental Education A08 Investigation Methods of Water Environment C08 Environmental Organizations and Environmental

Movements A09 River Ecology C09 Objectives and Strategies of Environmental

Education A10 Evolution of Taiwan’s Environment C10 Environmental Education Teaching

Methodology –Water Environment Themes Module B Module D Theme: Water Environment and Earth Science Theme: Water Environment and Management B01 The Atmospheric System and Global Climate Change

D01 Characteristics of River Environment and Its Evolution

B02 Hydrologic Cycle (I) – Atmosphere and Ocean D02 Crisis of the River Environment B03 Hydrologic Cycle (II) – Surface Water and Ground Water

D03 River Engineering and Management

Theme: Water Environment and Hydraulic Engineering

D04 Stories of Successful River Restoration Projects

B04 River Engineering and Flood Protection D05 Geographical Characteristics and Change of Taiwan’s Coastal Zone

B05 Water Resources Development, Conservation and Water Supply

D06 Development and Management of Coastal Zones

Theme: Water Environment Pollution and Natural Hazards

D07 Taiwan’s Groundwater Resources and Quality

B06 Surface Water Pollution and Prevention D08 Soil and Groundwater Pollution and Protection B07 Ocean Pollution And Prevention Theme: Environmental Education B08 Soil and Ground Water Pollution and Prevention D09 Sustainable Campus (I) – Sustainable Campus

Initiative and Program B09 Natural Hazards of Water Environment and Mountainous Regions and Hazard Mitigation

D10 Sustainable Campus (II) – Concept of Sustainable Campus Architecture

B10 Environmental Course Design and Teaching Skills The instructors include university professors, senior water managers and engineers of

government agencies. The experts of different domains work together smoothly. Coordination of instructors is often the most difficult task of managing a training program; however, the group of instructors worked together in harmony and delivered a course program with a perfect mix of key elements of water education.

The e-learning program was designed to incorporate studies on e-learning relative issues with the participants as the subjects. Since the most of participants are school teachers who are required to finish 20 hours of continued education each year, this program becomes one of the best way of the teachers to fulfill this requirement. As a result, the participants are included as the subjects of research in this project. The participants are required to finish questionnaires and tests before and after the on-line courses in order to be cleared as completing the course program and be eligible to receive formal certificate. The questionnaires and tests are built as web pages such unless the participants finished answering the questions, they are not allow to go into next steps to view on-line courses or receive final proof of completion. The flow char of the process is shown in Fig 2. The process also included the nine instructional events to facilitate a successful e-learning program.

Figure 2 E-Learning and Investigation Process

Registration

On-Line Welcome Ceremony

On-Line Introduction of LMS and Course Program

Questionnaire on Background and Expectation

Pre-Test

On-Line Course

On-Line Course

Quiz Quiz

Post-Test

Questionnaire on Program and Self Expectation

Field TripF2F ClassDiscussionWorkshop

In order to enhance the achievement of learning, the following policies are designed and applied:

40 courses are organized into 4 modules, each module contains 10 courses. Each student has to enroll at least 1 category. The certificate is only awarded to the participants after completion of modules. The minimum hours on the certificate is thus 15 hours. For each category

Each week two new e-learning courses were put on-line. Each course is open for 2 weeks.

There were also optional one-day face to face course and field trip available to the more active participants. These face to face sessions provide an opportunity to enhance interaction between instructors and participants and among themselves.

The Website and the LMS

The on-line course is accessed through the water environment e-learning website hosted in NCHC with the URL http://water.emc.nctu.edu.tw. In addition to the on-line courses, the website also contains other crucial components for e-learning: announcement board, concise articles on various topics regarding water environment, discussion boards, assorted and annotated internet links to environmental education related websites, and participants’ email list. The learning management system (LMS) is developed by ITS EduMedia Studio and fully conforms to SCORM standards.

Figure 3 Sample Screen of the On-Line Course

The courses were produced using Microsoft® Producer to display video of lecturing and

Powerpoint presentation at the same time. Figure 3 is a screen shot from the on-line lecture on “Earth Environment and Sustainable Development.” According to the course content, each 1.5-hour course was divided into sections of about 15 to 20 minutes which is the time span the participants can maintain highly concentrated. This short time section also allowed the participants to fully utilize fragmented time between their classes and other activities in schools or at home. All videos are encrypted by Microsoft® DRM (Digital Rights Management) technology to protect intellectual property rights. Microsoft® media servers were deployed to provide video 150 Kbps course video streaming to participants’ computers which is connected through TANet (Taiwan Academic Network) or ADSL.

Results of the Training Program and the Study

Participants’ Background Analysis A total of 4 terms of e-learning were delivered from 2004 through 2006. The numbers of

participants registered and completed at least one module are listed in Table 4. The dropout rate significantly decreased over time. In order to evaluate the quality of on-line courses and the participants’ responses on the e-learning model, the participants were asked to complete a background survey before the courses and an evaluation survey after completion of the courses.

Table 4 Number of Participants of the Program 2004 2005 1 2005 2 2006

Registered 1305 1252 1392 1668 Completed 702 915 987 1336 Dropout Rate 46.2% 26.9% 29.1% 19.9%

From the age distribution of participants shown in Fig. 4, it is obvious that this program

attracts much higher percentage of younger teachers as compared with statistics on teachers nationwide (MOE, 2006). About 80% participants are between 20 and 40. It is also important to note that approximately 82% of the participants never participated in any education or training programs on water environment and 68% had no previous e-learning experience. However, most of the participants are already frequent computer users where more than 80% of the participants use computers at least 5 hours per week (Fig. 5).

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

<20 21-30 31-40 41-50 >50

ParticipantWhole nation

Figure 4 Comparison of Age Distribution of the Participants and All Teachers in Taiwan

0% 5% 10% 15% 20% 25% 30% 35% 40%

Seldom (< 1 hr@week)

Little (< 3 hr@week)

Often (5 - 10 hr@week)

Frequent (10 - 20hr@week)

Intensive (>20 hr@week)

Figure 5 Intensity of Computer Usage of Participants

Expectation on E-Learning Programs Although the design of the e-learning program was completed without prior analysis of

the potential participants’ background and expectation, the questionnaire done before the on-line courses started provided vital information for the project team to modify the course structure and implementation strategy for terms except the first one. A total of about 10 questions were asked and a part of the results are presented herein.

When participants were asked whether they are more interested in attending e-learning programs than traditional F2F programs, more than 80% of the participants responded positively (Fig. 6). The percentage increases with time moderately. The participants strongly expected that e-learning programs should be able to let them enjoy the flexibility of taking classes when compared to F2F programs (Fig. 7).

0%10%20%30%40%50%60%70%80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

20042005 12005 22006

Figure 6 Interest in E-Learning Program

0%

10%

20%

30%

40%

50%

60%

70%

80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

2004 2005 12005 2 2006

Figure 7 E-Learning Program Should Have Greater Flexibility

The participants also expected e-learning programs can provide more additional

information related to the course subjects than F2F learning (Fig. 8) and they also thought that there should be more discussion on water environment related topics among peer participants (Fig. 9).

0%

10%

20%

30%

40%

50%

60%

70%

80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

20042005 12005 22006

Figure 8 More Additional Information from E-Learning Program

0%

10%

20%

30%

40%

50%

60%

70%

80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

20042005 12005 22006

Figure 9 Expect to Have More Discussion on Water Environment with Peer

Nevertheless, the participants were significantly more conservative when considering

whether they would have more interaction with e-learning programs and whether they would have more fun from the e-learning programs when compared with F2F programs. Only about 60% of the participants expected that they might have more interaction with peer participants (Fig. 10). On the other hand, about 65% of the participants thought they could have more fun during e-learning sessions (Fig. 11). It has to be reminded that 32% of the participants have previous experience with e-learning. It might be interesting to do a cross analysis to find out whether these participants in particular are more realistic due to their personal experience.

0%

10%

20%

30%

40%

50%

60%

70%

80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

20042005 12005 22006

Figure 10 Expect to Have More Interaction with Peer Participants

0%

10%

20%

30%

40%

50%

60%

70%

80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

20042005 12005 22006

Figure 11 Expect to Have More Fun from E-Learning Program

A very important while also interesting expectation of the participants was that they wish

to have instructor available whenever they have questions or want to have a discussion (Fig. 12). Although these participants had known clearly that it would be an e-learning program, they still consider that they need the instructor to be presence all the time.

0%

10%

20%

30%

40%

50%

60%

70%

80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

20042005 12005 22006

Figure 12 Expectation on Instructors’ Availability

Results of Learning Effectiveness Assessment

The effective of learning was assessed by results of (1) pre-test and post-test and (2) after course quiz. The test was designed to evaluate both the knowledge of Taiwan school teachers and the effectiveness of the e-learning program. The same questions (Table 5) were asked in on-line test before and after the completion of course modules. The participants were not notified of the tests thus could not “prepare” for the tests. The results of the tests are shown in Table 6. The percentage of improvement of participants understanding of important concepts varies with the highest close to 20%.

Table 5 Questions for Quiz before and after the E-Learning Courses Single Choice Questions 1. What is the average annual rainfall in Taiwan? (Answer: C)

A)1000 B)1500 C)2500 D)500 E)5000 mm 2. Hydrological cycle is a very important material cycle of nature. Part of the rainwater that is not

evaporated nor infiltrated becomes which of the following? (Answer: B) A)Percolation B)Runoff C)Transpiration D)Condensation E)Water vapor

3. Which of the following is not one of the major functions of reservoirs? (Answer: D) A)Storage of water B)Recreation C)Electricity generation D)Climate stabilization E)Flood regulation

4. What proportion of the surface of the earth is covered by water? (Answer: D) A)20% B)40% C)50% D)70% E)90%

5. What is the proportion of fresh water in the water on earth? (Answer: E) A)50% B)35% C)15% D)7% E)3%

6. If the river is polluted by animal waste which of the following item will be present in the water? (Answer：B)

A)Heavy metals B)Nitrogen as in ammonium C)Hardness D)Strong acid E)Color 7. Which is the longest river in Taiwan? (Answer：C)

A)Tanshui River B)Kaoping River C)Chuoshui River D)Shewguluan River E)Tadu River 8. Which is the largest proportion of water use in Taiwan? (Answer: C) A)Domestic B)Industrial C)Agricultural D)Mining E)Fish farming 9. Which of the following country’s rate of sewer service is lower than Taiwan? (Answer：E)

A)USA B)Philippines C)Malaysia D)Mexico E)None of the above. 10. The average domestic water usage per person per day is around 100 – 150 liters. What is the water

usage per person per day for Taiwan? (Answer: E) A)100 B)150 C)200 D)250 E)350

11. When water contains a lot of nutrients, the water quality is very good. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree

12. Most of the stream flow in the river comes from precipitation and the amount of water from groundwater recharge is so small that it can be ignored. (Answer: B)

A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree 13. Intercepting and treatment of sewage in a big centralized treatment plant before discharging into the

sea is the most advanced and effective way to treat waste water. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree

14. When rocks falling or sliding down slopes during heavy rain, it becomes debris flow. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree

15. Reclamation land from sea is an environmentally sound way of development. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree

16. Utilizing available water and geography and turning a grassland into a wetland for landscaping can benefit the ecosystem. (Answer: B)

A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree 17. The enhanced greenhouse effect due to climate change will cause temperature to rise globally and

most of the world will be threatened by draught. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree

18. In order to meet the ecological principles, engineers should design a permeable layer over the concrete pavement. (Answer: B)

A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree

Table 6 Improvement of Test Results after the E-Learning Courses No. of

Problem Correct Rate of Pre-Course Test

Correct Rate of Post-Course Tests

Increase (%)

1 59.63% 76.67% 17.04% 2 73.65% 78.94% 5.29% 3 62.68% 63.99% 1.31% 4 88.96% 89.69% 0.73% 5 52.17% 62.85% 10.69% 6 85.38% 87.20% 1.82% 7 83.32% 82.45% -0.87% 8 36.79% 56.74% 19.95% 9 29.40% 37.94% 8.54% 10 9.44% 13.14% 3.7% 11 86.75% 90.49% 3.74% 12 64.89% 70.55% 5.66% 13 32.67% 46.21% 13.54% 14 51.26% 57.76% 6.5% 15 86.60% 89.69% 3.36% 16 55.06% 56.96% 1.9% 17 36.94% 47.23% 10.29%

18 51.87% 58.21% 6.34%

Results of Course Evaluation

The design, organization and management of the program were evaluated with compulsory participant survey. As shown in Fig. 13, the participants give the e-learning program and the LMS very good rating. However, currently the quality of video is as satisfactory due to the limitation of the bandwidth on the users’ end and can be easily enhanced in the future. The most critical issue is that the interaction among the participants and/or with instructor is significantly less satisfactory. The limitation comes from the nature of asynchronous on-line course and the browser. The participants interacted with the instructor and among each other through discussion board and file sharing, which is far from spontaneous as face to face discussion. This particular result points out a very specific aspect for future improvement.

0% 10% 20% 30% 40% 50% 60%

Very poor

Rather unsatisfactory

Fair

Good

Excellent

Interaction ModelAV QualityWeb Site Designe Learning model

Figure 13 Rating of the E-Learning Program and the LMS Platform

When participants were asked whether they obtain enough useful knowledge on water

environment, more than 95% of the participants agree. Many participants think they obtained more knowledge than they expected as shown by comparing the results from questionnaire before and after the e-learning courses (Fig. 14). Statistics comparing the expectation and the self assessment of the participants regarding whether the courses will be helpful for their teaching also showed similar results (Fig. 15). Furthermore, more than 60% of the participants will utilize the knowledge on water environment and skills in their classroom for teaching in the future with a 60% or higher chances (Fig. 16).

0%

10%

20%

30%

40%

50%

60%

70%

80%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

ExpectationResult

Figure 14 Comparison between Results and Expectation on Knowledge Obtained

Finally, 97% of the participant expressed they would like to join on-line learning

program on other subjects. In general, although about 68% of participants had their first experience in e-learning by joining this program and become supporters of e-learning. Considering the difficulties of many of the participants had with the software, operating systems, hardware, and internet connections at the start of the courses, the quick acceptance and preference of e-learning programs are quite encouraging.

0%

10%

20%

30%

40%

50%

60%

70%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

ExpectationResult

Figure 15 Comparison between Results and Expectation of the Usefulness of the Courses

0%5%

10%15%20%25%30%35%40%45%50%

0-20% 20-40% 40-60% 60-80% 80-100%

20042005 12005 22006

Figure 16 Likelihood of Utilizing the Knowledge

Lessons Learned

There are several important lessons the project team learned from the program:

(1) Service manpower loading can be heavy The on-line course had a rough start since most of the participants did not have any

experiences such that they run into problems with their computer operating systems, internet browsers, or internet connections. Two LMS engineers were responsible to solve all the technical problems. Almost all of the technical difficulties were resolved quickly and some of them become the content of the website’s FAQ. As time went on, technical problems such as internet, operating system, and browser decreased. However, the loading of two staff members who served as teaching assistants remained as heavy as the start. Participants have all sorts of problems regarding asking register after closing date, how the course hours was recorded, how the certificate can be awarded, requesting for make up classes, and questions of the content of the courses or quizzes. This requires the staff constantly monitored the website and answered questions. Surprisingly, there is a distinct shift of user behavior over time.

Those who join earliest and most interested in water environment are most active all through the term including Saturdays and Sundays (Fig. 17). Moreover, they logged in the system weeks after the end of the term for discussion (Fig. 18). The later participants were much less interested and much less active, which in a way relieved some pressure on the staff.

0

1000

2000

3000

4000

5000

6000

7000

8000

Logi

n C

ount

.

Mon Tue Wed Thu Fri Sat Sun

Days

Week 7

Week 6

Week 5

Week 4

Week 3

Week 2

Week 1

A = 1378B = 1302Total = 2680Number = 1426

(a) 2005 1 E-Learning Term

0

2000

4000

6000

8000

10000

12000

14000

16000

Logi

n C

ount

.

Mon Tue Wed Thu Fri Sat Sun

Day

Week 7

Week 6

Week 5

Week 4

Week 3

Week 2

Week 1

A = 1487B = 1382C = 1180D = 1187Total = 5236Number = 1782

(b) 2006 E-Learning Term

Figure 17 2006 E-Learning Term Login Record Week Analysis

0

200

400

600

800

1000

1200

1400

1600

1800

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81

Day

Logi

n C

ount

.

A = 1378B = 1302Total = 2680Number = 1426

End

(a) 2004 E-Learning Term

0

500

1000

1500

2000

2500

3000

3500

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67

Day

Logi

n C

ount

.

A = 1487B = 1382C = 1180D = 1187Total = 5236Number = 1782

End

(b) 2005 2 E-Learning Term

Figure 18 E-Learning Term Login Record On the other hand, since most of the participants were school teachers, the time they

watched the on-line courses and joined the discussion was quite predictable. They started to login the system around 9 a.m. in the mornings and logout around 5 or 6 p.m. A lot of participants login around 7 p.m. to watch the courses and logout around 11 p.m. The engineers and staffs were available from 8 a.m. until 9 p.m. each day to manage the program and handle all the requests and problems of the participants.

0 2000 4000 6000 8000 10000 12000 14000

0 - 1

2 - 3

4 - 5

6 - 7

8 - 9

10 - 11

12 - 13

14 - 15

16 - 17

18 - 19

20 - 21

22 - 23

Hou

r .

Login

2005 12005 22006

Figure 19 Statistics of Login Hours

(2) Facilitation of interaction/discussion is important

Although more than 50% participants considered themselves to have more frequent communication than in classroom learning, interaction remains the least satisfactory part of the e-learning experience (Fig. 20) just as they expected before the program began. The knowledge transfer and time management is significantly better than interaction.

0% 10% 20% 30% 40% 50% 60%

Very poor

Rather unsatisfactory

Fair

Good

Excellent

InteractionTime managementLearning

Figure 20 Self-Assessment on E-Learning Ability

0% 5% 10% 15% 20% 25% 30% 35% 40%

Course subjects

Course induced

Environmental issues

Program Administration

Others Total = 266

Figure 21 Distribution of Discussion Categories of Topics

The statistics shows even worse. There were only 266 discussion subjects posted by the participants on the discussion board. The project manager and teaching assistants spent a lot of effort in responding the messages and keeping the discussion alive. These messages only got 1,211 responding posted messages and were read 45,219 times. In other words, most participants never actively interact with any fellow participants. Figure 17 shows the categories of discussion subjects posted. Most of them are related to the courses or environmental issues (Fig. 21). In addition, there were actually only 130 participants that are actively interacting with peers (Fig. 22). Interaction among participants remains to be an important issue to be addressed even though this was an asynchronous e-learning program.

0 20 40 60 80 100 120

1

2

3

4

5

6 to 10

11 to 15

> 15

Number of posts

Total = 130 person

Figure 22 Statistics of Active Participants

(3) Expanded use of the program is limitless

A lot of participants login the LMS and play back the on-line course in the geography, earth science, and social science class they are teaching and directly let the students watch the on-line course. Moreover, the courses have been rearranged to become formal on-line courses delivered in the universities and community colleges. Government agencies also requested the

project team to reorganize the courses to establish training programs for public servants and volunteer workers.

Conclusions

The water education innovative project is the pioneering in Taiwan. Vital experiences regarding the design and management of the program, the LMS, the on-line courses, and behavior of the learners were obtained.

It is shown that e-learning program can easily attract potential participants because it provides great flexibility in terms of time and place of learning. However, the nature of e-learning and the limitation of ICT up to now still can not overcome the less satisfactory while very, if not the most, important aspect of learning, i.e., interaction. Nevertheless, e-learning is still the most efficient method for large scale training programs such as the water education program described in this paper. However, supplemental face to face courses should be incorporated to augment the effectiveness of learning and also designing of the programs.

The final phase of the projects will be carried out in 2008. Fifteen new on-line courses will be developed and delivered, including 5 delivered in English to establish a module on Asian water environment. In addition, experiment with co-life on-line conferencing system to facilitate interaction will be incorporated into the program. The system will be used for case history discussion host by the instructors for experimental purpose. This system developed by NCHC allows 60 people to joint a discussion session simultaneously with full video and file presentation/exchange capacity. Application of new technologies will continued to be explored with the support from NCHC and other partners.

Acknowledgement

The authors thank the Ministry of Education, the Water Resources Agency, and the Environmental Protection Agency of Taiwan for the financial and administrative support. The authors also liked to thank all the members of the project team and the invited instructors for contributing their time and knowledge to this project.

References Berger, C., and Kam, R. (1996). Definitions of Instructional Design.

http://www.umich.edu~ed626/defin/html Accessed 28/06/2002

Dick, W., and Carey, L. (1978). The Systematic Design of Instruction. Glenview, IL.: Scott, Foresman.

Figueira, E. (2003). Evaluating the Effectiveness of E-Learning Strategies for Small and Medium Enterprises. European Seminars - Exploring models and partnerships for eLearning in SMEs

Gagne, R. M., Briggs, L. J., and Wagner, W. W. (1992). Principles of Instructional Design (4th ed.). New York: Harcourt Brace Jovanovich College Publishers.

Gerlach, V. S., and Ely, D. P. (1980). Teaching and Media: A systematic Approach. Englewood Cliffs, NJ: Prentice-Hall, Inc.

Hannifin, M. and Peck, K. (1988). The Design, Development, and Evaluation of Instructional Software. New York, NY: Macmillan.

Kemp, J. E. (1985). The Instructional Design Process. New York: Harper & Row.

Knirk, F. G. and Gustafson, K.L. (1986). Instructional Technology: A Systematic Approach to Education. Fort Worth, TX: Holt, Rinehart & Winston.

Microsoft. Microsoft Producer.

Microsoft. Microsoft Digital Rights Management.

Microsoft. Microsoft Streaming Technology.

Ministry of Education MOE. (2006). 2006 Education in Taiwan

Shankar B. Ponnekanti, F. Armando, P. Hanrahan, and T. Winograd I. (2001). Crafter: A Service Framework for Ubiquitous Computing Environments. Proceedings of Ubicomp 2001, September 30-October 2, 2001

Taiwan Network Information Center. (2006). Taiwan Broadband Network Usage Survey