A Meaningful Learning based u-Learning Evaluation Model

Po-Sheng Chiu1, Yen-Hung Kuo1, Yueh-Ming Huang1, Tzung-Shi Chen2, 1Department of Engineering Science, National Cheng Kung University, Taiwan

2Department of Computer Science and Information Engineering, National University of Tainan, Taiwan pschiu@easylearn.org, keh@easylearn.org, huang@mail.ncku.edu.tw, chents@mail.nutn.edu.tw

Abstract

In recent years, there has been a dramatic

proliferation of research concerned with the ubiquitous learning (u-learning). The u-learning systems have to be continuously evaluated and improved for ensuring the system reliability. Therefore, this work based on meaningful learning aspect to propose a u-learning evaluation model. The model blends features of u-learning and meaningful learning to construct a hierarchy decision model. According to the hierarchy structure, domain experts can develop AHP-based questionnaire survey to collect learners’ opinions. Following that, system developers can realize the relative strength and weakness of the u-learning system from a meaningful learning viewpoint by analyzing the surveyed data, and they can further to improve and refine current u-learning systems accordingly. Consequently, existing u-learning systems can be revalidated by our evaluation model, and then based on the produced suggestions to improve toward the meaningful learning. 1. Introduction

Advances in ubiquitous computing technologies have furnished teachers with the opportunity of engaging in novel educational processes. Along with these novelties, they not only affect our life but also shift our ways of learning. Nowadays, researches in terms of ubiquitous learning (u-learning) are also being stimulated by these changes. The u-learning environment provided an interoperable, pervasive, interactivity, and seamless learning architecture to integrate, connect, and share learning resources among appropriate identities [1][2]. The major characteristic of u-learning is context-aware that can sense the learners’ information and the information around the learners in the real world, and then to provide personalized services accordingly [3]. Therefore, learners can learn the knowledge, skills and problem solving abilities while interacting with the real world by authentic settings.

Last few years, the u-learning has been extensively

investigated and applied to domain knowledge training, such as history, linguistic, nature science, and so on[4]. For reliability, these u-learning applications have to be evaluated and improved constantly by learners, instructors and domain experts to ensure their academic performances [5][6]. To this end, a systematic evaluation method is needed to evaluate these u-learning works. Since u-learning has its own unique characteristics, it cannot direct apply previous evaluation methodologies which were used to evaluate e-learning systems, to evaluate it. In order to objectively evaluate u-learning systems, this study bases on meaningful learning and introduces a systematic evaluation approach for u-learning environment. A learning activity/system conforms to meaningful learning must keep following principles [7][8]: active, authentic, constructive, cooperative, and integrated, which are the fundamentals of the meaningful learning and are considered as criteria to evaluate u-learning systems. Based on these criteria, we can have an insight into whether the ubiquitous computing technologies can play a harmony with pedagogies in u-learning environment.

In this study, we have developed an evaluation model based on both the principles of meaningful and ubiquitous learning. In order to take the two sets of principle into account simultaneously, the analytic hierarchy process (AHP) [9] has been utilized to perform a systematic fusion. The AHP is usually adopted to perform multi-criteria decision making. In addition, during the decision process, the critical/relative important criterion which can affect the decision goal can be found. Based on such property, we raise a research hypothesis to help us to develop a u-learning evaluation method by using AHP model. It assumes that the learners thought one system function is relative important means that the design of the function is better than other system functions, and vice versa. Based on the hypothesis, u-learning developers can realize the relative strength and weakness of their u-learning systems according to the evaluation model. Moreover, the evaluation result will also indicate the clearly refinement directions for developers to work toward better meaningful learning based u-learning environment.

Eighth IEEE International Conference on Advanced Learning Technologies

978-0-7695-3167-0/08 $25.00 © 2008 IEEEDOI 10.1109/ICALT.2008.100

77

In the final stage of this study, a case study, the ubiquitous tree-watching activity, is developed and conducted to make this work more comprehensible. During Mar.-May 2007, six teachers and sixty-one fifth grade students in an elementary school in southern Taiwan were engaged in the ubiquitous tree-watching activity and they were then surveyed by our AHP-based questionnaire. The evaluative results show that the tree-watching activity is performed better in authentic and integrated dimensions of the meaningful learning, but the cooperative is relatively weak and needs developers to pay more efforts for improvement.

The remainder of this article is organized as follows. In Section 2, we survey the relevant literatures. According to previous results, Sections 3 introduces our evaluation model, and the corresponding case study has been drawn in Section 4. Subsequently, Section 5 shows the evaluation result derived from the case study, and finally, Section 6 shows the research limitations and sums up this study, as well as pointing out the future direction. 2. Literature review 2.1. Ubiquitous learning

Many researches considered utilizing context-aware and ubiquitous computing technologies in learning environments that had explicit learning goals to encourage the motive and performance of learners. In summary, the main characteristics of u-learning are described in following [1][2][3][4][10]: Urgency of learning need: The u-learning

environment can be used for an urgent matter of learning.

Initiative of knowledge acquisition: The u-learning systems can provide the information, which closes to learners’ requests in time.

Interactivity of learning process: Learners can communicate with peers, teachers, and experts effectively through the interfaces of u-learning systems.

Situation of instructional activity: In u-learning environment, the learning process can be embedded in daily life, as well as the knowledge requirements are presented in authentic context.

Context-awareness: The u-learning environment is context-awareness, which based on learners’ statuses or the situations of the authentic environment to provide the related information to learners.

Actively provides personalized services: Based on the context around learners, the u-learning systems would actively provide personalized supports to

learners by the right way, in the right place, and at the right time.

Self-regulated learning: Self-regulated learning is worth to discuss in the u-learning activity, in which learners could actively control their learning progresses by themselves. Moreover, such learning activities can also bring up learners’ self-regulated abilities.

Seamless learning: The u-learning environment enables seamless learning at anywhere and anytime. The learners are allowed to learn without being interrupted while moving from place to place.

Adapt the subject contents: The u-learning environment is able to adapt the subject contents to suit the capability of various learning devices.

Learning community: The u-learning environment help online community with bring field experience on the Internet to enrich the learning interaction between learners and teachers.

2.2. Meaningful learning

In this study, the attributes of meaningful learning are formulated for designing and evaluating u-learning environments. According to Jonassen [7] and Grabe et al. [8], the characteristics of the meaningful learning and the corresponding advantages are summarized hereinafter: Active: The learners are organisms to interact with

the environment, in which they processed their learning and monitor the leaning process. Therefore, the learners are dynamic roles in the learning activities.

Authentic: Learning in authentic environment as that resorts to learning tasks which are situated in meaningful real world tasks.

Constructive: Constructive learning means that learners accommodate new ideas into their prior knowledge/experiences.

Cooperative: Working in knowledge building community makes it possible for learners to exploit each other’s skills and provide social support and modeling for other learners.

Integrated: Content knowledge and technology should be integrated to furnish teaching/learning process with smooth and vivid applications.

3. Evaluation model for u-learning

To achieve our evaluation goal: Evaluating the consistency between u-learning system and meaningful learning. The evaluation model is constructed from the analysis of literature by exploring the factors of meaningful learning and characteristics of u-learning.

78

To build a hierarchical decision structure, we utilize the factors of meaningful learning as the dimensions and then adopt the u-learning features as the criteria to embody the dimensions. The built hierarchy decision structure is shown in Figure 1. Based on the built model, the AHP-based questionnaire survey can be developed by domain experts and teachers, and the built questionnaire can then be utilized to collect learners’ opinions. Through the questionnaire survey result, system developer can test the reliability of u-learning systems and to explore the weight ratio of dimensions and criteria from the meaningful learning perspective. Additionally, the higher weight ratio means the dimension or criterion is more important than other factors, and also indicates the factor adopted more appropriately (research hypothesis).

Figure 1. The hierarchy structure for evaluating

u-learning environment The proposed evaluation model can not only work

alone but also work as a partial evaluation function with other existing frameworks. As shown in Figure 2, the waterfall model presents a straightforward view of the system development life cycle, which includes five stages: analysis, design, code, test and maintenance [11]. Our evaluation model can be applied to the analysis and test stage of the system life cycle. During the analysis stage, the u-learning system functions/prototype and designs of learning activity can be introduced to learners by slides or multimedia presentation before implementation. Subsequently, applying our evaluation model to collect learner’s opinions, and then discovering the latent defects of the u-learning system by analyzing the learners’ feedbacks. Such application can avoid redundant or inappropriate design before implementing the u-learning system, and further to assist developers in working toward a meaningful learning based u-learning system. Similarly, we can apply the evaluation model to the test stage of

system life cycle, thus the evaluation results can indicate a clear refinement direction for system designers to let the u-learning system accommodate to meaningful learning.

Figure 2. Waterfall lifecycle model [11]

4. Case Study Application in Ubiquitous Tree-Watching Activity

In order to make this work more comprehensible, in this section, we are going to introduce a case study to explain how to apply our evaluation model to a real world case: The ubiquitous tree-watching activity.

In this case study, authors have built a ubiquitous tree-watching environment in an elementary school in southern Taiwan. The participants consist of 61 fifth grade students, and the investigation period was 2 months, spanning from Mar – June, 2007.

Figure 3. The u-learning environment structure Figure 3 shows the tree-watching u-learning

environment, where the learning management system

79

(LMS) is responsible for providing necessary materials form backend database to particular learning activities while preserving learning behaviors into Learner Portfolio. In addition, we attached RFID tags on trees and assigned PDAs (with RFID reader) to learners. When learners explore the environment by using PDAs, the RFID tags can furnish LMS with learners’ location information. According to the location data, the LMS can further transmit the trees’ (which are near learner) information/problems to learners for learning/solving the concept/quiz of the trees, especially for the trees’ problems, which may ask learners for exploiting their peer’s (cooperation) knowledge to accomplish the task. Moreover, through field observation, learners’ impressions of the natural environment can be deepened, so that the tree-related knowledge can be formed in learners’ minds. Figure 4 shows the situations that the learners concentrated on the observation of the tree.

Figure 4. The ubiquitous tree-watching activity Following explains that how the proposed

ubiquitous tree-watching activity can accommodate to meaningful learning. The theory of meaningful learning indicates that knowledge exists in the real world environment [7][8], where is the situated context the learner has to enter and acquires knowledge. Engaging in tree-watching learning activities, learners will have realistic interactions with the environment, therefore the authenticity of the learning activities can be achieved. Furthermore, the knowledge which has to be leant is presented in authentic context, and the new information is accommodated to their prior tree knowledge, the authentic and constructive factors of meaningful learning are performed accordingly. In the u-learning environment, learners can instantly receive prompts and explanations to facilitate their learning through their own PDAs. In addition, the system can also actively provide contextual materials to learners by depending on their identification, location, and time. Since content knowledge can be integrated with the u-learning environment, the learners can learn

knowledge without being interrupted while moving from place to place. Such feature achieves integrate. The ubiquitous tree-watching activity can provide the learning tasks or problems to learners for exploiting peers’ skills. Sequentially, the learners were organisms to interact with their peers and the environment, in which they processed their learning and monitor the learning process. Above achieves the cooperative and active of meaningful learning. Finally, through the interactions among learners, learning systems, and the authentic natural environment, learners’ knowledge about trees can be progressively built up.

Based on the settings proposed in the section, next section introduces the results of using the proposed evaluation model to evaluate the tree-watching u-learning environment. 5. Results of Case Study

The research data were collected through the AHP questionnaire survey, which was developed by domain experts (natural science experts) and elementary school teachers who teaches natural science. Tables 1 and 2 show the weights of dimensions and criteria respectively. Note that all results obtained by AHP have passed the consistency test (a value of consistency ration less than 0.1).

Considering Table 1, it found that learners regard the authentic as being the most important dimension and the cooperative as the least important dimension. Table 2 shows that the context-awareness is the most important criteria, and the learning community is relatively unimportant. According to our research hypothesis, which the characteristics of u-learning are regarded by the learners as having least relative importance that would be considered the defect of the function in the u-learning environment. Consequently, the results pointed out cooperative dimension and learning community criteria in this case study would be suggested to be improved toward more conform to the meaningful learning.

Table 1. Weights of dimensions # Dimension Weight Consistency ratio 1 Active 0.181 0.003 2 Authentic 0.248 3 Constructive 0.182 4 Cooperative 0.166 5 Integrated 0.233

80

Table 2. Weights of criteria # Dimension Criterion Local weight Overall weight Consistency ratio 1 Active Urgency of learning need 0.477 0.073 0.001 2 Active Initiative of knowledge acquisition 0.523 0.079 3 Authentic Situation of instructional activity 0.489 0.121 0.001 4 Authentic Context awareness 0.501 0.123 5 Constructive Actively provides personalized 0.486 0.080 0.001 6 Constructive Self-regulated learning 0.514 0.084 7 Cooperative Interactivity of learning process 0.576 0.073 0.002 8 Cooperative Learning community 0.424 0.061 9 Integrated Seamless learning 0.498 0.112 0.001

10 Integrated Adapt the subject contents 0.502 0.113 6. Conclusions

Since u-learning has its own unique characteristics, it cannot direct apply previous evaluation methodologies which were used to evaluate e-learning systems, to evaluate it. In order to objectively evaluate u-learning systems, this study based on meaningful learning to introduce a systematic evaluation approach for u-learning environment. Particularly, it blends the features of the u-learning and the meaningful learning to construct a hierarchy decision model, and then based on the hierarchy structure to develop the AHP-based questionnaire survey for collecting learners’ responses about the u-learning environments. Through the evaluation results, the system developers can realize the relative strong/weak design of the u-learning system from the meaningful learning point of view. Consequently, the developers can then pay their effort to improve the weakness or to enhance the strength of present u-learning system toward the meaningful learning manner.

Although the evaluation results can reveal the relative strength/weakness of dimensions and criteria, we cannot realize the overall u-learning system performance of meaningful learning from an absolute aspect. While this study has this research limitation, it has still raised a comprehensive method to evaluate the consistency of u-learning systems and meaningful learning. Our further research direction is to expand current u-learning evaluation method to propose a novel and reliable evaluation methodology, which takes into account both the relative and absolute performances of meaningful learning simultaneously. 7. Acknowledgement

This work is supported by National Science Council, Taiwan under grant NSC 96-2524-S-032-001. 8. References

[1] S.J.H. Yang, "Context Aware Ubiquitous Learning Environments for Peer-to-Peer Collaborative Learning," Educational Technology & Society, Vol. 9, Issue 1, 2006, pp. 188-201.

[2] G.J. Hwang, "Criteria and Strategies of Ubiquitous Learning," in Sensor Networks, Ubiquitous, and Trustworthy Computing, 2006. IEEE International Conference on, 2006, pp. 72-77.

[3] H. Ogata and Y. Yano, "Context-Aware Support for Computer-Supported Ubiquitous Learning," in Wireless and Mobile Technologies in Education, JungLi, Taiwan, 2004, pp. 27-34.

[4] G.D. Chen, C.K. Chang, and C.Y. Wang, "Ubiquitous Learning Website: Scaffold Learners by Mobile Devices with Information-Aware Techniques," Computers & Education, Vol. 50, No. 1, 2008, pp. 77-90.

[5] L.F. Motiwalla, "Mobile Learning: A Framework and Evaluation," Computers & Education, Vol. 49, No. 3, 2007, pp. 581-596.

[6] T. Hall and L. Bannon, "Designing Ubiquitous Computing to Enhance Children's Learning in Museums," Journal of Computer Assisted Learning, Vol. 22, Issue 4, 2006, pp. 231-243.

[7] D.H. Jonassen, "Supporting Communities of Learners with Technology: A Vision for Integrating Technology with Learning in Schools.," Educational Technology, 1995, pp. 60-63.

[8] M. Grabe, and C. Grabe, Integrating technology for meaningful learning (5th ed.), New York, NY: Houghton Mifflin Company, 2007.

[9] T.L. Saaty, The Analytic Hierarchy Process. New York: McGraw-Hill, 1980.

[10] Y.S. Chen, T.C. Kao, and J.P. Sheu, "A Mobile Learning System for Scaffolding Bird Watching Learning," Journal of Computer Assisted Learning, Vol. 19, Issue 3, 2003, pp. 347-359.

[11] F.P. James and P. Witold, Software Engineering: An Engineering Approach. New York, NY, USA John Wiley & Sons, Inc., 1998.

81