- 1 -

Contents

Part I Rationale...........................................................................................................................2Part II Conceptual Framework....................................................................................................3

1. Algebraic skills...................................................................................................................52. ICT tool use.........................................................................................................................83. Assessment........................................................................................................................144. Integrating theory..............................................................................................................225. Choosing content that makes symbol sense......................................................................236. ICT Tools for assessment..................................................................................................257. Designing the prototype and instruction...........................................................................29

Part III Methodology.................................................................................................................30Appendix A...........................................................................................................................32Appendix B...........................................................................................................................36Appendix C...........................................................................................................................38

DITwis..............................................................................................................................38Algebra Tutor....................................................................................................................39Calmaeth...........................................................................................................................40Math Xpert:.......................................................................................................................41Aplusix..............................................................................................................................42L’Algebrista......................................................................................................................43webMathematica...............................................................................................................44Wiris..................................................................................................................................45AiM: Assessment in Mathematics....................................................................................46CABLE.............................................................................................................................47Hot potatoes......................................................................................................................48Question Mark Perception................................................................................................49Wintoets............................................................................................................................50Moodle quiz module with extensions...............................................................................51Wallis................................................................................................................................52WebWork..........................................................................................................................53TI interactive.....................................................................................................................54Cognitive Tutor.................................................................................................................55Algebra Buster..................................................................................................................56

Appendix D...........................................................................................................................57Maple TA..........................................................................................................................57Digital Mathematical Environment...................................................................................58Activemath........................................................................................................................59STACK.............................................................................................................................60Wims.................................................................................................................................61

Appendix E...........................................................................................................................62Appendix F............................................................................................................................65

References.................................................................................................................................69

- 2 -

Part I Rationale

For several years now the skill level of students leaving secondary education in the Netherlands has been questioned. Lecturers in higher education often complain of an apparent lack of algebraic skills, for example. I was personally confronted with this challenge when I redesigned the entry exam of the Free University from 2001-2004. This problem has only grown larger in the last few years. In 2006 a national project was started to address and scrutinize this gap in mathematical skills, called NKBW. In the same period use of ICT in mathematics education has also increased. It is our conviction that ICT can be used to aid bridging this gap.

Therefore this research will focus on two relevant issues in mathematics education in secondary schools in the Netherlands: on the one hand signals from higher education that freshmen students have a lack of algebraic skills, on the other hand the use of ICT in mathematics education.

Relation to current curricular developments in math educationThese developments have to be seen in a larger context. In 2007 the cTwo commission (commission on the future of mathematics education) published a vision document (2007) which has all the ingredients for this research.

First of all the importance of numbers, formulas, functions, change, space and chance are stressed (viewpoint 4). On an algebraic level this corresponds with the sources of meaning (Radford, 2004) for algebra. Activities are: modeling, manipulating formulas.

Also, the role of ICT in this process is described (viewpoint 10). ICT should be "use to learn" and not "learn to use". This strict dichotomy will be difficult to accomplish, as they go hand in hand. This will be elaborated on in the chapter on tool use.

In viewpoint 14 a specific case is made for the transition of students from secondary education towards higher education. Again, it is stressed that this transition needs more attention.

Viewpoint 15 stresses the importance of assessment of algebraic skills.

Finally, viewpoint 16 mentions the pen-and-paper aspect of mathematics.

Why with a computer tool?But why should we use a computer in learning algebra? We contend that computers can aid understanding of algebra by providing a learning environment that enables you to practice algebra anytime, anyplace, anywhere, because:

- Randomization of exercises means there are many more questions. - It is possible to use several representations- The applets can be used anyplace, anytime, anywhere. - Automated feedback can help in this process- Students tend to be more motivated

We will elaborate on this in our conceptual framework.

- 3 -

Part II Conceptual Framework

This research focuses on the question:

In what way can the use of ICT in secondary education support learning, testing and assessing relevant mathematical skills?

First it is useful to analyze our question word-for-word.

In what way. To us it is not a question whether ICT can be used to support learning, testing and assessing mathematical skills, but how this should take place.

Secondary education. In this research we focus on upper secondary education and in particular students preparing to go on to higher education.

Learning, testing and assessing. Not only grades and scores are important, but also the way in which mathematical concepts are learned and tested diagnostically. We specifically aim to find out more about all three aspects.

Relevant mathematical skills. When students leave secondary education they are expected to have learned certain skills. Here we focus on algebraic skills, with particular attention given to “real understanding of concepts”, symbol sense.

So following a pragmatic approach three key issues are part of this research question: skills, assessment and ICT tool use.

The structure of part II is as follows:

First we discuss the three key concepts algebraic skills, assessment and tool use in chapters 1, 2 and 3. Every section starts with a problem statement, then gives an overview of relevant literature and ends with some words on the implications for my research.

In chapter 4 we integrate these concepts into one framework for my research.

Based on this conceptual framework two major decisions have to be made:- Which ICT tool to use for assessment. For this we will formulate criteria based on the

conceptual framework and give an overview of available ICT tools for assessment.- What content to use for learning, testing and assessing algebraic skills. Per question

we will motivate why the question is relevant for this research.In chapters 5 and 6 these two decisions are explicated.

Together they will make up the design principles for our first prototype, which will be summarized in chapter 7. In part III we then discuss the methodology we use

- 4 -

Rationale

Research question(s)

Algebraic skills(chapter 1)

Tool use(chapter 2)

Assessment(chapter 3)

Content choice(chapter 5)

Tool choice(chapter 6)

Prototypical design(chapter 7)

Methodology

Part I

Part II

Conceptual framework(chapter 4)

- 5 -

1. Algebraic skillsIn this chapter we focus on algebraic skills and symbol sense. For this it is important to sketch a general outline of the subject at hand. In recent years

A. Problem statementAlgebraic skills of students are decreasing. We want to make sure that students really understand algebraic concepts, so just testing basic skills is insufficient. What defines real algebraic understanding?

B. Theoretical overviewIn a historical context al-Khwarizmi, Vieta and Euler considered algebra to be a "tool for manipulating symbols and for solving problems." In the 80s Fey and Good (1985) argued that the "function concept is at the heart of the curriculum". More recently Laughbaum (2007) sees ground for this statement in neuroscience.

To get a clear picture of algebraic skills and the purpose of algebra we have to look into the theoretical foundations.

Meaning of algebraRadford (2004) sees several sources of meaning in algebra:

1. Meaning from within mathematics, which can be divided into:

(a) Meaning from the algebraic structure itself, involving the letter-symbol form. This is also referred to as "structure of expressions" or “structure sense” (Hoch & Dreyfus, 2005). I would like to use the term " symbol sense" here, in line with Arcavi (1994) and Drijvers (2003).

(b) Meaning from other mathematical representations, including multiple representations. This corresponds with the "multirepresentational" views of Janvier (1987), Kaput (1989) and van Streun (2000)

2. Meaning from the problem context.3. Meaning derived from that which is exterior to the mathematics/problem context

(gestures, bodily movements, words, metaphors, artifacts use)

Ideally, all these sources would be addressed in an instructional sequence.

To focus more on the actual concepts that are learned Kieran's (1996) GTG model combines several theories into one framework. In this model three activities are distinguished: Generational, Transformational and Global/Meta-level activities.

In upper secondary and college level these activities apply:

Generational activity with a Primary focus on the letter-symbolic form: form and structure (Hoch & Dreyfus, 2005) and parameters.

Generational activity with multiple representations: functions and their meaning, symbolic and graphical representations hand in hand.

Transformational activity related to notions of equivalence. Transformational activity related to equations and inequalities.

- 6 -

Transformational activity related to factoring expressions. Transformational activity involving the integration of graphical and symbolic work.

Global/Meta-level activity involving problem solving Global/Meta-level activity involving modelling.

Algebraic activities in schoolIt is essential to have a clear view on what activities in secondary education have to with algebra. A non-limitative list of activities include:

implicit or explicit generalization investigation of patterns and numerical relations problem solving though applying general or specific rules reasoning with unknown or undetermined quantities arithmetic operations with literal variables symbolizing numerical operations and relations tables and graphs represent formulas and are used to investigate them formulas and expressions are compared and transformed formulas and expressions are used to describe situations in which measures and

quantities play a role solution processes contain steps based on rules, but without meaning in the context

Grouping these activities one can distinguish two dimensions of algebraic skills: basic skills, including algebraic calculations (procedural) and symbol sense (conceptual). The latter is “actual understanding” of algebraic concepts.

One can not do without the other. Both should be trained, making use of several influential models on learning mathematics.

Or as Zorn (2002) puts it: "By symbol sense I mean a very general ability to extract mathematical meaning and structure from symbols, to encode meaning efficiently in symbols, and to manipulate symbols effectively to discover new mathematical meaning and structure."

Symbol SenseThe notion of “actual understanding” of mathematical concepts has been given different names. Hoch called this "structure sense"  at the beginning of 2003. Arcavi (1994) used the term "symbol sense" , analogue to the term "number sense". It is an intuitive feel for when to call on symbols in the process of solving a problem, and conversely, when to abandon a symbolic treatment for better tools.

Drijvers (2006) sees an important role for both basic skills and symbol sense. The declining algebraic skills of students is concerning. As Tall and Thomas (Tall & Thomas, 1991) put it:

Algebraic Skills

Basic skills: algebraic calculation, procedural routine

Symbol sense: algebraic reasoning, strategic skills (Arcavi, 1994)

- 7 -

"There is a stage in the curriculum when the introduction of algebra may make things hard, but not teaching algebra will soon render it impossible to make hard things simple."

Several problems with symbol sense are:

process-object duality: a student thinks in terms of activity rather than objects. visual properties of expressions lack of flexibility lack of meaning of algebraic expressions lack of exercises

Building on this last observation Kop and Drijvers (Kop & Drijvers, in press) have suggested a categorization of “symbol sense” type questions. This source will –together with other sources- provide a starting point for designing a prototype.

Impact of technologyTechnology has an impact on mathematics education. Research with calculators (Ellington, 2003) has shown that the pedagogical role of tool use should not be underestimated. The use of tools seems to strengthen a positive attitude towards education, showing that there is more to learning than just practicing and testing. van Streun (2000), Lagrange, Artigue, Laborde and Trouche (2001) all determined enriched solution repertoires and a better understanding of functions, especially through the use of multiple representations. However, use should not be haphazard, but for prolonged use.

The next step in using tools for algebra was in the use of Computer Algebra Systems (CAS). The first large-scale study on the use of CAS was by (1997)

It is also important to stress the changing roles of students and teachers. Guin and Trouche (1999) noticed that students have different "styles" of coping with problems: random, mechanical, rational, resourceful and theoretical.

The modes of graphing calculator used by Doerr and Zangor (2000) could also be applied to the use of applets: computational, transformational, visualizing, verification and data collection and analysis tool.

Finally, the advent of computing technology has also strengthened believe that multiple representations of mathematical objects could be fully integrated in mathematics curriculum. This could provide a valuable source of implicit feedback, making sure that the added value  of (formative) assessment could be greatly enhanced.

According to Lester (2007) three factors are important in technology-related studies concerning algebra: time, the nature of the task and the role played by the teacher in orchestrating the development of algebraic thought by means of appropriate classroom discussion. One extra factor has to do with the instrumental genesis of the tool used. Transfer of what has been learned has to take place. Therefore the relation between tool use and pen-and-paper has to be taken into account. More on this in the second chapter.

C. Algebraic skills in this researchWe want to study whether algebraic skills, and in particular symbol sense, can be improved by using an ICT tool.

- 8 -

2. ICT tool useIn this chapter I focus on the use of ICT1 tools in (mathematics) education.

A. Problem statementIt is important to study the way in which tools can be used to facilitate learning. How are tools used and what characteristics do they have to have.

B. Theoretical overviewFirst I will sketch a general overview, ending in a description of the instrumental approach of tool use. Here I will use the construct of figure 1, looking at how tools affect aspects of the teaching and learning of mathematics.

From the Second Handbook of Research on Mathematics Teaching and Learning (Lester jr., 2007)

Not only mathematical activity, students, teachers and curriculum are affected, also the relationship between these aspects.

Technology and Mathematical activity Many people use tools all the time. The Vygotskian notions on tool use (Vygotsky, 1978) sees a tool as a mediator, a " new intermediary element between the object and the psychic operation, directed at it" . In mathematics, tools have to have certain characteristics to be beneficial. The Handbook on Research mentions three important issues:

Externalization of representations

Heid (1997) also mentions this. The important question remains: how is mathematical activity influenced or changed by tool use? Feedback is mentioned. Otherwise time-consuming "production work" as well. Unlike the physical tool a cognitive tool provides a "constraint-support system" (Kaput, 1992) for mathematical activity.

Mathematical fidelity

A representation must be faithful to the underlying mathematical properties; this is mathematical fidelity (Dick, 2007). In essence this means that a tool can represent maths incorrectly. This also has to do, in my opinion, with the difference between "use to learn" and "learn to use" (cTwo, 2007), as the latter means one has to know the shortcomings of a tool,

1 Information and Communication Technology. When use “tools” I mean “ICT tools”.

Student

Teacher

MathematicalActivity

CurriculumContent

Tool

- 9 -

but this knowledge could also lead to a better understanding of a concept, thus a tool is used to learn.

Another aspect is the underlying machine code for a certain tool. It often is the case that certain extreme values yield strange results. So an important question is: "is the mathematical fidelity of a math system good enough to support maths at secondary school?". It will almost surely be a trade-off between this and the amount of time needed to improve the system.

Cognitive fidelity

This is the "degree to which the computer's method of solution resembles a person's method of solution.". To make sure that transfer of knowledge or skill takes place

Technology and students

An important distinction in type of activity is between exploratory and expressive tools and activities (Bliss & J., 1989). They reside on a continuum. So when a procedure is described it´s exploratory but choosing one’s own procedure is expressive (albeit somewhat limited). Initial play with a technological tool is often beneficial: it stimulates expression but also builds a purposeful relationship with the tool , and thus instrumental genesis (Guin & Trouche, 1999) can take place. However, structured guidance is often necessary, as to avoid the "play paradox" (Hoyles & Noss, 1992). This means that " playing" with a tool sometimes enables students to accomplish an activity without learning the intended concepts. To solve the paradox "reflection" on the task at hand is advised.

When studying student use of a tool the construct of a 'work method' could work: Guin and Trouche (1999) see five work methods: random work method, mechanical work method, a resourceful work method, a rational work method and a theoretical work method.

The combination of the type of activity (exploratory, expressive) and work method should enable us to deduct what students are thinking.

Technology and practice

In it important that there is pedagogical fidelity in tool use. This means that students actually learn what the teacher has intended. So here we consider the match between technology and practice..

An interesting choice is whether sometimes "privileging" is appropriate: using tools when basics are known and rules are "internalized" (in a sense trivialized). Before that, tool use is prohibited. The concept of privileging can also apply to certain mental activities, like proofs etc. This coincides with the white box versus black box discussion (Buchberger, 1989), which states that “privileging” with tools –meaning that tools may only be used when a concept is understood- is necessary. This prevents students from just “executing an algorithm" without knowing what they are doing.

Using technology in practice also means that the teacher role could change. This is also an aspect that can be studied throught the construct of teacher role, e.g. Counselor and Technical Assistant (Zbiek & Hollebrands, 2007).

- 10 -

The use of technology changes this role. But: this could very well clash with teacher's expectations. Beaudin and Bowes (1997), and later Zbiek and Hollebrands introduced the PURIA model for CAS implementation:

personal Play, personal Use, Recommendation, Implementation, and Assessment.

This could also be applied to the implementation of DME2 use in the Netherlands.

Technology and curriculum

There are several reasons why technology is adapted:

Representational fluency: technology makes it possible to move easily between several representations. This also belongs to good design principles for technological environments (Underwood et al., 2005). For example, the applet Algebra Arrows has a multirepresentational aspect. One could think of the distinction: context-table-graph-formula.

Mathematical concordance is another construct looking at the level intended and real knowledge building are the same or not. In analyzing the way that teachers and students interact with cognitive tools, it is helpful to consider the mathematics of the tool, the mathematics of the teacher, the mathematics the teacher intended through particular technology-based activities, the mathematics that the student engaged in as a result of the technology-based activity arranged by the teacher, and the mathematics that is learned.

Amplifiers and Reorganizers can be used to describe curricular roles of technology. (Pea 1985) Amplifiers accept the goals of the current curriculum and work to achieve goals better. Reorganizers change the goal of the curriculum or the way the goals are obtained by replacing, adding and reordering parts. I would say the WELP3 project would be an amplifier.

“The French school”

In the early 90s the use of Computer Algebra Systems was seen as a possible means to get rid of manipulations (routine skills) and focus more on concepts and complex problem solving.

Artigue, in “the French school”, noticed that actual tool use should be scrutinized, as to discover obstacles and difficulties in the classroom. Her thoughts in the 90s on this become clear in this quotation:

“... we needed other frameworks in order to overcome some research traps that we weremore and more sensitive to, the first one being what we called the “technical-conceptualcut”. Indeed, theoretical approaches used at that time ... tended to use this reference toconstructivism in order to caution in some sense the technical-conceptual cut, and wefelt the need to take some distance from these. Artigue (2002, p.247)” 

2 Digital Mathematical Environment, http://www.fi.uu.nl/dwo/en/3 Wiskunde En Les Praktijk, dutch project that was aimed at using applets in an algebra curriculum

- 11 -

Two approaches were combined to overcome these traps. On the one hand Vérillon and Rabardels work on instrumentation ((1995), the ergonomic approach) and on the other hand Chevallards anthropological approach (Task, Techniques used to solve Tasks, Technology or Talk used to explain and justify Techniques, and Theory).

Instrumental approach

The instrumental approach of tool use is easily summed up In this research the instrumental approach of tool use is more important:

Instrument = artefact + instrumentation scheme.

Verillon and Rabardel (1995) distinguish an artifact and an instrument. An artefact is only the tool. An instrument is the psychological notion: the relationship between a person and an artifact. Only when this relationship is established one can call this a "user agent". The mental processes that come with this are called schemes.

Instrumental genesis is the process of an artifact becoming an instrument. In this process both conceptual and technical knowledge play a role (again, "use to learn" and "learn to use"). In instrumental genesis three aspects come together: task, theory, technology. They are closely connected.

Trouche (2003) distinguishes a tool component with instrumentation (how the tool shapes the tool-use) and instrumentalisation (the way the user shapes a tool), and also a psychological component with schemes (Piaget & Inhelder, 1969) . According to several studies (Artigue, (2002); (Guin, Ruthven, & Trouche, 2005) genesis for computer algebra systems is a timeconsuming and lengthy process.

When focusing on particular aspects of instrumental genesis, for example instrumentation, instrumentalization and technique (Guin & Trouche, 1999), it becomes more clear how students can use tools more effectively and what obstacles hinder conceptual and technical understanding of a tool. Trouche sees three functions: a pragmatic one (it allows an agent to do something), heuristic (it allows the agent to anticipate and plan actions) and epistemic (it allows the agent to understand something).

Also instrumental orchestration concerns the external steering of students’ instrumental genesis (Guin & Trouche, 1999)

So there are also conceptual aspects within the cognitive instrumentation schemes. The instrumental approach provides a good framework for looking at the relation between tool use and learning from an individual perspective. Yackel and Cobb (1996) argued that coordinating both perspectives is expected to explain a lot on the advent an use of computer tools. Tool use and learning is especially apparent in mathematics education. Integration in the classroom is essential, and to understand this we need to observe instrumentisation.

Anthropological approach

As tools are used in practice, a context, in activity one’s view on practice becomes important. In the anthropological approach we discern task, techniques, technology and theory:

- 12 -

Technology and theory can be defined as knowledge per seTask and technique: know-how relevant to a particular theory and technology

So, Artigue and Lagrange focus primarily on Task and Technique. Also, the distinction between, for example, pragmatic (efficient) techniques for doing tasks, and epistemic techniques (more focused, as I see it, on concepts on real symbol sense)

As Artigue says: “Professional worlds as well as society at large have a pragmatic relationship with computational tools: their legitimacy is mainly linked to their efficiency. But what schools aim for ... is much more than developing an effective instrumented mathematical practice. The educational legitimacy of tools for mathematical work has thus both epistemic and pragmatic sources: tools must be helpful for producing results but their use must also support and promote mathematical learning and understanding. (Artigue, 2005)”

 So use to task, theory and technique go hand in hand. Again citing Lagrange (1999):

“The argument for this is essentially in five parts:1. Technical work does not disappear when doing mathematics with CAS, it is

transformed.2. Within a theory, every topic has an accompanying set of tasks and techniques.

Novices progressively become skilled in techniques by doing, talking about, and seeing the limits of techniques. This eventually leads to a theoretical understanding of the topic.

3. Although rote repetition of a specific technique for a specific task is a mathematically impoverishing experience, this is not a reason to jettison techniques per se.

4. Techniques and schemes are linked. Students need time to develop rich schemes by using techniques.

5. The empirical observation that diminishing the role of techniques encourages teachers to avoid talking about them (Chevallard’s “technology”).”

 Monaghan (Monaghan & Ozmantar, 2006) points out several issues in “the French school”, for example the tension in the “technique”. He wonders whether the two approaches could be integrated. Two missing aspects in French theory are teachers and affect. Concerning the first: one could say that orchestration has to do with teachers. This however seems more concerned with “what can be done” and not so much a description of teachers’ practices. Perhaps it would be good if studies would focus more on developing accounts of teachers coming to instrumental genesis. Concerning affect: as it surely has a large impact on tool use (beliefs, attitudes, motivations, emotions) it should have more attention. Monaghan (2005) also has argued that de notion of schemes could be cut by looking towards “activity theory”:

“Activity theory considers the actions of a person towards an objective (affective motives are thus essential). Activity is mediated through artifacts, social procedures, and language, and Trouche may find the ideas of mediation relevant to his considerations of orchestration.”So here we see common ground between two frameworks: mediation can take place through artifacts, procedures and language, corresponding well with the notion of orchestration.

- 13 -

C. ICT tool use in this researchWe claim that tools can facilitate in the learning of algebraic skills. Therefore we want to study how instrumental genesis takes place when using a prototype.

- 14 -

3. Assessment In this chapter we look at the role of assessment in learning..

A. Problem statementHow can we efficiently make use of assessment for learning, and in particular for learning algebra?.

B. Theoretical overviewIntroductionBlack and William (1998) have argued for many years now that assessment should be a part of learning and apart from learning. They describe how attention goes out to formal methods of assessment, but that there are informal ways of looking at assessment. To make sure that the whole learning process is served well more attention should be given to this aspect of learning. Just measuring a student's existing state of knowledge then, is not enough. The Second Handbook of Research on Mathematics Teaching and Learning (Lester jr., 2007) specifically deals with assessment as a " bridge"  between teaching and learning. In my research -apart from this aspect- I would also like to stress the use of assessment activities to promote learning based assessment.

Purpose of assessmentBlack and Williams (2004) broadly see three functions:

supporting learning (formative) certifying the achievements or potential of individuals (summative) evaluating the quality of educational programs or institutions (evaluative)

Summative assessment is also characterised as assessment of learning and is contrasted with formative assessment, which is assessment for learning.

Summative assessmentIn most curricula summative assessment is used. Summative assessment is mostly aimed at grading and scoring. Some researchers argue that, instead of providing a certain grade which seems to say what "level of knowledge" a student has, formative tests give the student an insight into the nature of -for example- their misconceptions.

Formative assessmentBlack and William have made a case for more formative assessment. In their article from 1998 they state that "improving formative assessment raises standards". Actively involving the student, implementing formative assessment as an essential part of the curriculum and motivating students through self-assessment are key benefits of formative assessment. Means to do this are feedback, self-assessment, reflection and interaction. This makes a good case for tools that aid these factors.Formative assessment is a process in which self-reflection should result in more insight (Crooks, 2001). Cowie and Bell (1999) define it as the process between teacher and student to “enhance, recognise and respond to the learning”. Black and Wiliam (1998) only define assessment as being ‘formative’ when the feedback from learning activities is actually used to modify teaching to meet the learner's needs.

- 15 -

Formative versus summativeRecently a shift from more summative towards formative assessment has taken place. This shift has also stressed the apparent disadvantages of formative assessment. Firstly the fact that less teachers available means more work as formative assessment is more time-consuming. Secondly modularization means that students learn about their mistakes when finished with a subject. This promotes a grade-culture, for assessment -albeit formative- is used as a means to test whether a student has learned enough. Here we clearly see tension between summative and formative assessment. As Black & Wiliam (2004) put it:

“teachers seemed to be trapped between their new commitment to formative assessment and the different, often contradictory demands of the external test system” (p. 45).

A positive connection between formative and summative assessment is sorely needed (Broadfoot & Black, 2004) In this sense one could argue that summative and formative assessment are potentially complementary (e.g.(Biggs, 1998); (Harlen, 2005)) and should be integrated more (Shavelson, Black, Wiliam, & Coffey, 2002). This opposed to the viewpoint that they should generally be kept apart (e.g. (Knight & Yorke, 2003))

Pre-emptive formative assessmentIn reality education seems to tend to a balance of formative and summative assessment. They have one element in common: they both are mostly conducted after learning. To be able to actually learn during the process of learning means assessing throughout the course of a module. Black and William call this pre-emptive formative assessment. It builds on constructivist learning principles (Black & Wiliam, 2003) learning starts from the learner's existing knowledge and learning entails actively incorporating new knowledge into an existing knowledge framework. Using pre-emptive formative assessment means using feedback as a central element in the learning process (Hattie, Biggs, & Purdie, 1996). Instead of serving up feedback too little too late, feedback is used "pre-emptively", to make sure whether a student is on the right track or not. The role of feedback in formative evaluation was already stated in the 60s by Bloom (1969):

"Quite in contrast in the use of 'formative evaluation' to provide feedback and correctives at each stage in the teaching-learning process. By formative evaluation we mean evaluation by brief tests used by teachers and students as aids in the learning process. While such tests may be graded and used as part of the judging and classificatory function of evaluation, we see much more effective use of formative evaluation if it is separated from the grading process and used primarily as an aid to teaching".

I contend that both a score-based approach, combined with the strength of formative assessment is a good approach to learn more. Scoring has an inherent motivational aspect. Also, adding several " modes"  from practice to " exam" and in-between will facilitate different uses of assessment, both formative and summative.

Carless (2007) suggests that a suitable timing for pre-emptive formative assessment is the class, classes or a longer period preceding high stakes assessment. (see (Lester jr., 2007))

Framework for Classroom Assessment in Mathematics

- 16 -

In De Langes “Framework for Classroom Assessment in Mathematics” several principles for good assessment in mathematics classrooms. This framework connects well to the OECD framework that was designed for the PISA program.

In operationalizing these principles, especially principle 5, it is important to distinguish different mathematical levels of thinking. In chapter 6 –when we choose our content- we will see that De Langes “Assesment pyramid” has many similarities with other categorizations of mathematical skill level.

The role of feedbackUsing feedback after learning does not provide an incentive to actually use it to adjust knowledge and/or beliefs. This means that feedback is often very ineffective (D. Carless, 2006), caused by the lack of iterative cycles of feedback and revision. In line with this criticism Gibbs and Simpson (2004) have stated that timeliness and potential for student action are key components of good feedback. This means that timing is important and that it should be possible to act on the basis of the feedback provided, for example by enabling the possibility to correct an answer. As some research suggests written feedback is less effective than oral feedback (Boulet, 1990), as students pay little attention to teachers' written comments (Zellermayer, 1989) or find them difficult to understand and act upon (Clarke, 2000).  These two criteria form a basis for a pre-emptive approach of assessment. On the teacher side a teacher should have insight into the misconceptions a student has.

Feedback is routinely applied to any information that a student is given about their performance. Historically the term was used in systems engineering by Norbert Wiener in 1940 working on guns. In education feedback tends to be used differently. (Ramaprasad, 1983) says: " Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way" .

“Principles for Classroom Assessment1. The main purpose of classroom assessment is to improve learning

(Gronlund, 1968; de Lange, 1987; Black & Wiliam, 1998; and many others).2. The mathematics is embedded in worthwhile (engaging, educative,

authentic) problems that are part of the students’ real world.3. Methods of assessment should be such that they enable students to reveal

what they know, rather than what they do not know (Cockroft, 1982).4. A balanced assessment plan should include multiple and varied opportunities

(formats) for students to display and document their achievements (Wiggins, 1992).

5. Tasks should operationalize all the goals of the curricula (not just the “lower” ones). Helpful tools to achieve this are performance standards, including indications of the different levels of mathematical thinking (de Lange, 1987).

6. Grading criteria should be public and consistently applied; and should include examples of earlier grading showing exemplary work and work that is less than exemplary.

7. The assessment process, including scoring and and grading, should be open to students.

8. Students should have opportunities to receive genuine feedback on their work.

9. The quality of a task is not defined by its accessibility to objective scoring, reliability, or validity in the traditional sense but by its authenticity, fairness, and the extent to which it meets the above principles (de Lange, 1987).”

- 17 -

 An interesting aspect, as Sadler (1989) noted, is that feedback is used to influence a certain gap. So if we don't use recorded information for this purpose, it is not considered feedback. This means that formative assessments -Ramaprasads feedback- are closely linked to instructional consequences. The function of assessment becomes formative if student's learning is served, through feeding feedback back into the system. An example clarifies this: telling someone to work harder is not formative, as it doesn't involve any feedback as how to work harder. Telling someone to use more steps when solving an equation is formative. Perhaps the distinction becomes even clearer if we see monitoring assessments provide information of whether a student is learning or not, diagnostic assessments provide information on what is going wrong and formative assessment provides information on what to do about it.

Principles of feedbackIn the learning process adapting instruction to meet students learning needs showed substantial benefits, for example in studies by (Carpenter, Fennema, Peterson, Chiang, & Loef, 1989) and (Black, Harrison, Lee, Marshall, & Wiliam, 2003). As the role of feedback had to be taken into account, drill-and-practice use of the computer made formative assessment difficult. An interesting question is whether the use of "more intelligent" new technology makes a difference in this respect. (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991) found that not being able to see the answer before trying a question is better. Also giving details of the right answer, instead of just wrong or right, seemed more effective, as other research has also confirmed ((Elshout-Mohr, 1994), (Dempster, 1991, 1992).

How effective is feedback?Reviews conducted by (Natriello, 1987), (Crooks, 1988), (Bangert-Drowns et al., 1991) and (P. Black & D. Wiliam, 1998) showed that that not all kinds of feedback to students about their work are equally effective.

Mason and Bruning distinguish eight types of feedback based on available research {Mason, 2001 #157. Nyquist (2003) reviewed 185 studies in higher education, developing a typology of different kinds of formative assessment: 

Weaker feedback only: students are given only the knowledge of their own score or grade, often described as " knowledge of results";

Feedback only: students are given their own score or grade, together with either clear goals to work towards or feedback on the correct answers to the questions they attempt, often described as "knowledge of correct results";

Weak formative assessment: students are given information about the correct results, together with some explanation.

Moderate formative assessment: students are given information about the correct results, some explanation, and some specific suggestions for improvement.

- 18 -

Strong formative assessment: students are given information about correct results, some explanation, and specific activities to undertake in order to improve.

 In these levels typology I do miss a category concerning feedback on the process of problem solving. This also involves feedback on sub-parts of the solution. Also allowing for several representations can provide insight in a given problem. In my research I will take into account this process and representational feedback.

Several studies -including Nyquists- seem to show more effectiveness in assessment when using feedback. The stronger the feedback the larger the effect seems to be. ((Elawar & Corno, 1985); see (Greer, 2000)) for many more references). Another issue is the use of scoring: scoring seems to have a negative impact (Butler & Nisan, 1986). Butler (1988) even concluded that the effects of diagnostic remarks completely disappeared when grades were added. There also are indication that games (reference) enhance motivation. So here we have a dilemma: do we use grades/marks and feedback together, hoping motivation will overcome the disadvantages or not. We have indications that games do motivate students, but how does this correspond with less motivation through grading.

Felix (2003) asserts humanizing feedback is important and proposes this can be done by providing structural hints, personalized hints, structural graphics, personalized graphics and games.

It should also be noted that some research indicated that (Simmons & Cope, 1993) that the possibility of using ICT tools, by providing room for the strategy "trial and improvement", requires less mental effort than the "harder" way with paper-and-pencil. More is not always better, scaffolding an exercise seems to be more effective than providing a complete solution with a new assignment. It remains unclear how effective feedback actually is, because the effectiveness seems to be determined by variables not yet well understood. It is also clear that learning intentions play a significant role. Of course, students themselves also play a large role: metacognition (more metacognition from students enhance learning), motivation and learning. This is important to acknowledge as we defined feedback as "influencing learning". But then we have to take into account those factors that influence learning as well.

Nicol and MacFarlane (Nicol & MacFarlane-Dick, 2006) reinterpreted existing literature on formative assessment and the use of feedback to their respective roles in self-regulated learning. These “seven principles of good feedback practice” are provided in table 1.

Good feedback:1. helps clarify what good performance is (goals, criteria, standards)2. facilitates the development of self-assessment and reflection in learning3. delivers high quality information to students about their learning4. encourages teacher and peer dialogue around learning5. encourages positive motivational beliefs and self esteem6. provides opportunities to close the gap between current and desired performance7. provides information to teachers that can be used to help shape teaching.

- 19 -

Table 1: Seven principles of good feedback practice(Nicol & MacFarlane-Dick, 2006)

Gibbs and Simpson (Gibbs & Simpson, 2004) were able to extract 11 conditions under which assessment might support student learning and improve chances of success. Table 1 gives these 11 conditions.

Assessment tasks [conditions 1-4]· Capture sufficient study time and effort (in and out of class)· Are spread evenly across topics and weeks· Lead to productive learning activity (deep rather than surface learning)· Communicate clear and high expectations.

Feedback [conditions 5-11]· Is sufficient (in frequency, detail)· Is provided quickly enough to be useful· Focuses on learning rather than marks· Is linked to assessment criteria/expected learning outcomes· Makes sense to students· Is received by students and attended to· Is acted upon to improve work and learning

Table 2: Gibbs and Simpson’s (2004) 11 conditions

- 20 -

Computer Aided AssessmentIn computer aided assessment (CAA) there has also been a shift from more summative use towards formative. This of course corresponds with the qualities and limitations of computers. As computers become more powerful and "smarter" more innovative use of feedback is called for.

Harlen and Deakon Crick {Harlen, 2003 #155} conducted a review of studies to find out whether there was evidence of the impact of the use of ICT for assessment of creative and critical thinking skills on students and teachers. Evidence from two high-weight studies showed that:

- Teachers are helped in understanding students in their understanding of new material, when information is stored an recorded.

- Taking over some roles of assessing and providing feedback frees up time for the teacher to focus on other ways to support learning.

- Computer feedback during a test improves student performance when the same test material is used again later on.

Sangwin's  (2006) research on Computer Algebra Systems (CAS) provides an indication of these new possibilities. For formative evaluation more open questions would be helpful, contrary to multiple choice questions with distracters. Also, multi-step exercises, and feedback on these steps, could provide formative evaluation for the student and teacher alike.

Types of questionsFor CAA in mathematics Provided Response Questions (PRQ) are most common. This category includes Multiple choice and matching questions. There are many problems with this type of question: question distortion, only lower order thinking is stimulated and strategic thinking. As Sangwin states this especially holds true for reversible mathematical skills where one direction is significantly more difficult than the other, e.g. factoring and expanding. Providing PRQ questions then probably means testing something the teacher doesn't intend to test.

These types of questions correspond a lot with the well-known behavioural paradigm or “drill instruction”. Also, because the questions and the answers are simple to construct and simple to evaluate large-scale assessment, providing many numbers and figures (“16% of students scored correctly on questions concerning integrals”) For management purposes this perhaps would suffice, but for math educators a more qualitative approach would be better: what type of mistakes are made, so I can address this issue in the next lesson. The learned lessons from this, in fact a grouped list of misconceptions, could not only provide input for classroom practice, but in the future also for improvements on software. This also holds for feedback on the process of solving equations.

In another paper (Sangwin & Grove, 2006) also point out that ease of authoring tests for teachers should not be neglected.  Implementing feedback in toolsImplementing more elaborate feedback in mathematics software is still is at an amateur level. As Jeuring (2007) points out: much can be improved, especially on the level of providing strategies. Approaches that can be used can be Sangwin's approach, but also based on rewrite rules and state transitions (Goguadze, Gonzalez Palomo, & Melis, 2005). Unfortunately this also means that interoperability issues (Goguadze, Mavrikis, & Palomo, 2006) exist.

- 21 -

Pyramid

C. Assessment in this research  We argue that a shift from summative assessment towards pre-emptive formative assessment could be beneficial for learning. Feedback plays a large role in this type of assessment. We will focus on using a tool that can pre-emptively perform formative assessment for learning algebraic skills. Here we will use the eleven conditions under which assessment might support student learning and improve chances of success (Gibbs & Simpson, 2004), as well as the seven principles of good feedback practice (Nicol & MacFarlane-Dick, 2006). The ability to cater for these conditions will be used –along with other criteria- to choose a tool for assessment.

- 22 -

4. Integrating theory

We have seen that three “ingredients” come together in learning, testing and assessing relevant mathematical skills.

I propose the following conceptual framework:

Learning, testing and assessing relevant mathematical skills with ICT can be scrutinized in three different ways:

As tool use . Through instrumental genesis learning is facilitated. As ‘learning of algebra’. We look at symbol sense for ‘real learning’, and basic skills

for algorithms. As assessment. Whereby pre-emptive formative assessment not only shows the

learning result but also aids learning. Feedback is an important factor in this type of assessment.

We aim to integrate the assessment of algebraic skills into one prototypical design, thus giving us insight into these three aspects and their dependencies. This ‘triangulation’ helps in creating a ‘body of knowledge’ on acquiring algebraic skills with the help of computer tools. Hopefully this will lead to common ground within the research fields of algebra, assessment and tool use.

We will now sum up the implications of this conceptual framework, and underlying theoretical frameworks for the three topics, for our prototype.

ICT tool use Algebraic skills

Assessment

Learning

ICT tools that facilitate algebraic skills

ICT tools for assessment

Assessing algebraic skills

- 23 -

5. Choosing content that makes symbol sense

In this chapter we will discuss how we choose questions for acquiring, testing and assessing relevant algebraic skills. Together these questions will make up the content for our first prototype. First we describe what we mean by relevant algebraic skills, then we use several sources to choose relevant questions for our prototype and describe why they are relevant for our research.

To choose our content we first have to determine what our algebraic focus will be. As we are interested in ‘symbol sense’:

- What skill level does a question have?- What type of ‘symbol sense’ does the question address?

Classification of skill levelIn the NKBW project (2007) a classification for skill level of a question was used, involving the letters A,B and C. This classification was used earlier in the Webspijkeren project (Kaper, Heck, & Tempelaar, 2005), and drew on work by Pointon and Sangwin (2003).

Building on the general schema of Bloom's Taxonomy, Smith (Smith, Wood, M., & Stephenson, 1996) came to eight categories.

  Group A Group B Group C

1. Recall factual knowledge 4. Information transfer

6. Justifying and interpreting

2. Comprehension 5. Application in new situations

7. Implications, conjectures and comparisons

3. Routine use of procedures   8. Evaluation

 Using this classification scheme several exams and questions were classified . Some of them were already analyzed in the NKBW project. From all these questions a selection was made, based on the skill level of the question. Level “C” is almost impossible to cater for with ICT tools. We decided to choose level B questions or level A questions with an adapted level B approach.

It is interesting to point out the similarities of this approach to, for example, the assessment pyramid by de Lange (1999) used for PISA, but also the TIMMS framework uses equivalent skill levels:

These three levels are:

1. Reproduction, definitions, computations. (lower level)2. Connections and integration for problem solving. (middle level)

- 24 -

3. Mathematization, mathematical thinking, generalization, and insight. (higher level)

We then analyzed, also with the help from work from NKBW project, documents that are closely related with the expected algebraic skills going from secondary to higher education:

- Several entry exams from university and HBO; - An exit exam for secondary education;

- The book ‘basiswiskunde’ by Rob Bosch and v/d Craats;

- Articles on symbol sense

Keeping in mind the skill levels we listed several suitable ‘symbol sense’ type questions. It was now time to determine what to focus on.

Algebraic focus

Now the collection of questions had to be narrowed down. Final questions were selected by focusing on Arcavi’s ‘flexible manipulation skills’ and the ‘choice of symbols’

In appendix Athe chosen questions are categorized into Arcavi behaviors and a rationale is provided what every question is all about. Per question we answer:

1. Why is this an interesting question?2. What skill or behavior is assessed here?

3. What answers do we expect?

4. What could be obstacles in answering this question?

5. What feedback could be given for this question?

6. What tools could be used to model this question?

As we use Arcavi’s categorization it is also a good idea to acknowledge his instructional implications

- symbolic manipulations should be taught in rich contexts which provide opportunities to learn when and how to use these manipulations.

- give a complex function (and graphing tool) and ask the function.

- informal sense-making makes sense.

- use algebraic symbolism early to empower symbols.

- make use of post-mortem analysis of problem solutions

- classroom dialogues and what if questions.

- 25 -

These implications will be used when we finalize the prototypical design and formulate a didactical scenario

- 26 -

6. ICT Tools for assessment

In this chapter we will give criteria for ICT tools for assessment. Based on these criteria an ICT tool will be chosen for the first prototype. This is our method:

1. First we compiled a large list of criteria, based on our theoretical framework and own experiences;

2. From this list we extracted some minimal requirements and categories of requirements.The categories were determined by collecting all possible criteria from the theoretical background, grouping them and reformulating a category; The raw list of criteria can be found in appendix B

3. A large list if tools was compiled, based on experiences from earlier projects, the Special Interest Group Mathematics ‘toetsstandaarden’ working group {Bedaux, 2007 #160}, KLOO research {Jonker, #22}, the FI math wiki on digital assessment and math software (http://www.fi.uu.nl/wiki/index.php/Categorie:Ict), and google searches. As there are hundreds of math tools an initial selection was based on the tool having at least some characteristics of tools for assessment. Based on these requirements we:

4. First selected tools that met the minimal requirements. Tools that didn’t, were described but not considered any further. For this we used a template.

5. Then the remaining tools were graded on the other categories by:a. Browsing the web on more information and usage on the tool;b. Installing the tool locally;c. Using the tool with already existing content. We aim at using quadratic equations as

this tends to be subject that is catered for almost always;d. Authoring our own content from chapter 5. Here it is possible that not all the finesses

of a tool become apparent. A minimal requirement is that authoring can be used. This means for most tools that they have to be installed. For every installed tool I keep a log of screenshots.

6. This resulted in a separate descriptions of the tools and a matrix, giving an overview of strong and weak points of several tools.

Minimal requirements- Webbased: we find it essential that using the tool can be anytime, anyplace, anywhere, using

just a web browser.- Ease of authoring, configurable: it should be possible to add own content.- Actively developed: it is important that the tool is supported and has some sort of continuity.- Minimal math support: formulas should be displayed correctly and support basic mathematical

operations.- Storage of progress: it should be possible to store results, so students can come back later, if

necessary.

CategoriesFor every item we answer these questions

What does this item mean? Why is it important?

How is it scored/weighted?

- 27 -

It is also important to note that we are scoring these tools in the light of assessment and algebra. Therefore other aspects could give plus-points, but can never be the deciding factor.

Math specific

In this category we mention the variables that are -in our eyes- specific for mathematics. Here we asked ourselves the question "what are -comparing with other subjects- aspects that distinguishes  math from other subjects.

1. Representational (sound representations, mathematical and cognitive fidelity)It is important that the way a student can work resembles the 'paper-and-pencil' way. This means being able to use graphs, tables and a certain resemblance to this way. We assert that using multiple representations (Van Streun) and a connection to today's practices, increase transfer.

2. RandomizationAssignments that are provided should not always be the same, but have varying values. Randomization caters for this. 

3. Multiple steps (within one question, from one to other question)Here we distinguish two typesof  'multiple step'  exercises.

a. Within a question: an open environment is provided and a student can choose what path to follow. This can be just one step, but also twenty steps.

b. Between questions: some complex questions consist of several subquestions. Often one questions builds on an earlier one. 

4. Integrated CASFor complex operations a CAS is necessary. We distinguish the possibility of using a CAS and the availability of a CAS, and also the type of CAS.

Ease of use

5. Teacher, authoring (questions, text, links, graphs, multimedia)Teachers should be able to make their own qustions, with text, links, graphs etc., in a user-friendly way. This aspect scores -based on my own usage- how well authoring can take place.

6. Student, usage (e.g. input editor, learning curve)Students should be able to work with tool. This means that the user intrafce and structure of the tool should be intuitive. The same holds for the input of mathematical formulae.

Registration

7. AnswersHow much of the answers is stored? Are all the answers -also the wrong ones- stored, or only the last answer.

8. ProcessAnd how much of the process. How did a student come to a certain answer?

Assessment

9. Possibility of modes (pratice, exam)Asessing formatively means that asessment is for learning. Therefore asessment takes place

- 28 -

during a course, not only at the end in an exam setting. Providing several modes is a good facility.

10. Feedback (process, answer, global)

a. Process: giving functional feedback on the strategy used.

b. Answer: providing feedback on the answer given. The more specific this is, the more a student can learn from it.

c. Global: overall mastery of a subject. Giving insight in a students progress on amore global scale.

11. HintsThe possibility of providing hints or that hints are provided.

12. Review mode (what has he/she done wrong or right)The possibility of scrutinizing ones wrong or correct answers, as to learn from them, including process. The finer the granularity the more information -when needed- can be used.

13. Question typesSometims only open questions are not enought. Other question types could provide more flexibility.

14. ScoringBeing able to use game-like scoring could help motivate students 'getting the highest score'

Content management

15. Question managementHow well can questions be managed? Can they be copied easily? Can modules be recycled easily?

16. Use of standardsThis also implies that a certain compliance to standards like QTI or SCORM is a plus.

17. Available contentIf teachers are reluctant to make their own questions, a large user base and questions could help.

Other

18. CostUsing a tool in secondary and higher education . We mean the bare licenses for using the tool. Other aspects of costs (support, training) are scored in aspects like 'ease of use' and 'continuity'.

19. License and modifiabilityWhat type of license does this tool have? Does this license make it possible to modify the software to ones own wishes (open source).

20. Technical requirements (also own installation)How easy is it to install locally? How high are the technical requirements?

- 29 -

21. ContinuityIs there a community, firm or organisation that can provide help with a tool? Our experience is that tools with small user bases and little support, provide less continuity.

22. LanguagesHow multilingual is the tool?

23. Stability and performanceOf course this item also depends on the somputer it is installed on. However, using and installing a tool does give an impression on the overall performance and stability.

24. LooksIn education a tool should look good.

25. Integration for VLE

We propose to use the weights provided in the matrix in appendix E.

We would have liked to ask experts on the scores in the matrix, but this would imply that experts should have detailed knowledge of all the tools. Although one could always argue that the scores presented are arbitrary, we would contend that it provides a cgood picture of weak and strong points of the tools.

Conclusion:

Wiris is an attaractive tool for standalone use within for example Moodle. However, the lack of assessment functions means it is not suitable enough for our research.

WIMS probably is the quickest and most 'complete' of the tools with room for geometry, algebra, etc. There also is a fair amount of content available, also in dutch. It is let down by the feedback and the fact only one answer can be entered. Of course this can be programmed, as it is a very powerful package, but here we see a steep learning curve.

Digital Mathematical Environment. Strong points are the performance, multisteps within exercises plus feedback, authoring capability, SCORM export. Disadvantages: emphasis on algebra (not extendable, dependent on the programmer), source code not avaialble.

STACK has a good philosophy with 'potential responses' and multistep questions. Also, the integration with a VLE -unfortunately only moodle- is a plus. Installation, stability and performance is a negative (slow and cumbersome), as well as its looks. It is also very experimental, providing almost no continuity.

Maple TA has many points that STACK has, but with better looks and no real support for 'potential responses' and ' multistep questions'. They can be programmed, but this means -like WIMS- coping with a steep learning curve. As it has its roots in assessment software question types are well provided for. One could say that Maple TA started as assessment software and his moving towards software for learning, while STACK started with learning and is moving towards assessment software.

Activemath is more of an Intelligent Tutoring System than a tool for assessment. The question module (ecstasy) is powerful, providing transition diagrams. However, authoring and technical aspects make it less suitable for the key aspects we want to observe.

- 30 -

For summative assessment Maple TA is the best contender. For both summative (scores) and formative assessment DME is best used, with STACK a runner up. For algebra DME is best used because of the fact the 'process' is central. For all other mathematical topics WIMS is most versatile. For an interactive book without assessment features Activemath could be used best. The best standalone mathematical environment (CAS) is provided by Wiris

We conclude that DME is best suited for answering our research question, as it:

- mixes formative and summative aspects of asessment into one tool;- provides an open algebra environment, opposed to the more closed questions (one answer)

that other tools provide;

- is a stable and attractive tool (motivation);

The descriptions of the tools that were considered but did not meet the minimal requirements can be found in appendix C. Descriptions of the tools that did meet the minimal requirements can be found in appendix D. A comparative matrix assessing the strong and weak points of these tools can be found in appendix E. In the long term we aim to publish these assessments in a wiki type environment so they can be kept up to date.

Now we will use our tool to model the questions from chapter 5.

- 31 -

7. Designing the prototype and instruction

So based on a rationale and a problem statement, a conceptual framework was formulated, leading to a motivated choice in ICT tool and content, plus design principles. In this chapter we describe what prototype resulted from all of this. This prototype will be the prototype used in the actual research cycles.

I want to make sure that transfer from the tool towards pen and pencil takes place (Kieran & Drijvers, 2006). This is why I design an instructional sequence with both tool use and pen/paper tests. This sequence has some similarities with the Hypothetical Assessment Trajectory in the CATCH project.

For our prototype this means Formulate an orchestration for tool use A visual approach that facilitates transfer from computer to pen-and-paper

A prototype of the test can be found on

http://www.fi.uu.nl/dwo/voho

Login name: vohodemoPassword: omedohov

In appendix F some screenshots can be found of the authoring process.

The first implementation has no randomization. The second implementation will have more questions making use of randomization.

Authoring capabilities are available on request.

- 32 -

Part III Methodology

(This part is under construction for the research plan. Deadline: 1/4/08)

See research plan: hypothetical assessment trajectorySelf-assessmentIt is tempting to quote here a postulate by Wiggins (1993): “An authentic education makes self-assessment central.”

In 1998, Black and Wiliam were surprised to see how little attention in the research literature had been given to task characteristics and the effectiveness of feedback. They concluded that feedback appears to be less successful in “heavily-cued” situations (e.g., those found in computer-based instruction and programmed learning sequences) and relatively more successful in situations that involve “higher-order” thinking (e.g., unstructured test comprehension exercises).

“Let us start with the Professional Standards for School Mathematics (NCTM, 1991). These standards envision teachers’ responsibilities in four key areas:

Setting goals and selecting or creating mathematical tasks to help students achieve

these goals.

Stimulating and managing classroom discourse so that both the students and the

teacher are clearer about what is being learned.

Creating a classroom environment to support teaching and learning mathematics.

Analyzing student learning, the mathematical tasks, and the environment in order to

make ongoing instructional decisions.”

The consideration of (a) the learning goals, (b) the learning activities, and (c) the thinking and learning in which the students might engage is called the hypothetical learning trajectory (Simon, 1995).

Our basic assumptions will be the following: there is a clearly defined curriculum for the whole year—including bypasses and scenic roads—and the time unit of coherent teaching within a cluster of related concepts is about a month. So that means that a teacher has learning trajectories with at least three “zoom” levels. The global level is the curriculum, the middle level is the next four weeks, and the micro level is the next hour(s). These levels will also have consequences for assessment: end-of-the-year assessment, end-of-the-unit assessment, and ongoing formative assessment.

Hypothetical Assessment Trajectory.Some of the ideas we describe have been suggested by Dekker and Querelle (1998).BeforeEntry test. A short, written entry test consisting of open-ended questions.

DuringDuring. While in the trajectory, there are several issues that are of importance to teachers and students alike. One is the occurrence of misconceptions of core ideas and concepts

- 33 -

Short quizzes, sometimes consisting in part of one or more problems taken directly

from student materials.

Homework as an assessment format (if handled as described in our earlier section on

homework).

Self-assessment—preferable when working in small groups. Potential important diffi-

culties will be dealt with in whole-class discussion.

This ongoing and continuous process of formative assessment, coupled with the teachers’ so-called intuitive feel for students’ progress, completes the picture of the learning trajectory that the teacher builds.

After. At the end of a unit, a longer chapter, or the treatment of a cluster of connected concepts, the teacher wants to assess whether the students have reached the goals of the learning trajectory. This test has both formative and summative aspects depending of the place of this part of the curriculum in the whole curriculum.

- 34 -

Appendix A

This appendix describes the chosen questions in more detail, answering the following questions.

1. Why is this an interesting question?2. What skill or behavior is assessed here?

3. What answers do we expect?

4. What could be obstacles in answering this question?

5. What feedback could be given for this question?

6. What tools could be used to model this question?

1

1. This question addresses whether a student has ‘gestalt’ quality: does her or she recognize similar parts of an equation.

2. Behavior #6: flexible manipulation skills.3. When a student recognizes similar parts he or she would:

or

(Note: students could easily forget that becoming 0 yields two answers.)

or

or or

Students could also be attracted by the brackets and tempted to lose them. Perhaps the fact that this is a lot of work will keep them from doing so. Perhaps an easier but similar question like would be dealt with this way.

4. Kop and Drijvers (Kop & Drijvers, in press) mention some causes: firstly the fact that students see formulas as recipes (process) instead of coherent objects. Secondly visual characteristics of questions play a role (‘visually salient’). In question one it is tempting to lose the brackets. In question 2 it is tempting to square the roots, as to lose them. In both cases this gives a lot of work, but brings us no closer to the solution. Thirdly a lack of flexibility in manipulating expressions. Fourthly a lack of meaning. Perhaps using a model helps

- 35 -

understanding what is asked. Lastly there often is a lack of practice in solving such questions. With this last point we explicitly contend that insight also comes from practicing. Or to quote Freudenthal: “Advocates of insightful learning are often accused of being soft on training. Rather than against training, my objection to drill is that it endangers retention of insight. There is, however a way of training — including memorisation — where every little step adds something to the treasure of insight: training integrated with insightful learning.”(Freudenthal, 1991)

5. Perhaps giving the possibility of substitution, or at least making visual or symbolic groupings could help a student. Pointing out that a student should look for corresponding terms.

6. Here giving the possibility to plot functions and thus to see what solutions an equation has, could be helpful. A tool should also enable students to algebraically solve an equation, preferably in ‘What you See Is What You Get’ form, to help transfer from the tool to ‘pen and paper’.

2 solved for v

1. This question addresses whether a student has ‘gestalt’ quality: does her or she recognize similar parts of an equation, what characteristics are ‘visually salient’

2. Behavior #6: flexible manipulation skills.3. This question has to do with –as Arcavi puts it- ‘gestalt’ and ‘circularity’. Wenger

(Wenger, 1987) : “If you can see your way past the morass of symbols and observe the equation #1 (

which is required to be solved for v) is linear in v, the

problem is essentially solved: an equation of the form av=b+cv, has a solution of the form v=b/(a-c), if a≠c, no matter how complicated the expressions a, b and c may be. Yet students consistently have great difficulty with such problems. They will often perform legal transformations of the equations, but with the result that the equations become harder to deal with; they may go “round in circles” and after three of four manipulations recreate an equation that they had already derived…Note that in these examples the students sometimes perform the manipulations correctly…”

4. Here the roots are important: they attract attention, but actually trying to take squares on both sides would be a mistake. The chance that students are not able to continue or come ‘back where they were’ (circularity) is large. Also, the fact that there are two variables, contrary to question 1, poses a problem.

5. Perhaps giving the possibility of substitution, or at least making visual or symbolic groupings could help a student. Pointing out that a student should look for corresponding terms, but also what ‘solved for v’ means.

- 36 -

6. Here giving the possibility to plot functions and thus to see what solutions an equation has, could be helpful. A tool should also enable students to algebraically solve an equation, preferably in ‘What you See Is What You Get’ form, to help transfer from the tool to ‘pen and paper’. Two variables should be supported.

3

1. This question addresses whether a student has ‘gestalt’ quality: does her or she recognize similar parts of an equation, what characteristics are ‘visually salient’

2. Behavior #6: flexible manipulation skills.3. This question again has to do with –as Arcavi puts it- ‘gestalt’ and ‘circularity’.

This equation can be solved ‘the easy way’ of ‘the hard way’. Noticing that 4. Here the square and the brackets are important: they attract attention, but actually

rewriting the left side of the equation would be a mistake. The chance that students are not able to continue or come ‘back where they were’ (circularity) is large. This

mistake could be even more tempting to make when is used.5. Perhaps giving the possibility of substitution, or at least making visual or symbolic

groupings could help a student. 6. Here giving the possibility to plot functions and thus to see what solutions an

equation has, could be helpful. A tool should also enable students to algebraically solve an equation, preferably in ‘What you See Is What You Get’ form, to help transfer from the tool to ‘pen and paper’.

4

Find the pattern and proof the result always is 2(source: Martin Kindt)

1. This question addresses whether a student understands that using certain symbols could help solving a problem.

2. Behavior #5: the choice of symbols. In this question Arcavi would call some premonitory feeling for an optimal choice of symbols an important part of symbol sense.

3. Here we expect students to make a choice of symbols and then rewrite the expression. So there are two steps. It could also be interesting to see whether the choice of symbols differs, but of course both giving 2 as answer. For example:

versus

- 37 -

Choosing symbols wisely also could make the next step easier or more difficult.4. Of course, not choosing ones symbols wisely here –or not even having a clue as to

how to choose symbols- poses a problem in this question.5. A suggestion for choosing a variable good be given, for example “remember that

subsequent numbers could be modeled by n, n+1, n+2”.6. A tool should be able to ask for an expression, determine whether it is correct, and

enable the student to rewrite the expressions. So the solution process is very important.

5Of a function f(x) we know:

How much is ?

(Note: the original question is a multiple choice question)

1. This question addresses whether a student has ‘gestalt’ quality: does her or she recognize similar parts of an equation.

2. Behavior #6: flexible manipulation skills.3. Students would have to recognize that a function is not given, and hopefully be

triggered that something else plays a role, in this case transformations. If this is recognized it should not be too hard to see that the function was shifted two to the right and the integral as well.

4. Here the difficulty of the question lies in the fact that the function f is unknown. Students will probably wonder how the integral can be calculated without actually knowing the function. A more specific question arises when we choose a certain function, or first start with two question with given functions, and then the general statement.

5. Hinting on the fact that this is a general statement - it holds for all function- is important. Therefore any chosen function could provide insight. Also the process of transformation of a function is an important concept.

6. Here giving the possibility to plot functions could be helpful.

- 38 -

Appendix B

CategoriesContent: algebra

Multistep exercises, process feedback Sound representations and operations (!!!): mathematical and cognitive fidelity Possibility of using more than one representation;

Multiple representations can help better understanding of concepts. Randomization Wysiwyg formulas and input editor Integrated CAS

Level of competencies.

Gradual difficulty curve. Coherence and balance.

Technical: tool Results and answers should be stored and accessible for both student and teacher. Sharability and use of standards (questions and units) Easy authoring for teachers (because teachers are often “neglected learners” (Sangwin

& Grove, 2006)) Scoring as a game, for motivation. Other tools available Performance Possibility of using multimedia Ease of use Customizability by teacher

Contexts.

Prerequisites server and client-side Stability Languages

Assessment It should provide feedback at the right moment Several "modes" ranging from diagnostics to exam. “Zoom” level. Several training

modes (from practice to exam). Enables a qualitative analysis of student work to reap the benefits of formative

evaluation; on a meta-level scoring should also be implemented to allow for quantitative data; A student should be able to correct an answer. Possibility to correct an answer The combination of elements from both summative and formative assessment should

enable a teacher to adapt better to student's learning. Go from feedback only to weak formative assessment or even moderate. This has a lot

to do with the state of the art. (level of feedback) The 11 conditions under which assessment might support student learning and

improve chances of success. Seven principles of good feedback practice.

- 39 -

Open questions providing an expressive environment Halfopen/closed questions. Open/closed questions Functional feedback on the answers

Formative assessment Feedback on the process` Global feedback: mastery level within curriculum Possibility of providing hints One answer versus multiple steps Storage of answers Storage of the process Question management Adaptivity

General Licenses User base for content availability Continuity

- 40 -

Appendix C

Not considered: WebLearn from RMIT Melbourne, mathxl from Addison-Wesley, Algebra Interactive, Core-Plus Mathematics

Minimal requirements- Webbased: we find it essential that using the tool can be anytime, anyplace, anywhere, using

just a web browser.- Ease of authoring, configurable: it should be possible to add own content.- Actively developed: it is important that the tool is supported and has some sort of continuity.- Minimal math support: formulas should be displayed correctly and support basic mathematical

operations.- Storage of progress: it should be possible to store results, so students can come back later, if

necessary.

DITwis

Name DITwisDate 20080217Version DIT5Key words Too difficult too author. No multiple step solutions. No CAS in the

background. Web-based javascript. SCORM possible.http://wiskunde.stmichaelcollege.nl/DITwis/

Screenshots

Doesn’t adher to minimal requirement

too difficult to author needed functionality

- 41 -

Algebra Tutor

Name Algebra TutorDate 20080217Version UnknownKey words Interesting Eliza-type tutor. There seems to be no authoring option. Only

algebra, no graphics. http//www.algebratutor.org

Screenshots

Doesn’t adher to minimal requirement

Not possible to add own questions

- 42 -

Calmaeth

Name CalmaethDate 20080229Version UnknownKey words https://calmaeth.maths.uwa.edu.au/Screenshots

Doesn’t adher to minimal requirement

Not possible to add own questions

- 43 -

Math Xpert:

Name Math XpertDate 20080217Version UnknownKey words Standalone windows. Not web-based. But with steps. 89.95 dollar per license

for the three topics covered. This looks like a suitable program, but it’s not web-based. This would hinder the ‘anytime-anyplace’ criterium.http://mathware.stores.yahoo.net/mathpert.html

It Shows the Steps!

MathXpert focuses on learning mathematics by helping the student to work through problems successfully. Students often encounter difficulty in problem solving because they've made a trivial mistake that compounds in later stages, or because they've forgotten an important step of the strategy. Either of these problems result in the student becoming stuck, blocking momentum and halting the learning process. MathXpert is designed to eliminate these roadblocks by actively helping students solve any problem correctly. The easy-to-use interface allows the student to focus on the correct strategy of problem-solving.

Screenshots

Doesn’t adher to minimal requirement

Not web-based.

- 44 -

Aplusix

Name AplusixDate 20080217Version IIKey words There are four modes: training, test, self-correction, observation. Two

dimensional editor. Scores. 400 patterns of exercises. A disadvantage: it’s not web-based, so every students needs a copy of this software. Replay mode for looking at the process.

Aplusix is a new sort of software for arithmetic and algebra which lets students solve exercises and provides feedback: it verifies the correctness of the calculations and of the end of the exercises. Aplusix has been designed to be integrated into the regular work of the class: it is close to the paper-pencil environment, it uses a very intuitive editor of algebraic expressions (in two dimensions); it contains 400 patterns of exercises. Experiments in several countries and in several situations, from 2002, have had very positive results, measured with pre-test and post-test.

http://aplusix.imag.fr/en/index.html Distributor is http://www.rhombus.be/index1.html . For 600 euro there is a sitewide license. As far as I can see then you need to pay 10 euro per student.

Screenshots

Doesn’t adher to minimal requirement

Not web-based

- 45 -

L’Algebrista

Name L’AlgebristaDate 20080217Version UnknownKey words

http://www-studenti.dm.unipi.it/~cerulli/LAlgebrista/index.php?lang=ita doesn’t work. http://telearn.noe-kaleidoscope.org/warehouse/Cerulli-M-Mariotti-M-2000-bis.pdf and http://www.dm.unipi.it/~didattica/CERME3/proceedings/Groups/TG8/TG8_Cerulli_cerme3.pdf are papers about this program.

ScreenshotsDoesn’t adher to minimal requirement

URL doesn’t work

- 46 -

webMathematica

Name webMathematicaDate 20080217Version 2Key words Only seems to provide a frontend for the mathematica program. There are

some quizzes on algebra. I’m not impressed. There also is a license fee.http://www.wolfram.com/products/webmathematica/index.html

Screenshots

Doesn’t adher to minimal requirement

No assessment software, storage of results.

- 47 -

Wiris

Name WirisDate 20080218Version 2Key words Because of the integration with moodle we included this in the matrix, even

though one minimal requirement hadn’t been met.http://www.mathsformore.com/ http://www.wirisonline.net/

Screenshots

Doesn’t adher to minimal requirement

No assessment software, storage of results.

- 48 -

AiM: Assessment in Mathematics

Name AiM: Assessment in MathematicsDate 20080217Version 3.0Key words This open source program is a . Maple is running in the background. This is

probably –Maplesoft has it’s Maple TA- why it has been discontinued. The developers wanted a similar program, based on open source CAS. See STACK and CABLE

"AiM is an open-source system for computer-aided assessment in mathematics and related disciplines, with emphasis on formative assessment."

http://maths.york.ac.uk/aiminfo/ Screenshots

Doesn’t adher to minimal requirement

Software discontinued

- 49 -

CABLE

Name CABLEDate 20080217Version 1.0Key words Similar to a very first version of STACK and following up on the AiM

program. In the background the open source CAS axiom runs. OverviewComputer algebra systems (CAS) are well-established research tools for performing symbolic manipulation of mathematical expressions, such as algebra and calculus. CABLE is designed to be a cost-effective, online infrastructure for writing, testing and databasing lightweight and flexible mathematical learning objects. This infrastructure allows both staff and students to interact with computer algebra for the following functions: Instantiation and delivery of objects with randomised parameters.

Evaluation and feedback of answers to questions contained in these objects. Typically each object has one question associated with it.

Generation of worked solutions from templates with reference to the instantiated parameters.

Simple analysis of answers from cohorts of students.

Design has preferred formative learning rather than assessment.

http://www.cable.bham.ac.uk/Screenshots

Doesn’t adher to minimal requirement

Software not maintained

- 50 -

Hot potatoes

Name Hot PotatoesDate 20080217Version 6.2Key words Quiz software

The Hot Potatoes suite includes six applications, enabling you to create interactive multiple-choice, short-answer, jumbled-sentence, crossword, matching/ordering and gap-fill exercises for the World Wide Web. Hot Potatoes is not freeware, but it is free of charge for those working for publicly-funded non-profit-making educational institutions, who make their pages available on the web. Other users must pay for a licence. Check out the Hot Potatoes licencing terms and pricing on the Half-Baked Software Website.

Unfortunately the mathematics support is quite bad. By using SCORM tracking scores within a VLE is possible. No support for a CAS.http://web.uvic.ca/hrd/halfbaked/

Screenshots

Doesn’t adher to minimal requirement

No good math support

- 51 -

Question Mark Perception

Name Question Mark PerceptionDate 20080217Version UnknownKey words Is assessment software with no particular support for CAS related questions.

Use of MathML is possible.http://www.questionmark.com/. Example at http://www.questionmark.com/us/tryitout_k12.aspx

Screenshots

Doesn’t adher to minimal requirement

No good math support

- 52 -

Wintoets

Name WintoetsDate 20080217Version 4.0 WEBKey words Is assessment software with no particular support for CAS related questions.

Dutch.http://www.drp.nl/Producten/WinToets-40-WEB.html

ScreenshotsDoesn’t adher to minimal requirement

No good math support

- 53 -

Moodle quiz module with extensions

Name Moodle quiz moduleDate 20080217Version All moodle versionsKey words Limited question types but can be extended with STACK (see appendix C)

http://www.moodle.org . Communication through the OPAQUE protocol: http://docs.moodle.org/en/Development:Open_protocol_for_accessing_question_engines As this technology actually uses STACK in the default installation, we will look at the possibilities. Moreover: Moodle is required for STACK.

Screenshots

Doesn’t adher to minimal requirement

No math support (CAS)

- 54 -

Wallis

Name WallisDate 20080217Version UnknownKey words http://www.maths.ed.ac.uk/~wallis/ Screenshots

Doesn’t adher to minimal requirement

Software not maintained

- 55 -

WebWork

Name WebWorkDate 20080217Version 2.4.1Key words

WeBWorK is a web-based interactive system designed to make homework in mathematics and the sciences more effective and efficient. It can be used with moodle.http://webwork.maa.org/moodle/

Screenshots

Doesn’t adher to minimal requirement

Difficult to install and use

- 56 -

TI interactive

Name TI interactiveDate 20080229VersionKey words http://www.t3ww.org/pdf/TII.pdf

http://education.ti.com/educationportal/sites/US/productDetail/us_ti_interactive.html

Screenshots

Doesn’t adher to minimal requirement

No registration for assessment

- 57 -

Cognitive Tutor

Name Cognitive TutorDate 20080229Version UnknownKey words http://www.carnegielearning.com/products.cfm

Screenshot: http://www.carnegielearning.com/products_algebraI.cfmScreenshots

Doesn’t adher to minimal requirement

No authoring function.

- 58 -

Algebra Buster

Name Algebra Buster / AlgebratorDate 20080229Version UnknownKey words Looks nice.

http://www.algebra-online.com/Solving%20systems%20of%20equations.htmhttp://www.softmath.com/algebra-help/free-printable-linear-equation.html

Screenshots

Doesn’t adher to minimal requirement

Not web-based

- 59 -

Appendix D

Maple TA

Name Maple TADate 20080229Version 3.0Key words http://www.maplesoft.com/

Dutch distributor: http://www.candiensten.nl/software/details.php?id=26 Mathmatch: http://www.mathmatch.nl/onderwerpen.diag.phpMore content: http://mapleta.can.nl/classes/kamminga/Tested via: experiences from earlier workshops, {Heck, 2004 #2}

Screenshots

- 60 -

Digital Mathematical Environment

Name Digital Mathematical EnvironmentDate 20080229Version UnknownKey words This evaluation is based on (past) experiences of the Galois and Sage projects.

Several tests were made in a special environment for secondary and higher education topics: http://www.fi.uu.nl/dwo/voho

After registering the test becomes available.

schoollogin: vo-hostudent password: passw_leerling_vo

Screenshots

- 61 -

Activemath

Name ActivemathDate 20080229Version 1.0Key words http://www.activemath.org/

Activemath is a comprehensive tool with much more than is our focus in our research (tutorial component, mathematical learner model).

Screenshots

- 62 -

STACK

Name STACKDate 20080229Version 2.0Key words In the new version 2.0 of STACK Moodle is required. Installing STACK is

quite difficult. http://www.stack.bham.ac.uk/ The possibilities of Stack are also evaluated using our knowledge of the earlier version 1.0 and the moodle quiz module. There is a lot of higher education material.

Screenshots

- 63 -

Wims

Name WIMSDate 20080217Version 3.57 (3.62 released)Key words http://wims.unice.fr/

Exercises can be made by using Createxo. Power modules are also possible, but require more administrator rights. For dutch content: http://wims.math.leidenuniv.nl/wims -> studentenbereik -> Gestructureerde aanpak algebraische vaardigheden

(administrator: Relinde Jurrius)

username test, password gaav

Screenshots

- 64 -

Appendix EA more detailed rationale for every score can be found in the worksheet matrix_tools_290208.xls                  Wm=WIMS, S=STACK, M=Maple TA, D=DME, Wr=Wiris, A=Activemath  Tool Wm S M D Wr A Wgt Notes

Scr Item              

Key aspect of our research is: formatively assessing symbol sense by using an ICT tool. Therefore, even if an item is valuable, weights were determined by keeping this clearly in mind.

1 Representational 3 3 3 5 4 3 5Here we mean both the fact that the mathematics behind the tool should be sound, but also that content can be displayed in several representations. (Van Streun, Janvier)

2 Randomization 5 5 5 5 1 3 4 One strong point of computer tools is that they can randomly create exercises.

3aMultisteps: within question 4 2 2 5 1 3 5

Asking for one answer is one thing, being able to find your 'own way' towards an answer is considered a strong point.

3bMultisteps: between questions 2 5 2 2 1 2 3

The ideal tool should be able to connect several exercises into one exercise, making it possible to make more complex (type C) questions.

4 Integrated CAS 4 5 5 3 5 5 3An integrated CAS is a plus, but we should keep in mind that our research focuses on relatively simple mathematics.

5 Teacher, authoring 2 3 3 4 1 2 5As Sangwin stated teachers are 'neglected learners'. Ease of authoring should be an important variable.

6 Student, usage 4 2 3 5 5 3 5 Of course, ease of use for students should also be taken into account.

7Registration: answers 3 5 4 4 1 2 5

We want a tool to store all answers a student gives in. Also storing wrong answers is a plus.

- 65 -

8Registration: process 1 2 2 5 1 2 5

To get insight in WHAT a student has done right/wrong one has to know the 'way to an answer'.

9 Possibility of modes 5 5 5 5 1 1 5In a didactical scenario, both practicing and testing plays a role. Providing these modes is a part of formative assessment

10a Feedback: process 1 1 1 3 1 2 5

Providing feedback on the way a student answers a problem, could be a part of formative assessment. Here we can distinguish general remarks (we see these as Hints) and actually saying something about the process.

10b Feedback: answer 3 5 3 4 3 5 5

The possibility of giving more feedback than right/wrong is deemed valuable. We score whether it is possible, and how easy it is to author. Feedback on 'in-between' answers is seen as a plus. Ease of authoring these questions is also taken into account.

10c Feedback: global 2 2 2 2 1 4 5An overall view of one's mastery of a subject is seen as global feedback. It is valued slightly lower than the other feedback types.

11 Hints 4 4 4 2 1 5 3 Feedback can also be provided in the form of hints.

12 Review mode 1 3 4 5 1 1 5Essential for formative assessment, as a student can examine his/her answers and mistakes, and subsequently learn from them.

13 Question types 3 2 5 1 1 3 2Providing a variety of question types makes a tool more versatile. As an item it isn't as important as the other items.

14 Scoring 3 4 3 3 1 1 2We see scoring as a way to motivate students. This is contrary to literature on formative assessment.

15Question management 3 4 5 3 1 2 3

Being able to copy, delete and recycle questions is an important aspect for practical tool use.

- 66 -

16 Use of standards 2 3 4 4 3 4 3Sharing content -both formulas, questions and whole packages- is easier when standards are being used.

17 Available content 4 2 5 4 3 2 3 The availability of existing content.

18 Cost 5 5 1 3 2 5 2 The pricing of a tool.

19License and modifiability 5 5 1 3 2 4 2 We value open source higher than closed source, because we can easier adapt code.

20Technical requirements 1 1 3 4 4 2 2

We score the technical requirements of a tool. Here we make use of our experiences installing the tool, but also previous experiences with the used technologies.

21 Continuity 2 1 4 3 4 3 4It is very important that teachers can rely on a tool to be supported for a while. This score reflects this.

22 Languages 5 2 2 3 4 4 1 Is the tool available in more than one language?

23Stability and performance 4 2 3 4 4 2 2

Being able to use the tool in classroom practice, with a fair amount of students, is important. This score is also based on earlier experiments with tools.

24 Looks 2 1 3 4 4 3 1 Here we also mean the structure a tool provides.

25 Integration for VLE 3 5 5 5 3 3 2 3=it exists but very experimental, 5=exists and works

Tot. SCORE 285 309 318 374 204 271    

- 67 -

Appendix FNote: these screenshots are from the DWO, the dutch equivalent of the DME.

Here we see the difficulty of ‘open’ questions. Of course just providing the answer is quite easy, but how can we make sure that partially correct answers are graded that way?

The solution in this case was to add this possibility to the answer space. However, this limits the ‘open’ character of the question somewhat, as the student has to provide at least that answer to get all the points.

Rewriting both sides without brackets is possible.

The question remains whether one should give a hint.

- 68 -

This implementation of the second question relies on the expression to be seen as an equation. It remains to be seen whether other expressions with v= are graded as correct or not.

In question 3 substitution is possible. Circularity –in the form of rewriting the left side of the equation- is possible, and can also be awarded points

- 69 -

Here we encounter some difficulties modeling the open character of the question. We could force an expression with n to be used, but the whole point is that that a student could choose his/her own variable(s).Using hints and a more closed instruction could also help.

Integrals can be computed using mathematica in the background. However, there is no function here, as the characteristic holds for all functions.

- 70 -

Just using DME/DWO in a more limited way, not asking steps but only one answer, is also an option.

- 71 -

References