Developing questions for Maple TA using Maple library modules and non-mathematical ... · 2011. 11. 4. · Developing questions for Maple TA using Maple library modules and non-mathematical

Developing questions for Maple TA using Maple library modulesand non-mathematical computation

Bruce CharDepartment of Computer Science

Drexel UniversityPhiladelphia, PA 19104

[email protected]

August 1, 2011

Abstract

Maple TA is a system for authoring, administering, and auto-grading web-based student assignmentsthat has the Maple technical computation system as a back end service. Its strength is its ability tofacilitate authoring, presenting, and automatically grading technical questions with symbolic or numericanswers. We have used it to develop questions for a course introducing problem-solving through technicalcomputation and programming to over 1000 freshmen engineers per year at Drexel University. We havefound that additional Maple programming can extend the flexibility of question generation and answerchecking. We explain how to use Maple modules to generate questions, answers, and answering checkingalgorithms in Maple TA. We discuss how this this has changed our software engineering practices inquestion development and maintenance.

1 IntroductionInteractive computer-based testing has been around since the early days of interactive computing with thePLATO system [1, 11]. Its widespread use in university instruction had to await the favorable economicconditions – institutional budgets for reliable computing infrastructure (networking and web servers), andsoftware systems that make development of computerized study aids, assignments, and quizzes possiblethrough sustainable, conventional amounts of effort by instructional staff.

For the past four years, we have run Computation Lab, a three quarter, one credit course introducingproblem-solving via technical computing at Drexel University. During the period 2008-2011, it was given to allfreshmen engineering students, resulting in enrollments of 600-1000 students per term. The course introducesuse of a technical computation system (in our case, Maple[7, 13]). Maple is an interactive system developedbeginning in the 1980s to support symbolic mathematical computation. It has become a commercial systemwith strengths in symbolic and numeric computation. It also has a document worksheet interface gearedtoward technical word processing and result presentation. As a programming language, it has grown toinclude general-purpose features similar to other interactive languages such as Perl or Python.

We chose Maple because of it allows Palette/menu-driven, WYSIWYG entry of mathematical formulaeand computation using a robust computational library for the standard technical tasks Since technical wordprocessing, visualization, and formula-oriented derivations can be combined with computation in a singledocument, we can in the first lab, present and work on engineering, mathematical, or scientific computationsthat the students see in their other courses.

Students are taught to use interactive calculation to gain insight and to answer questions involving theapplication of mathematical modeling. They are also taught basic scripting and programming to handlemore problems that require more elaborate modeling or simulation. Major course objectives are to havestudents develop problem-solving, planning, and troubleshooting skills that should transfer from Maple toany similar interactive computational system. The course has three components:

1

1. Pre-lab out-of-class activities that prepare students for the upcoming lab. These include reading thecourse materials, and taking an on-line quiz that checks conceptual comprehension and the ability todo simple tasks that students are expected to master on their own.

2. In-class lab activities with face-to-face instruction and support by instructional staff. These activitiesbuild on the understanding supposedly established by the pre-lab activities. They typically provideboth warm-up exercises to check out the student’s mastery of the new concepts in an easy situation,and more challenging applications of the concepts to more realistic modeling and simulation problems.

3. An on-line post-lab quiz/assignment that typically requires more practice with the cycle’s concepts.

4. An end-of-term proficiency exam that asks the students to demonstrate in class that they can useMaple on their own to solve variations of problems that they’ve seen earlier in the term.

We rely on a proprietary system, Maple TA[12], to provide automated support for the testing that occursin items 1,3, and 4 above. Maple TA uses Maple as a back end math engine for a web-based system forautomatic testing and grading. Because the language that Maple TA uses for its computation is the same asthat being used by the students, we can ask programming questions as well as answers with mathematical ornumeric answers. Instructors fill the web server back end database with formatted questions, and constructassignments built from collections of questions. Maple TA account administration gives each student in aclass an account to log onto the system and do an assignment interactively through the web browser. Studentresponses are logged and automatically graded if the question’s author has supplied the grading algorithm forthe question. Instructors (and by permission, students) can retrieve individual and aggregated results for anassignment. Assignments can be configured to allow multiple attempts with instructor-configurable amountsof feedback. At Drexel, we use a dual core 2Ghz system (Linux, with Apache Tomcat as the web service)with 4Gb memory to serve a class of 1000 students in a technical course on computing and programming.Close to quiz or assignment deadlines, there may be up to 150 simultaneous quiz-takers.

Maple TA has been used primarily for on-line mathematics assignments and examinations. Our use of itfor instruction has caused us to use extended amounts of Maple programming to generate and automaticallygrade problems. In this paper, we describe:

1. Our instructional objectives for these extensions.

2. Our use of Maple modules and other Maple programming techniques to achieve them.

3. Software engineering rationale that justifies the extra work typically needed to implement these exten-sions.

2 Blending on-line use of Maple TA with face-to-face instructionWe run our course as a blend of face-to-face instruction in a computer lab with the on-line Maple TA activitiesboth in and outside of the classroom. We use Maple TA to do things that we could not afford to do withthe human staffing resources available to us:

1. It provides a way of requiring and monitoring student involvement with the course between classmeetings. Pre-lab work with course content is crucial to make the in-class lab time productive. Post-lab out-of-classroom practice can consolidate and expand on what has been learned in the lab.

2. We can ask questions that implicitly require procedural “how to do” knowledge in a strong way, since wecan establish question-generation and answer-checking algorithms of arbitrary complexity – anythingthat Maple can be programmed to do. We use these capabilities so that our questions often givedifferent versions of the problem to every student and have answers that are difficult to guess correctlywithout deep insight or non-trivial computation.

3. No longer is it necessary to ask the same question to everyone for the purposes of grading convenience.It is possible to generate questions whose the coefficients used in an equation, or whether a problemcondition was “greater than” or “less than or equal to” varies from student to student under program

2

control. This encourages students to become familiar with the solution process rather trying to workon particular version of the problem. It also makes it easy for the instructor to ask students to practicesolving problems that are highly similar in their solution technique.

4. We estimate that each year Maple TA grades over 120,000 questions (some with multiple parts) for ourclass. We estimate that this allows us to use several hundred hours of staff time per year on face-to-faceinstruction or tutorage instead of grading. We can ask many different questions (or different versionsof the same question, or repeated tests) to a class without incurring more staff time in grading.

5. It provides immediate feedback to students. Computerized feedback may not be as extensive as thatgiven by a human evaluator, but it is available 24/7 and ready whenever the student is willing to work.

6. On assignments, we permit re-tries with immediate feedback given about whether the answer is orisn’t correct. This allows us to expect that students will “work until correct” rather than “stop whenyou have an answer to turn in, and hope that you can pass the course through partial credit”. This isconfirmed by the summary statistics, where problems posted for out-of-class work each typically havea success rate in excess of 90% of items fully correct.

7. We can arrange multiple versions of a quiz or assignment. This allows us to scale our assigned workto dozens of sections scheduled across many days of the week.

8. With access to performance statistics in summary form or a question-by-question, student-by-studentlevel, we can evaluate learning progress as soon as an exam is over, or even while it is in progress.

9. Since an electronic record of student submitted answers is kept until the end of the term, review ofgrading results (with possible manual override) is easy to do even several weeks after a test has beentaken.

10. The availability of assignments can be limited to a pre-set length and window of administration. Thisallows us to create an in-class proficiency exam where computers are used “live” to answer versionsof questions previously seen in assignments and exercises in Maple TA. The results of the exam areavailable to instructors and students immediately after it concludes. Because we ask dozens of questionsvia Maple TA during the term it is straightforward to make up multiple exams of selected subsets ofthe questions. Since we have the summary statistics for every question, we have data to work withthat provides assurances that the exams should be of comparable difficulty. Furthermore, we have theinformation immediately after an exam is over to confirm this. Students may do well by being preparedto correctly answer any of the questions, but cannot predict which questions will be asked on theirversion of the exam. The students can practice with the questions for a week before the exam, withthe same feedback given during the practice as would be given after an assignment. This allows us torun the exam across over 20 time slots without giving extra information from advance knowledge tolater exam-takers. The fact that the questions vary from instance to instance is a key feature exploitedin making unlimited practice with feedback possible.

11. Hints and supplementary explanation can be given conditionally.

Maple TA’s limitations include:

1. Maple TA is a web-based application. The server only records and responds to what is sent to it(answers to questions). It does not reside on the student’s computer so it can’t gather informationon how students are working on the desktop, including what they may have done with Maple togenerate the answers they input into Maple TA. Furthermore, access to performance or student profileinformation (e.g. the student’s identity, or their performance on prior questions or quizzes) is notavailable to the Maple TA question author. Maple TA does not award of partial credit for an answersdue to this1. Because of the limited access to information on student cognitive processes, we havenot tried to create automatic tutoring or remediation such might be possible by software such as theCognitive Tutors developed by Anderson et. al [2, 15].

1One can still ask multi-part questions and award partial credit for getting parts correct.

3

2. It does not allow “survey” questions that ask for opinions or sentiments. We can’t use it to poll what thestudents think or feel about their learning or even about the test questions they are getting. Of course,there are many other ways of conducting on-line polls, but they would require more administrationwork and another tool for instructors and students to get accustomed to.

3. We are fortunate that we are with a subject at a level where the functionality provided by MapleTA auto-grading can be used to cover effectively much of the skills and knowledge. However, weare also not suggesting that Maple TA is a replacement for all forms of instruction. Work withMaple TA provides experience working with problems, but it isn’t particularly effective in explainingor introducing concepts, unless the point is for the student to use methods of their own devisingand reading materials of their own selection to figure things out. In addition to the contact withinstructional staff provided during lab time, we provide staffed optional tutoring and homework helpsessions that are popular with many students during each assignment cycle, particularly as the MapleTA out-of-class assignment due dates loom.

3 Basic Maple TA scriptingIn this section, we discuss how Maple TA works and what its programming language looks like. We assumefamiliarity with Maple’s programming language[5] which is distinct and unrelated.

Maple TA provides a capability on-line testing and question authoring similar in concept to that providedby well-known course systems such as Respondus[17] or Blackboard Vista [10, pp. 80-110],[6]. Questions arecomposed in a WYSIWYG Question Designer, which facilitates the input and layout of question content.The Question Designer the creation for a variety of automatically or manually checked questions: multiplechoice, matching, numeric, text, or mathematical, including responses that use mathematical formulas orthe syntax of the Maple programming language. Once the questions have been created, assignments canbe created using those questions. Reports can be generated whose scope can range from the results of onestudent’s assignment, to summary statistics for the entire class on a number of assignments.

Maple TA has a scripting language to allow question authors to specify algorithms for computing orchecking answers. An example of a short script is given in Figure3. A Maple TA script can assign dynamicallydefined variables with values which can be numeric, string, or derived from Maple results. Random generationand selection of values is allowed. This allows questions to vary between students through the use ofrandomly selected coefficients or other components. The variables can be used in the text of the questionor in the specification or grading of student answers. Maple TA uses a WYSIWYG Question Designer forstraightforward layout of the question. The Question Designer also allows specification of how answers shouldbe checked for correctness if they are free-form text, algebraic, or numeric responses.

An important feature of Maple TA is that the answer checking can reference user-provided Maple pro-gramming as well as randomly-generated values of script variables. For example, in Figure 3, Maple’snumerical solver fsolve is invoked to solve an equation. This features allows questions that ask the studentto do non-trivial amounts of computation and programming to answer them. It also allows answer checkingfor questions to be generated on-the-fly through randomization.

Figures 1 through 6 illustrate a sample question we have developed for our course using Maple TA. Theproblem is an enhanced version of a problem taken from a calculus text[3, p. 99], which we use in ourcourse to revisit and review pre-calculus concepts used in first year technical courses. Figures 1and 2showthe top and bottom portions of a screen that the student would see when they connect to Maple TA andsee a question. Figure 3 shows the Maple TA script (algorithm) for the question. It uses random numbergeneration (the Maple TA built-in function range, which selects a randomly generated integer from thespecified range) to create a formula. It then invokes Maple to calculate some of the quantities used in theposing of the problem and the answers the student is asked to calculate on their own. Those results arestored in the Maple TA script variables $lowtime , $hightime , and $midTime . Maple is also invoked todynamically generate portions of the problem’s presentation to students: $eq , the textual (Maple syntaxform) of the equation to be solved, as well as $qml , the MathML[4] format of the cooling law used for thisinstance of the problem. Portions of the presentation of the question developed in the Maple TA questioneditor is shown in Figures 45. The placement of the MathML result $qml in the Maple TA Question Designer

4

Figure 1: A Maple TA question, part 1

is shown in1. The student sees the presented form of the question (Figure 1) with the MathML is renderedas two dimensional mathematics.

A screenshot from the Question Designer displaying the answer-checking options for one part of thequestion is shown in Figure 6. It specifies that the student answer should be compared against the computedreference answer $lowTime . The Question Designer dialog box allows the question author to specify thatthe student’s response should agree with the calculated answer $lowtime within 0.1% (e.g. about 3 decimaldigits’ agreement) in order to be graded as correct. This typically permits minor variations in floating pointround off or of math library implementation. In this way such questions can be used even if computation isbeing done with hand calculators, a different version of Maple, or another system such as Matlab2.

Figure 7 describes the software architecture of this simple way of using Maple within Maple TA.

4 Five aspects of question development in Maple TA1. Dynamic problem creation. Maple TA developers have the choice between developing static problems –

problems that are the same each time they are given –or problems whose content changes (one that maychange each time a test or assignment is given to a student). For the latter, we characterize questions asinvolving either dynamic generation of the problem (e.g. use the random number generation facilitiesto pick parameter values to be used in a formula for each student as their test is given), or dynamicselection (randomly select from a list of versions of a problem established by the question author). Ofcourse, questions can combine the two approaches as well.

2Particular care must be taken in setting up randomized questions that use floating point computation. Randomization canlead to tricky issues in numerical or mathematical analysis. It is the responsibility of the question author to ensure that domainof questions being randomly generated does not lead to situations where numerical results cause fair automatic grading to bedifficult due to ill-conditioning or being beyond the limits of what the math libraries can handle.

5

Figure 2: Maple TA question, part 2

Figure 3: Maple TA question – the algorithm

6

Figure 4: Initial portion of question in Maple TA question editor

Figure 5: Another portion of question in Maple TA question editor

7

Figure 6: Maple TA question – an answer checking specification

Figure 7: Schematic of a simple Maple TA problem implementation

8

(a) Dynamic generation through randomized coefficients. The script in 1 illustrates dynamic genera-tion where randomly chosen numbers are used in formula coefficients and numerical conditions3.Generating a problem then becomes selecting values for each parameter, typically through ran-dom number generation for each parameter. Visualizations (graphs or animations) based on theformulae that thus arise can also be generated dynamically.

(b) Dynamic selection. As an example of dynamic selection, the multiple choice problem in Figures19-21 selects the options randomly from a collection of correct and incorrect responses built intothe problem’s algorithm. Maple’s combinat[randperm] function is used to scramble the lists ofcorrect and incorrect options. The first four options of the scrambled list $named are selected,and the first of the scrambled list $unnamed is picked. The “permuting” choice in the questioneditor specifies that the five options will be randomly permuted between versions of the question.Students taking a quiz more than once, or a collection of students taking the same quiz, will seedifferent options in their versions of the question, with the correct answer’s position varying.

(c) Dynamic formula and code manipulation. Maple can perform symbolic manipulation of expres-sions and of programs. The “first class citizenship” of expression and procedure data types fa-cilitates the programming of such manipulation. This allows operations, not just coefficients, tobe changed dynamically – for example having a problem which might specify the use of either“greater than” or “less than or equal to” as the selection criterion for searching through a list ofvalues. Symbolic manipulation can also be used to vary expressions to be used, or to vary theorder or nature of the steps used in a solution procedure. This advanced technique is discussedfurther in section 7. Varying the problem in this way also typically requires dynamic generationof the problem description, discussed further in Section 3, since such changes typically requiregoing beyond substitution of coefficients to rewording the text of the problem.

2. Answer generation. Whenever problems are being generated on the fly, the problems’ answers need tobe calculated dynamically as well. We discuss this further in Item 3 of Section 6.

3. Presentation generation. A variety of physical situations often have identical or highly similar mathe-matical models. For example, one can use exponential growth/delay formulas in word problems havingto do with baking (as in Figure 1), voltage of a charged capacitor, time-of-death body-cooling prob-lems, etc. The computational algorithm to generate problems and answers can be used for all of theseproblems. Once this is realized, it is a small step to add a little more machinery to generate the text ofthe problems as well as the mathematical formulae and answers, for example setting up the problem totalk of electricity and units of electrical current, versus talking of fluid flow and units of liters/minute.In this way, problems can be devised which ask students to practice their skills at mapping problemdescriptions to mathematical models, rather than in just doing calculations. Rather than typing a fixedproblem statement into the WYSIWYG Maple TA Question Editor, we include Maple programmingto dynamically create a string with HTML coding that presents the question. A Maple TA variable isassigned the result of a Maple printf of a the Maple string. The Maple TA variable is then referencedin the Question Editor. We can use this technique to dynamically generate tables with formulas or nu-merical data, or references to dynamically selected JPG figures or animations. The “Spline 3” problemin section D illustrates presentation generation.

4. Grading (answer checking). Automatically checking a student’s mathematical answer is not typicallyjust a matter of seeing whether the student input is exactly the same as the solution computed by theauthor or the answer generator. When numeric results are computed through floating point arithmetic,the non-commutativity of floating point arithmetic may produce small variances in student results fromthe application of steps in a slightly different order, or even possibly due to changes in the floating-point solvers between different versions of Maple. For these reasons, question authors often decide thatanything within a small range around the reference answer should be accepted using notions of relativeerror, or a certain number of digits’ correctness. If an answer is an algebraic formula, the student may

3The script does not go as far with randomization as it could. For example, in addition to randomly generating eachnumerical coefficient or exponent in the heating formula, it could also randomly generate the numbers for the beginning andending of “doneness”.

9

give a mathematically equivalent syntactically different response, e.g. x2 + y2 versus (x − y)(x + y).Fortunately, Maple TA provides built-in options for doing this [12, "A question with Infinitely ManyCorrect Answers", p. 97; "Numeric Question using the Question Designer", p. 35]. However, thereare situations where the built-in support for answer checking does not go far enough. Since part of ourcurricular agenda is to teach about data structures, some of our computational problems sometimesinvolve answers that are expressed as lists, vectors, or matrices of results. Sometimes the answersinvolve nested data structures, such as a list of lists. Each item in the structure needs to be checked item-wise against the reference answer possibly using numerical tolerances, or filtered through equivalencetesting. (An example of tolerance checking on structures occurs in $withinPairlist (Lines 53-56,section C.2).

5. Grading (procedure testing). Another part of our curricular agenda is to have students construct anduse procedures as part of their training in programming. Therefore, some of our questions ask studentsto supply a procedure as a answer. It can be difficult to determine functional equivalence of a student’sprocedure to the reference answer. We usually settle for checking that the student’s response agrees(within tolerance) with the reference result when executed on a set of test data. As part of the designfor questions with procedure testing, the question author needs to decide if they will reveal all testdata in the problem description, or leave some to be revealed after testing is complete. They also needto decide whether the test data and results will be generated statically or dynamically.

5 Anti-gaming strategies and the complexity of question authoringAny on-line system when used in a loosely supervised way out of the classroom, can be subjected to studentmanipulation of results. Here are some of the gaming strategies we have considered, and how we dealt withthe in Maple TA.

5.1 Static versus dynamically created questionsStatically generated questions can be answered without knowledge of the problem-solving process, if thestudent prepares by memorizing or copying the answer of someone else who has already taken the test andknows that their results have been graded correct. Randomization ensures that the problem-solving process(rather than “character string copying”) must be applied to each instance of student work.

5.2 Avoiding the “manual processing” strategyAs mentioned in item 5 of section 4, the difficulty of evaluating the correctness of code automatically naturallyleads question authors to ask students to input the results of executing the procedure, which are typicallymuch easier to grade automatically. However in our course about computation, the question author needsto set up the question in a way that requires the student to do the desired execution on a computer. Forexample, asking the student to write a procedure to find the minimum of a list of numbers, it is importantto avoid giving as the only tests lists that have only a handful of items. Given the amount of time it maytake a novice to write a program, it may often be the case that scanning a list of even fifty or 100 items forthe minimum will take less time than writing a program to do the processing. Giving very long lists (e.g.one of hundreds of items) changes the balance. This requires more work in presentation to ensure that thetests are clearly presented.

5.3 When feedback is a deterrent to learning – discouraging solution by ex-haustive enumeration

Usually feedback on errors is a good thing, as it helps the students learn how to avoid them. However, thepresence of feedback changes the way questions must be authored.

The nature of the grading in Maple TA means that either the student is given full credit or no credit atall for a particular graded item. Since Maple TA cannot see the work done to produce the answer and it isdifficult to get it to detect when the answer, while wrong, is close to correct, it is not possible to give partial

10

credit. This places the grader in the position of giving no credit for answers which would be quickly correctedin “real life” when there is feedback about syntactic correctness. For example, requiring completely accurateinput the first time would give no credit someone who through imprecise mouse motion omits to paste inthe leading bracket – a [ – of a long list of items copied in from an answer calculated in a Maple worksheet.To avoid extra work manually overriding such “unfair” evaluation, we typically set up assignments to allowindefinitely many retries with “how did I do” turned on. While this allows us to make it reasonable to insistthat the student try until they get things completely correct, we have found that this allows students toanswer question without using exhaustive enumeration rather than the desired problem-solving process. Forexample, if the question is “how many of the following list of times does the simulation return a positiveresult?”, it may be known that the answer must be between 0 and 15 if there are fifteen numbers in the list oftimes. The question could be answered by the student trying possibilities exhaustively, just using “how didI do” after entering each of 0, 1, 2, etc. While exhaustive trial is a useful problem-solving strategy in somesituations, we don’t want it to significantly distract students from pursuing learning about the modeling andsimulation that’s our course domain.

This strategy can be applied even when the answer desired is a floating point decimal number. Forexample, if a problem gives a function described by a formula, and then asks “at what value of x does thefunction achieve its maximum”, then a student could come up with an imprecise estimate of the result byjust from real world considerations associated with the problem. If the tolerance is set too broadly (say, twodigits accuracy), then the student can come up with a correct answer by exhaustively trying every two digitpossibility in the range. What this is saying is that problems set up this way are not the right way to useMaple TA to ask the student to demonstrate that they can use an optimization routine.

One strategy to take is to ask for two or more answers at once in a list, and use the greater number ofpossibilities to reduce the chances of successful guessing. For example, if we break up the list of 15 timesinto a list of 7 and a list of 8, and require an answer such as [2,3], then guessing must cover 56 possibilitiesrather than just 15.4

We could probably reduce the amount of such behavior if Maple TA had a way on limiting the numberof retries to some reasonable value between “none” and “infinite”, but until that happens, we aim to writequestions that make exhaustive trial an unlikely strategy for success. In addition to asking for numericalanswers to enough accuracy, one can ask for multiple answers in a single list, which increase the number ofpossibilities combinatorially.5

5.4 Anti-gaming strategies require more elaborate programmingOur anti-gaming measures typically rely on dynamic creation and more elaborate answer-checking, both ofwhich increase the amount of programming that goes into a Maple TA question. This has naturally led usto consider programming practices that will lessen the costs of Maple TA question development. In the nextsections, we discuss this.

6 Software engineering considerations in the development of MapleTA questions

We observe the following limitations to developing questions that use Maple embedded in a Maple TA scriptand entered or edited in the Maple TA Question developer:

1. Multi-line Maple procedures need the conventional tools for textual software development. Extensiveamounts (dozens or even a few hundred lines) of Maple code may be needed for some of our questions.It is difficult to easily read or maintain that amount of code in the Maple TA Question Developerenvironment, since source code manipulation is limited to the point-and-click capabilities of the De-veloper’s text editor which lacks tools such as regular expression search, global replacement, and othercode reformatting or refactoring features.

4We have seen students work through dozens of possibilities in an exam situation, so even 56 may not deter the desperate.5Other possible policies that would simplify anti-gaming: allowing grading to take into account multiple answers, or allowing

assignment/quiz authors to turn off “how did I do” selectively to some questions.

11

2. Multi-line procedures need documentation. The Maple TA scripting language does not yet have anability to comment code except in workarounds such as defining dummy script variables whose stringvalues contain comments. Maple source code allows conventional programming comments.

3. Random creation of problems needs support for automated testing. Dynamic selection or generation ofquestions often requires non-trivial amounts of testing of a large number of cases. We have found thatdespite our best efforts in analysis, it is not easy to avoid problems with randomly generated coefficientsthat result in nonsense versions of a problem (versions where the solutions would not result in aphysically realizable situation, equations that have no solution, equations which the Maple solvers findintractable or solve only after a large amount of time, etc.) . Testing is needed to have confidence thatproblem generation almost always results in a solvable and sensible version of the problem. The MapleTA Question Developer environment, like much of Maple TA, has a GUI/user-operated philosophyinvolving multiple clicks to navigate through question generation, algorithm/script execution, testanswer entry, and answer checking/grading. Manual case-by-case testing is uneconomically labor-intensive for most organizations. The bottom line is that more elaborate problems need an environmentthat supports more automated testing.

4. More robust answer checking needs support for automated testing. If student answers are data structuresor Maple procedures, as they are in our course, then the built-in tools for answer checking and studentfeedback must be supplemented with author-supplied procedures. For example, if a student is expectedto enter a list of 1000 rational numbers as the answer to a problem, then it is usually insufficient totell them “no, you didn’t enter a correct answer”. They may have forgotten to include the openingor closing bracket of the list, put in a trailing semi-colon, or included some floating point numbersbecause they haven’t learned that they are different entities than exact fractions. Adequate testingof the answer-checking algorithm requires that it be given a variety of test cases to ensure that itdoesn’t inadvertently grade an answer as correct if it is off-beat or unusual. This leads to more needfor automated multiple-case testing.

5. Extensive use of random numbers in problems benefits from control of the random number seed. De-bugging questions that use random number generation requires access to and control of the randomnumber seed, so that it is easy to reproduce the situation where bugs arise. Maple random numbergeneration facilities (eg. randomize) allow such control, although apparently Maple TA’s facilities suchas range do not [12, p. 74, pp. 81-87]. We have adopted the practice of setting the seed at the start ofthe Maple TA script by calling rint, a Maple TA procedure designed to produce random values. UnlikeMaple randomize, Maple TA rint guarantees that different random number sequences are used evenin concurrent student sessions[9]. We try to ensure that subsequent random number usage is based onthe original seed.

6. Development of multi-line Maple procedures needs access to Maple-generated error messages. MapleTA scripts do not show error messages arising from Maple computations performed within it[14, ,pages 140-145]. For example, running $result=maple(“solve(x=x/0,x)”); results in the messagejava.io.IOException: Maple computation error, rather than the more descriptive message numericexception: division by zero that results when the solve is executed in Maple itself. More seriously,Maple TA does not indicate where within a multi-line Maple procedure an error occurs. This leadsone to want to do most debugging of Maple code outside in the ordinary Maple environment so thatthe full diagnostic information from error messages can be seen6.

These considerations lead us to conclude that while Maple TA scripting is an environment conducive to thedevelopment of problems handled by a few lines of Maple TA or Maple code, more complicated questionsshould be developed in the ordinary Maple development environment, where most of the testing could thenoccur. Only after most of the code has been developed should a move to the Maple TA execution environment

6A change in how Maple TA deals with Maple errors would change the balance of costs and benefits between developing inthe Maple TA environment versus that of ordinary Maple. Some changes, though worthwhile, might be expensive to implementas they would require significant changes to how Maple’s run-time system. Changes could also be inadequate if they haven’tbeen verified to meet needs of developers.

12

be attempted, based on the assumption that such movement won’t require major code revisions. Some testingwithin Maple TA is necessary to check appearance and correct invocation of Maple, of course.

In the next section, we discuss a technique to do this: Maple programming using Maple library modules.

7 Creating questions in Maple TA with Maple library modulesUse of Maple library modules in Maple TA questions provides a way to address the development and testingissues described in Section 6. While Maple TA has features specifically included to encourage the use oflibrary modules in questions, the methodology is not well-explained in the Maple TA User Guide. In thenext few sections, we present details of how to use them for questions. We also present further issues thatwe have seen arise when this style of Maple programming is used.

7.1 Creating a Maple module in a library file and testing it on the desktopBriefly, the method is:

1. Create the Maple procedures for your Maple TA question and package them together as a Maple moduleon your own computer [5, Chapter 8, "Programming with Modules", pp. 323-364]. The procedures inthe module should provide means for problem generation, question presentation, and answer checking.Use the standard Maple development environment to develop source files defining the Maple module.The procedures you will want to invoke from the Maple TA script should be exported members of themodule.

2. Create the library files (.lib and .ind) for the module using the LibraryTools package from the Maplelibrary and then upload them to the Maple TA server. See [5, Ch. 10 "Writing Packages", pp. 387-404]for a description of setting up a library package.

3. Develop test suites for your Maple module and run them on the desktop. These tests can check thecorrectness and timeliness of randomized problem generation, question presentation, and the robustnessof answer checking.

Appendix E describes a shell script that automatically includes the Maple code form LibraryTools to dothis. It also automatically invokes the library on designated tests after it has been created.

7.2 Using the Maple module in a Maple TA questionOnce the module has been developed and tested, the technique for using a module in a Maple TA questionis:

1. To make the package accessible from Maple TA, it must be uploaded to the developer’s Maple TAserver. Using the Web Editor feature of Maple TA (see[12, pp. 67-68]), create a folder on the serverfor the library. (See Figures 8 and 9). Then upload the .lib, .ind, and any other supporting file for thequestion to that folder on the Maple TA server (Figures 10 and 11). The code is not directly readableby student users of the Maple TA server, so details of the problem can be securely stored on the server.The Web Editor feature allows uploading and automatic unpacking of .zip files, so one can often savea few clicks by creating a .zip file with the .lib and .ind files together, along with any other files suchas .gif graphics files for the question. The shell script described in Appendix E automatically createsa .zip file with with the .lib and .ind files in it after they have been created.

2. To use the module in a Maple TA question, use the Maple TA Question Editor to begin creatingyour question. Use the GUI interface of the Question Editor to select the .lib file of the module in thedirectory you created in step (1). This will insert the proper name through its point and click interface,as illustrated by Figure 12. Once the name has been revealed, you can textually copy and paste thereference to the module into subsequent invocations of module procedures within the Maple TA scriptusing the editable text fields displaying the question’s algorithm.

13

Figure 8: Maple TA Web Site editor.Folder Creation icon highlighted in pink. Upload icon next to that.

Figure 9: Sensor2 directory after it’s been created. Subfolder, upload, and delete icons highlighted in pink.

Appendix C.3 lists the code for “Sensor2”, the name of the module encapsulating the computation for aproblem. Appendix C.2 shows a complete Maple TA script that references procedures from a module storedon a Maple TA server.

8 An example of symbolic manipulation used in dynamic creationof questions

Maple permits symbolic manipulation of expressions and of programs. This allows dynamic generation to gobeyond substitution of numerical coefficients. Symbolic manipulation can also be used to vary expressionsto be used, or to vary the order or nature of the steps used in a solution procedure. This typically requiresdynamic generation of the problem description as well since such changes usually require rewording of theproblem rather than just substitution of numerical values in formulae or the problem statement. The pos-sibility of creating significant different problems through this kind of manipulation means that the questionauthor usually introduces a need for more analysis to avoid variants will that are too difficult or easy.

Figures 13 and 14 give two versions of a problem created dynamically. Not only do the different numbers

14

Figure 10: A zip file being uploaded

Figure 11: After zip file uploaded. It contained .lib and .ind files, as well as copies of source to the questionsand tests.

15

Figure 12: Maple TA question designer with reference to module Sensor2

The designer has specified that the variable $problemSpec should be assigned the result of the Maple com-putation Sensor2:-problemGen($seed), where $seed is the value of the random number seed established in aprior step of the Maple TA script. The code for Sensor2 is in .lib and .ind files on the Maple TA web serverin the LIB location, which has been generated using a pop-up window (not shown) invoked by clicking onthe Maple Repository button.

used in the list and for the selection criteria, the problem also mandates different selection criteria. Theproblem generator must manipulate the code templates for the solution, substituting in the dynamicallyselected operation that performs the specified selection. It then invokes the instantiated code template togenerate the numerical answers. The anti-gaming considerations mentioned in Section 5 led us to create liststhat are long enough so that the problem cannot be solved easily just through visual inspection, but shortenough to be easily copied between the browser and whatever screen or file the calculation is using for input.

Appendix B lists the Maple TA and Maple programming for this question.

9 Software engineering advantages of question development throughmodules

We have found that most of the question development described in Section 4 can be carried out by Mapleprogramming rather than Maple TA programming. If the Maple programming is placed in modules, thenthe Maple TA script for a question can be reduced to initializing a few key values (typically, the randomnumber seed, and the specification of problem size/difficulty parameters) and disassembling the results fromMaple into individual Maple TA variables that are used in the presentation or in answer checking. If this isdone, then most of the debugging and testing can occur on the question developer’s desktop machine in theconventional program development environment. Using this environment has the following advantages overthe Maple TA question developer:

1. We can use a full-featured text editor (global search and replacement, reformatting tools, etc.) todevelop the Maple programming.

2. We can use shell scripts to automate compound tasks such as module creation and testing. Our shellscript shell script makelib2.sh (see Appendix E) is an example of this. It automatically combinesmodule creation with execution of a test script for the module.

3. We gain access to Maple parse-time and execution-time error messages (recall Item 2 in Section 6)during testing.

16

Figure 13: A temperature calculation

17

Figure 14: Another version of the temperature problem

18

Figure 15: Schematic of a Maple TA problem invoking a Maple module

This reduces an iteration of edit-generate-test for the Maple programming of question to a matter of seconds,compared to the minute or two it takes to upload Maple programming onto a Maple TA server, and thenmanually execute a test in Maple TA. Even though Maple lacks an integrated development environment suchas Eclipse [8] or extensive tools for testing, we have become more productive.

We note that making most of the question’s source code executable on the desktop does not addresssome testing needs. The Maple TA script and answer-checking specifications must still be tested. Typicallythis must be done manually through the standard Maple TA interface, although web-testing harnesses suchas Selenium [16]hold promise for further testing automation in the future. In addition, the presentation ofthe problem description must also be checked for technical correctness, and that the problem as presentedworks well for the intended (student) audience in the production environment. For the foreseeable future,some amount of manual testing and feedback from human testing will always be necessary.

10 System limitations and programming workaroundsWhile Maple modules facilitate ambitious question authorship, there are downsides to larger computationsin Maple TA questions. In this section, we discuss two of the more significant we have run into.

10.1 The work of disaggregationMaple TA limitations make it difficult to return multiple results from Maple without a tedious sequence ofadditional operations. The natural impulse in problem generation is to have a single procedure that generatesall or many parts of the answer, and returns everything needed in a single data structure to Maple TA suchas a list. Using switch[14, p. 232], parts of the data structure could be assigned to individual Maple TAvariables and used for presentation text, answers, answer checking algorithms, etc. While this works in simplecases, the Maple TA scripting language makes it difficult to disassemble aggregated information from Maplewhen results contain subparts, such as a lists of lists. For example, in Figure 16, a Maple result consisting ofa lists of lists is created and assigned to the Maple TA script variable $result. For the first few operations,the Maple TA operation switch can be used to extract a part of the result computed by Maple. However,the third item of the list is a list itself, and the part extraction by Maple TA just gives a part of it. Thefourth item extracted through switch shows that Maple TA is using fundamentally different semantics fromMaple, in that “e” is interpreted as the base of the natural logarithm and approximated, rather than as a

19

symbol as it is treated in Maple. The workaround is to pass the result back to Maple and to extract it theresuch as occurs with $r3m in Figure 22 but this involves the overhead of another invocation of Maple. Ourdifficulties would be eased significantly if Maple TA adapted a feature that would allow multiple assignmentfrom a Maple result. We discuss our techniques for facilitating disaggregation in section 10.5.

It is difficult to break a large Maple computation into several parts in the Maple TA execution environment.Conventional software engineering says that a good way to handle task complexity is to decompose it intoseparate parts, and to execute each part in sequence, or in parallel as is appropriate. However, invocationof Maple in Maple TA is “stateless”: subsequent calls to Maple start with a fresh copy of Maple. Thusprior results must either be recomputed within the subsequent call, or they must be fed back into Maplefrom Maple TA variables that recorded the previous invocation’s results. We have not yet determined aconvenient way of doing this kind of daisy chaining that avoids tedious amounts of coding and non-trivialcomputational costs.

10.2 Three operational limits to returning Maple results to Maple TAIn addition to the “pain of coding”, we have encountered two limits to problem generation results:

1. An ambitious Maple computation may run too slowly in a production test-giving environment. If theauthor of a question gets sufficiently ambitious, the cost of generation and answer-checking for aquestion can take several seconds of cpu time. In our production environment, where over 100 studentsmay be trying to take a quiz at the same time on a dual processor server, any question that takes morethan a second of system time to produce may cause noticeable delays in response to the quiz-takers.Lengthy disaggregation or daisy-chaining sequences that pass large results back and forth betweenMaple and Maple TA are a further cause of additional slowdowns. We discuss our techniques fordealing with good problems that take too long to run in 10.4.

2. Maple results, if too large, cannot be returned to Maple TA. There is a currently undocumented (andhence, to the Maple TA question author, unknown) limit to the size of results that can be returned fromMaple to Maple TA. The limit appears to be less than a hundred kilobytes. This means that there aresome things that can be easily computed in less than a second of CPU time in a contemporary computersystem that cannot be used in a Maple TA question.We have run into this limit with elementary listprocessing programming problems with non-”toy” input sizes. We have also encountered this whengenerating Maple animations used for question presentation. We discuss our response to this in sections10.6and 10.7.

3. It is difficult to return enough diagnostic information to students when running the results of testson student-submitted procedures. As mentioned in Item 6 of Section 6, there is easy way to conveyMaple run-time or parse-time error messages for student submitted programs. While the programmablefeedback mechanism in Maple TA allows one to provide some automatic guidance (e.g. that the typeof result returned by the student procedure does not meet the type specifications, or that certain testsare failing), there are few tools to make this easy.

10.3 Workarounds to avoid grading codeEven though a major objective of the course is to get students to develop their own computational scriptsand procedures to solve technical problems, Maple TA does not strongly support grading student code. First,the “Maple syntax” entry box for answers does not accept scripts – multi-line sequences of code that are notMaple procedures. Secondly, as mentioned in Item 6 in Section 6, Maple TA does not provide diagnosticmessages when student-submitted procedures are executed during answer-checking. Thus students have littleinformation about why their code is rejected if it has syntax or run-time errors. While one could argue thatthis reinforces the lesson that students need to be responsible for getting their code to work before submittingit, the reality is that hundreds of students come for help during consultation hours because they don’t thinkthey have to means on their own to figure out why their code isn’t working.7

7There are also limitations in the kinds of procedures that Maple TA can accept. For example, procedures that contain thecharacter <tt>” </tt> – such as any procedure that uses a string constant – cannot be given as input to Maple TA even if

20

Figure 16: Maple TA script extracting parts of a list

Because of this, we have adopted the strategy of asking questions in a way that requires execution ofcode only on the student’s machine rather than on Maple TA,. In other words, we ask the student about theresults of running the program (or of implications of the results). We do not completely avoid submissionof code to Maple TA, but it becomes feasible for our large class only when the students have gotten to thepoint where basic procedure creation and execution is not a major problem for most. Having the ordinaryrun-time and parse-time messages from Maple displayable from Maple TA when grading answers given inMaple syntax would allow us to shift back to asking code/code testing problems more straightforwardly.

10.4 Workarounds for lengthy generation timesIt may not be necessary to generate problem versions on-the-fly. Pre-computing several dozen differentversions, storing them on the server, and using dynamic selection may meet pedagogical and exam securityneeds. Pre-generation requires an extra layer of Maple and shell programming to drive the generation andstorage of results. This extra work is balanced by an advantage to pre-computing, that one can exhaustivelytest problem versions, rather than trying to prove through theoretical analysis or statistical sampling thatthe the generation process is without fault.

Retrieval of pre-computed problems may generate its own timing/performance problems. If one wanted1000 different versions of a problem, with each version’s description needing 10Kb of file space, then dynamicselection would require reading in and parsing/processing 10Mb of Maple expressions. We experimented bygenerating Maple lists of integers of varying sizes, and timed how long it took to generate a simple questionthat involved the list. Table 1 shows that the time for file i/o and Maple TA processing is for files ofthis size is significant. Using questions that accessed stored data in a production environment with manysimultaneous students could lead to response problems or time-outs without a quiz being generated.

If it takes too long to read in all the precomputed problems, the approach could be refined by spreadingthe data for the problem versions onto multiple files. Dynamic selection would then include computing thefile name containing the problem to be selected.

10.5 Workarounds for limits to the size of results that can be returned fromMaple

There are undocumented limits to the length of results returned from Maple to Maple TA. We have encoun-tered these limits when generating animations during the question-generation activities. Often times it is

they are valid procedures in Maple.

21

Length of list in question 104 106 107

Size of .lib file (in kilobytes) 89.2 8,796.5 87,958.8Size of .ind file (in kilobytes) 40.9 40.9 40.9

Size of zip file 66.5 6,562.1 65,615.2Time to generate assignment < 2 seconds ~3 seconds ~21 seconds

Table 1: Time to generate a question reading data from a Maple library file stored on Maple TADetails of this experiment are given in Appendix F.

not possible to return multiple animations in a single result. To work around this, we have either resultedto daisy-chaining the problem generation (see Section 10.1), or precomputation (Section 10.4). Either way,there are additional costs in programming.

10.6 Workarounds due to length limits in answers returned to Maple TA fromMaple

In Section 5 we discussed the need to use larger problem sizes when the subject of the question is a simple tounderstand computation task. While Maple TA makes it easy to grade numerical or formula answers, it doesnot make it easy to grade code answers. The issue is that if one checks test results without inspecting thestudent’s coding, then there’s no explicit proof that the student used computation to get the answer. Shortproblems sizes for simple computational tasks can be done by hand. If one is setting up a data structuringproblem where the solution is a list with 50,000 floating point numbers, there’s currently no simple way youcan get the answer computed by the problem generator in Maple into Maple TA so that it can be used tocheck the student answer. The most obvious ploy is to generate shorter problems. However, for some classesof problems, where the programming activity is non-trivial even if the motivation is not, it’s important tohave problems and results large enough so that computation cannot be avoided. For example, if one gavea sorting, searching or counting exercise but only had problems with ten items in it, the problem could bedone by the student manually “by inspection”. A problem requiring the processing of several thousand itemsnecessitates computation. Hopefully one can find problems that are acceptably large but small enough tonot run into the Maple-Maple TA size limitation.

If the problem is too large, the only recourse is to find a smaller problem that will fit. If the problemdescription can be made to fit within the limit, there may still be a problem with the answer being too large toget from Maple to Maple TA. However, it may be possible to be reasonably certain that the student’s answeris correct even without access to the entire answer. For example, one could check a randomly generatedsubset of several items from the student’s answer. The answer generator can easily return the indices of thesubset as well as well as the items from its reference answer.

This kind of probabilistic checking is obviously different than what we’d do with a mathematics problemwith a short fixed answer, but indicates the different needs that a computational course that demands thesolution of problems that cannot be done through paper and pencil.

10.7 An example of using workarounds: sizable animationsWe have on occasion generated animations to include as part of the narrative of a question, to help the studentbetter understand the problem being solved. However we have found either that the animation generated istoo large to send between Maple and Maple TA, or that it takes too long to generate dynamically.

Figure 17 shows an example of this. The problem builds off a lab exercise where students learn how towrite control programs for a model robotic car moving around an arena with obstacles (shown in green) andtargets (show in blue). The current position of the car is shown in red and the path of taken so far in thesimulation is shown in grey. The object of the problem is to write a control program that sends the car formits starting point through the hole in the barrier to a post, where it then turns left and heads towards a target.Two examples of desired control program’s behavior on two test cases is given. Each case is presented as ananimated gif which is displayed in the web browser as a continuous animation of a few dozen frames each.We’ve found it too time consuming to generate these animations on the fly. Furthermore, the Maple-Maple

22

TA size limit would not let us transmit the animations through the Maple-Maple TA connection even if wewere to compute them in this way.

We’ve handled this limitation by combining the ideas of the previous few sections: pre-compute theanimations and generate animated gif files, storing them in separate files whose names differ in an obviousway through varying an index. The problem generator dynamically selects the version of the problem. Thenames of the files are assigned to Maple TA variables and referred to in the presentation. Maple TA doesnot appear to have performance issues sending the animated gif files to the student’s browser.

The Maple code segment in Figure 18 generates an animation in Maple and saves it as a animated gif inthe specified file after printing at the terminal the full path of the gif file. This segment is part of a largerscript where the value of id1 is varied as the various animation cases are generated.

The image tool in the Maple TA question editor can be used to display a file whose name is generateddynamically (e.g. $gifLocation). The Maple TA code randomly selects values for id1 and numTrials fromthe permissible ranges, and generates the file name of the appropriate gif. These values are also given to theMaple module for the problem to allow it to retrieve the other information necessary for presentation andgrading for the selected problem.

11 ConclusionWe have presented a way of developing Maple TA questions so that Maple’s mathematical, random numbergeneration, and text processing capabilities replace most of the use of Maple TA. This reduces the role ofMaple TA to labeling results computed through Maple, generating the original random number seed, andusing its GUI for question layout and answer-checking. Software engineering advantages include a betterorganized and document code structure, access to better code development tools, and more easily automatedtesting.

We have also discussed some of the issues in developing good Maple TA questions for our course –anti-gaming issues, workarounds due to size and performance limitations.

We believe that the advantages of the approach outweigh the work needed, but look forward to systemenhancements that would further facilitate such software development.

Acknowledgments We gratefully acknowledge the help of Mark Boady in the preparation of the codingexamples used in this paper. The Department of Computer Science of Drexel University (Jeremy Johnson,department head) provided the support for the development of the course materials. Jeremy Johnson andDavid Augenblick provided thoughtful feedback on the course, course questions, and this paper. Carl Hick-man and others in technical support at Maplesoft have answered many questions from us, including some ofthe technical details of creating using Maple modules in Maple TA.

References[1] D. Alpert and D. L. Bitzer. Advances in computer-based education. Science, 167(3925):1582–1590,

1970.

[2] John R. Anderson, Albert T. Corbett, Kenneth R. Koedinger, and Ray Pelletier. Cognitive tutors:Lessons learned. Journal of the Learning Sciences, 4(2):167, 1995.

[3] Howard Anton, Ira Bivens, and Stephen Davis. Calculus. Wiley, 8th edition, 2005.

[4] Ron Ausbrooks et al. Mathematical Markup Language (MathML) Version 3.0. Technical report, W3C,June 2011.

[5] L. Bernardin and others. Maple 15 Programming Guide. Maplesoft, 2011.

[6] Peter Bradford, Margaret Porciello, Nancy Balkon, and Debra Backus. The Blackboard LearningSystem: The Be All and End All in Educational Instruction? Journal of Educational TechnologySystems, 35(3):301 – 314, 2007.

23

Figure 17: Example of gif animations used for an assignment problem in Maple TA

Figure 18: Maple code segment for saving an animated gif in an index file

f i l ename := cat (" SecondTurn1−", id1 ,"−" , numTrials , " . g i f " ) ;p r i n t ( cu r r en td i r ( ) , f i l ename ) ;p l o t s e tup ( g i f , p lotoutput=f i l ename ) ;p r i n t ( i n t e r f a c e ( p l o td ev i c e ) , i n t e r f a c e ( p lotoutput ) ) ;p r i n t ( carMovie ( s ta teTab le ) ) ;

24

Figure 19: A randomized multiple choice question

[7] Bruce W. Char, Gregory J. Fee, Keith O. Geddes, Gaston H. Gonnet, and Michael B. Monagan. Atutorial introduction to Maple. J. Symb. Comput., 2:179–200, June 1986.

[8] Eclipse Foundation. Eclipse documentation (3.7 indigo), 2011.

[9] Carl Hickman. Personal communication. Multi-threaded/multi-process nature of rint versus Maplerandom number facilities., July 2011.

[10] Blackboard Inc. Blackboard learning system designer and instructor reference – vista enterprise license(release 8). Manual, documentationion version 8.0.0, 2006.

[11] Kirk L. Kroeker. Celebrating the legacy of PLATO. Communications of the ACM, 53(8):19–20, August2010.

[12] Maplesoft. Maple T.A. 6 User Guide. Maplesoft, 2010.

[13] Maplesoft. Maple 15 User Guide. Maplesoft, 2011.

[14] Maplesoft. Maple T.A. 7 User Guide. Maplesoft, 2011.

[15] Peter Pirolli and Margaret Recker. Learning strategies and transfer in the domain of programming.Cognition and Instruction, 12(3):235, 09 1994.

[16] Selenium Project. SeleniumHQ web application testing system – Selenium Documentation. http://seleniumhq.org/docs/.

[17] Respondus. Respondus 4.0 User Guide. Respondus, Inc., Juluy 2011.

A A dynamically selected multiple choice question

A.1 Problem descriptionThe problem tests whether students have read a particular passage in the course readings. The question asksfor the selection of one correct answer out of several possibility. The question author creates lists of correctand incorrect answers. When a version of the problem is generated for a student, correct and incorrectanswers are randomly selected from the lists. In question layout the incorrect answers are listed before thecorrect one, but the question format specifies that the student will see these in randomly scrambled ordermaking it difficult to predict the position of the correct answer.

A.2 A screenshot of the question

A.3 Maple TA script selecting options

1 $seed=r i n t (10 ,100000) ;2 $named=["Python " , "Matlab " , "C" , "Mathematica " , "C ++", "Java " , "MuPad" , "

Octave " , "Sage " , "Macsyma" , "Axiom " ] ;3 $unnamed=["C Sharp " , "Lisp " , "Visua l Bas ic " , " Per l " ] ;

25

Figure 20: Maple TA answer layout for multiple choice question

4 $named=maple ( " [ ‘ Python ‘ , ‘Matlab ‘ , ‘C‘ , ‘Mathematica ‘ , ‘C ++‘, ‘ Java ‘ , ‘MuPad‘ , ‘ Octave ‘ , ‘ Sage ‘ , ‘Macsyma ‘ , ‘Axiom ‘ ] " ) ;

5 $unnamed=maple ( " [ ‘C Sharp ‘ , ‘ Lisp ‘ , ‘ V i sua l Basic ‘ , ‘ Perl ‘ ] " ) ;6 $p i ck f ou r=maple (" St r ingToo l s [ Randomize ] ( $seed ) ;7 combinat [ randperm ] ( $named) ") ;8 $pickFunct ion=maple (" (p , i ) −> p r i n t f ( convert (p [ i ] , s t r i n g ) ) ") ;9 $a=maple (" apply ( $pickFunction , $p ick four , 1 ) ") ;10 $b=maple (" apply ( $pickFunction , $p ick four , 2 ) ") ;11 $c=maple (" apply ( $pickFunction , $p ick four , 3 ) ") ;12 $d=maple (" apply ( $pickFunction , $p ick four , 4 ) ") ;13 $pickone=range (0 , 3 ) ;14 $e=maple (" St r ingToo l s [ Randomize ] ( $seed+1) ;15 p r i n t f ( convert ( combinat [ randperm ] ( $unnamed) [ 1 ] , s t r i n g ) ) ") ;

A.4 Maple TA specification of answers

B The Temperature problem

B.1 Description of problemThis problem selects a computational task having to do with temperatures. A typical task might be “given thefollowing list of daily temperatures, count the number of days where the temperature was at least .5 degreesFahrenheit warmer than the previous day”. The temperature difference would be randomly generated, and

26

Figure 21: Maple TA answer layout for multiple choice question

whether or not the counting comparison should be “greater”, “less than”, “greater than or equal to”, etc. israndomly selected.

B.2 Maple TA script for temperature problem

1 $seed=r i n t (1 ,10000) ;2 $v=310;3 $M=60;4 $A=40;5 $P=365;6 $N=310;7 $p i cks=maple (" randomize ( $seed ) ; S t r ingToo l s [ Randomize ] ( $seed ) ; combinat [

randperm ] ( 4 ) ; " ) ;8 $d i f fTa rg e t=maple (" randomize ( $seed+1) ; RandomTools [ Generate ] ( f l o a t ( range

=1 . 0 . . 3 . 5 , d i g i t s =2) ) ") ;9 $pick1=switch (0 , $p i cks ) ;10 $TV=maple (" temperatures1 :−genProblem ($v , $seed ,$M, $A , $P , $N , $d i f fTa rg e t ) ,

l ibname=/mapleta/web/SanForCs002/Public_Html/ temperatures1 / temperatures1 .l i b ") ;

11 $count=maple (" temperatures1 :−countProc ( $d i f fTarge t , $pick1 ,$TV) , l ibname=/mapleta/web/SanForCs002/Public_Html/ temperatures1 / temperatures1 . l i b ") ;

12 $avgCount=maple (" temperatures1 :−avgProc ( $d i f fTarge t , $pick1 ,$TV) , l ibname=/mapleta/web/SanForCs002/Public_Html/ temperatures1 / temperatures1 . l i b ") ;

13 $pickmsg=maple (" p r i n t f ( temperatures1 :− p i ck I ( $d i f fTarge t , $pick1 ) [ 2 ] ) , l ibname=/mapleta/web/SanForCs002/Public_Html/ temperatures1 / temperatures1 . l i b ") ;

14 $pick2=switch (1 , $p i cks ) ; $d i f fTa rg e t 2=maple (" randomize ( $seed+2) ; RandomTools [Generate ] ( f l o a t ( range =1 . 0 . . 3 . 5 , d i g i t s =2) ) ") ; $count2=maple (" temperatures1 :−countProc ( $d i f fTarge t2 , $pick2 ,$TV) , l ibname=/mapleta/web/SanForCs002/Public_Html/ temperatures1 / temperatures1 . l i b ") ;

15 $avgCount2=maple (" temperatures1 :−avgProc ( $d i f fTarge t2 , $pick2 ,$TV) , l ibname=/mapleta/web/SanForCs002/Public_Html/ temperatures1 / temperatures1 . l i b ") ;

16 $pickmsg2=maple (" p r i n t f ( temperatures1 :− p i ck I ( $d i f fTarge t2 , $pick2 ) [ 2 ] ) , l ibname=/mapleta/web/SanForCs002/Public_Html/ temperatures1 / temperatures1 . l i b ") ;

27

B.3 Maple library module for temperature problem

1 temperatures1 := module ( )2 export genProblem , p ickI , countProc , avgProc ;34 genProblem:= proc (v , seed ,M,A,P,N, d i f fTa r g e t )5 l o c a l R, s , tempF , T, temps , LValues , TValues , vs , tv ;6 randomize ( seed ) ;7 R := S t a t i s t i c s [ RandomVariable ] ( Normal (0 , 5) ) ;8 vs := S t a t i s t i c s [ Sample ] (R,N) ;9 s := convert ( vs , l i s t ) :10 tempF:= ( t ) −> A∗ s i n ( t /P∗2∗Pi )+M;11 T :=( [ seq ( e v a l f ( tempF( i ) ) , i =1. .N) ] ) :12 p r i n t (" l ength o f T=",nops (T) ," l ength o f s=", nops ( s ) ) ;13 temps := s+T:14 LValues := so r t ( combinat [ randperm ] (N) [ 1 . . v ] ) :15 tv := x−>eva l f (x , 4 ) ;16 TValues := map( tv , [ seq ( temps [ LValues [ i ] ] , i =1. . v ) ] ) ;17 re turn TValues ;18 end ; #genProblem1920 #Pick the ques t i on g iven the index .21 p i ck I := proc ( d i f fTarge t , i )22 l o c a l p o s s i b i l i t i e s ;23 p o s s i b i l i t i e s := [ [ ’ ’ deltaT>=di f fTarge t ’ ’ , s p r i n t f (" the temperature i s at l e a s t

%.3 f degree s h igher than the prev ious day " , d i f fTa r g e t ) ] ,24 [ ’ ’− deltaT>=di f fTarge t ’ ’ , s p r i n t f (" the temperature i s at l e a s t %.3 f degree s

lower than the prev ious day " , d i f fTa r g e t ) ] ,25 [ ’ ’ abs ( deltaT )>=di f fTarge t ’ ’ , s p r i n t f (" the d i f f e r e n c e between the day and the

prev ious day ’ s temperatures i s at l e a s t %.3 f degree s in magnitude " ,d i f fTa r g e t ) ] ,

26 [ ’ ’ abs ( deltaT )<= di f fTarge t ’ ’ , s p r i n t f (" the d i f f e r e n c e between the day and theprev ious day ’ s temperatures i s at most %.3 f degree s in magnitude " ,d i f fTa r g e t ) ] ] ;

2728 re turn p o s s i b i l i t i e s [ i ] ;29 end ; #p i ck I3031 #Generate the procedure f o r count ing g iven the t a r g e t .32 countProc := proc ( d i f fTarge t , p icker , TValues )33 l o c a l count , n , pick , p , pmesg , i , deltaT , p o s s i b i l i t i e s ;34 count := 0 :35 p o s s i b i l i t i e s := [ [ ’ deltaT>=di f fTarge t ’ , s p r i n t f (" the temperature i s at l e a s t

%.3 f degree s h igher than the prev ious day " , d i f fTa r g e t ) ] ,36 [ ’− deltaT>=di f fTarge t ’ , s p r i n t f (" the temperature i s at l e a s t %.3 f degree s lower

than the prev ious day " , d i f fTa r g e t ) ] ,37 [ ’ abs ( deltaT )>=di f fTarge t ’ , s p r i n t f (" the d i f f e r e n c e between the day and the


38 [ ’ abs ( deltaT )<= di f fTarge t ’ , s p r i n t f (" the d i f f e r e n c e between the day and theprev ious day ’ s temperatures i s at most %.3 f degree s in magnitude " ,d i f fTa r g e t ) ] ] ;

3940 pick := p o s s i b i l i t i e s [ p i ck e r ] ;

28

41 p := pick [ 1 ] ;42 pmesg:=pick [ 2 ] ;43 n := nops ( TValues ) ;44 f o r i from 2 to n do45 deltaT := TValues [ i ]−TValues [ i −1] ;46 i f eva l (p)47 then count := count +1;48 # pr in t ( TValues [ i ] , TValues [ i −1]) ;49 end i f ;50 end do :51 re turn count ;52 end ; #countProc535455 #Generate the procedure f o r computing the average g iven the s e l e c t i o n .56 avgProc := proc ( d i f fTarge t , p icker , TValues )57 l o c a l count , n , pick , p , pmesg , i , deltaT , p o s s i b i l i t i e s ;58 count := 0 :59 p o s s i b i l i t i e s := [ [ ’ deltaT>=di f fTarge t ’ , s p r i n t f (" the temperature i s at l e a s t

%.3 f degree s h igher than the prev ious day " , d i f fTa r g e t ) ] ,60 [ ’− deltaT>=di f fTarge t ’ , s p r i n t f (" the temperature i s at l e a s t %.3 f degree s lower

than the prev ious day " , d i f fTa r g e t ) ] ,61 [ ’ abs ( deltaT )>=di f fTarge t ’ , s p r i n t f (" the d i f f e r e n c e between the day and the


62 [ ’ abs ( deltaT )<= di f fTarge t ’ , s p r i n t f (" the d i f f e r e n c e between the day and theprev ious day ’ s temperatures i s at most %.3 f degree s in magnitude " ,d i f fTa r g e t ) ] ] ;

6364 pick := p o s s i b i l i t i e s [ p i ck e r ] ;65 p := pick [ 1 ] ;66 pmesg:=pick [ 2 ] ;67 n := nops ( TValues ) ;68 to t := 0 . 0 ;69 f o r i from 2 to n do70 deltaT := TValues [ i ]−TValues [ i −1] ;71 i f eva l (p)72 then count := count +1;73 # pr in t ( TValues [ i ] , TValues [ i −1]) ;74 to t := tot+TValues [ i ] ;75 end i f ;76 end do :77 re turn e v a l f ( to t / count ) ;78 end ; #avgProc7980 end module :

C The Sensor problem

C.1 Description of problemIn an “Introduction to Engineering Design” class that most of the students are taking concurrently, thestudents were asked to place a distance-measuring infrared sensor at various distances from a target, and tocompile a table of the actual distance versus the sensor’s measured distance. In this problem the studentsare asked to develop a least-squares best-linear-fit formula for a dynamically generated measurement data

29

set, and to problem-solve using the resulting formula to make predictions and estimates.

C.2 Maple TA script for Sensor problemThe Maple TA scripting language does not easily allow program comments. We briefly describe the actionsin the script:

1. Define the random number seed and the numerical criterion defining how close the student’s numericalanswers have to be to the computed solutions before they are accepted as correct (lines 1-2)

2. Generate indices that will be used to generate the rest of the parameters of the problem (lines 3-10).

3. Generate the components for the first part of the problem using some of the indices of step2(lines 11-22). Some of these are the text of the particular version of the problem, some of them are parametricvalues used in the text. In this problem, there are four possible questions that could be asked. Two arepicked at random. Asking different questions of similar difficulty is another way to vary the problem.

1 $seed=rint (10 ,10000);2 $epsilon =.01;3 $problemSpec=maple(" Sensor2:-problemGen($seed), libname =/ mapleta/web/Cs12Wi20001/

Public_Html/Sensor2/Sensor2.lib");4 $problems=maple (" $problemSpec [1]");5 $parts=maple(" $problemSpec [2]");6 $part1=switch(0,$parts);7 $part2=switch(1,$parts);8 $prob1=switch(0, $problems);9 $prob2=switch(1, $problems);

10 $prob3=switch(2, $problems);11 $answer1=maple(" Sensor2:-problemGen2($problems), libname =/ mapleta/web/Cs12Wi20001/

Public_Html/Sensor2/Sensor2.lib");12 $proc2=maple("op(Sensor2:-problemGen3), libname =/ mapleta/web/Cs12Wi20001/

Public_Html/Sensor2/Sensor2.lib");13 $answerR=maple(" Sensor2:-genAnswer (3), libname =/ mapleta/web/Cs12Wi20001/

Public_Html/Sensor2/Sensor2.lib");14 $answerA=maple(" Sensor2:-InternalSeed2 , libname =/ mapleta/web/Cs12Wi20001/

Public_Html/Sensor2/Sensor2.lib");15 $answer2=maple(" Sensor2:-problemGen3($problems ,$answer1), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");16 $answer3=maple(" Sensor2:-problemGen4($problems ,$answer1), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");17 $intro=maple(" printf(Sensor2:-qsprintf[intro ]( $answer1)), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");18 $part11 = maple(" printf(Sensor2:-qsprintf[part1 ]( $answer1)), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");19 $answerPart11 = maple(" Sensor2:-qanswer[part1]( $answer1), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");20 $answerPart111 = maple (" $answerPart11 [1]");21 $answerPart112 = maple (" $answerPart11 [2]");22 $answerPart113 = maple (" $answerPart11 [3]");23 $part21 = maple(" printf(Sensor2:-qsprintf[part2 ]( $answer1)), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");24 $question01 = maple (" printf(Sensor2:-qsprintf [0]( $answer1)), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");25 $answer01 = maple ("( Sensor2:-qanswer [0]( $answer1)), libname =/ mapleta/web/

Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");26 $answer011 = switch(0, $answer01);27 $answer012 = switch(1, $answer01);28 $answer013 = maple(" $answer01 [3]");

30

29 $question11 = maple (" printf(Sensor2:-qsprintf[$part1 ]( $answer1)), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");

30 $answer11 = maple ("( Sensor2:-qanswer[$part1 ]( $answer1)), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");



33 $part22 = maple(" printf(Sensor2:-qsprintf[part3 ]( $answer2)), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");





38 $part23 = maple(" printf(Sensor2:-qsprintf[part4 ]( $answer3)), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");



41 $hint1 = maple(" printf(Sensor2:-hint[$part1 ]), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");

42 $hint2 = maple(" printf(Sensor2:-hint[$part2 ]), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");

43 $indVar=maple (" Sensor2:-getIndVar($answer1), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");

44 $depVar=maple (" Sensor2:-getDepVar($answer1), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");

45 $answerE=maple(" Sensor2:-problemGenExample($answer1), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");

46 $intro2=maple (" printf(Sensor2:-qsprintf[intro2 ]( $answerE)), libname =/ mapleta/web/Cs12Wi20001/Public_Html/Sensor2/Sensor2.lib");

47 $answerPart1E = maple(" $answerE [5]");48 $answer0E1 = maple(" $answerE [6]");49 $answer0E2 = maple(" $answerE [2]");50 $answer0E3 = maple(" $answerE [3]");51 $graphTitle = maple (" convert(sprintf(convert(‘$depVar =%a‘,string),$answer0E1 ,

$answer0E2 ,$indVar),name)");52 $myplot=plotmaple (" plots[display ]([ plot($answerPart1E ,style=POINT , symbol=cross ,

symbolsize =20,color=blue),plot($answer0E1 , $indVar =0..40 , color=red)],title=$graphTitle ,labels =[$indVar ,$depVar],scaling=constrained),plotdevice=’gif ’,plotoptions=’height =500, width =500 ’");

53 $withinPairList = maple("proc (l1, l2 , epsilon) local i;54 for i to nops(l1) do if max(op(map(abs , l1[i]-l2[i]))) < epsilon then NULL else

return false end if end do;55 return true end proc");

C.3 Maple program listing for Sensor2 module

1 #Module header −− l i s t names o f " pub l i c " procedures and data members2 #as export s34 Sensor2 := module ( )

31

Figure 22: Maple TA script source code for Sensor problem

1 $seed=maple (" randomize ( ) ") ;2 $ ep s i l on =.01;3 $problemSpec=maple (" Sensor2 :−problemGen ( $seed ) , l ibname=/mapleta/web/

Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;4 $problems=maple (" $problemSpec [ 1 ] " ) ;5 $part s=maple (" $problemSpec [ 2 ] " ) ;6 $part1=switch (0 , $part s ) ;7 $part2=switch (1 , $part s ) ;8 $prob1=switch (0 , $problems ) ;9 $prob2=switch (1 , $problems ) ;10 $prob3=switch (2 , $problems ) ;11 $answer1=maple (" Sensor2 :−problemGen2 ( $problems ) , l ibname=/mapleta/web/

Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;12 $proc2=maple (" op ( Sensor2 :−problemGen3 ) , l ibname=/mapleta/web/Cs12Wi20001/

Public_Html/ Sensor2 / Sensor2 . l i b ") ;13 $answerR=maple (" Sensor2 :−genAnswer (3 ) , l ibname=/mapleta/web/Cs12Wi20001/

Public_Html/ Sensor2 / Sensor2 . l i b ") ;14 $answerA=maple (" Sensor2 :− Interna lSeed2 , l ibname=/mapleta/web/Cs12Wi20001/

Public_Html/ Sensor2 / Sensor2 . l i b ") ;15 $answer2=maple (" Sensor2 :−problemGen3 ( $problems , $answer1 ) , l ibname=/mapleta/web

/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;16 $answer3=maple (" Sensor2 :−problemGen4 ( $problems , $answer1 ) , l ibname=/mapleta/web

/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;17 $ in t r o=maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ i n t r o ] ( $answer1 ) ) , l ibname=/mapleta/web

/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;18 $part11 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ part1 ] ( $answer1 ) ) , l ibname=/mapleta/

web/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;19 $answerPart11 = maple (" Sensor2 :−qanswer [ part1 ] ( $answer1 ) , l ibname=/mapleta/web

/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;20 $answerPart111 = maple (" $answerPart11 [ 1 ] " ) ;21 $answerPart112 = maple (" $answerPart11 [ 2 ] " ) ;22 $answerPart113 = maple (" $answerPart11 [ 3 ] " ) ;23 $part21 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ part2 ] ( $answer1 ) ) , l ibname=/mapleta/

web/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;24 $quest ion01 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ 0 ] ( $answer1 ) ) , l ibname=/mapleta/

web/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;25 $answer01 = maple (" ( Sensor2 :−qanswer [ 0 ] ( $answer1 ) ) , l ibname=/mapleta/web/

Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;26 $answer011 = switch (0 , $answer01 ) ;27 $answer012 = switch (1 , $answer01 ) ;28 $answer013 = maple (" $answer01 [ 3 ] " ) ;29 $quest ion11 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ $part1 ] ( $answer1 ) ) , l ibname=/

mapleta/web/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;30 $answer11 = maple (" ( Sensor2 :−qanswer [ $part1 ] ( $answer1 ) ) , l ibname=/mapleta/web

/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;31 $quest ion12 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ $part2 ] ( $answer1 ) ) , l ibname=/



web/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;34 $quest ion21 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ $part1 ] ( $answer2 ) ) , l ibname=/


/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;36 $quest ion22 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ $part2 ] ( $answer2 ) ) , l ibname=/



web/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;39 $quest ion32 = maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ $part2 ] ( $answer3 ) ) , l ibname=/


/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;41 $hint1 = maple (" p r i n t f ( Sensor2 :−hint [ $part1 ] ) , l ibname=/mapleta/web/

Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;42 $hint2 = maple (" p r i n t f ( Sensor2 :−hint [ $part2 ] ) , l ibname=/mapleta/web/

Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;43 $indVar=maple (" Sensor2 :−getIndVar ( $answer1 ) , l ibname=/mapleta/web/Cs12Wi20001/

Public_Html/ Sensor2 / Sensor2 . l i b ") ;44 $depVar=maple (" Sensor2 :−getDepVar ( $answer1 ) , l ibname=/mapleta/web/Cs12Wi20001/

Public_Html/ Sensor2 / Sensor2 . l i b ") ;45 $answerE=maple (" Sensor2 :−problemGenExample ( $answer1 ) , l ibname=/mapleta/web/

Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;46 $ in t ro2=maple (" p r i n t f ( Sensor2 :− q s p r i n t f [ i n t r o2 ] ( $answerE ) ) , l ibname=/mapleta/

web/Cs12Wi20001/Public_Html/ Sensor2 / Sensor2 . l i b ") ;47 $answerPart1E = maple (" $answerE [ 5 ] " ) ;48 $answer0E1 = maple (" $answerE [ 6 ] " ) ;49 $answer0E2 = maple (" $answerE [ 2 ] " ) ;50 $answer0E3 = maple (" $answerE [ 3 ] " ) ;51 $graphTit l e = maple (" convert ( s p r i n t f ( convert ( ‘ $depVar=%a ‘ , s t r i n g ) , $answer0E1 ,

$answer0E2 , $indVar ) ,name) ") ;52 $myplot=plotmaple (" p l o t s [ d i sp l ay ] ( [ p l o t ( $answerPart1E , s t y l e=POINT, symbol=

cros s , symbol s i ze =20, c o l o r=blue ) , p l o t ( $answer0E1 , $indVar =0. .40 , c o l o r=red )] , t i t l e=$graphTit le , l a b e l s =[$indVar , $depVar ] , s c a l i n g=cons t ra ined ) ,p l o td ev i c e=’ g i f ’ , p l o t op t i on s=’ he ight =500 ,width=500 ’") ;

53 $w i th inPa i rL i s t = maple (" proc ( l1 , l2 , e p s i l o n ) l o c a l i ;54 f o r i to nops ( l 1 ) do i f max( op (map( abs , l 1 [ i ]− l 2 [ i ] ) ) ) < ep s i l o n then NULL

e l s e re turn f a l s e end i f end do ;55 re turn true end proc ") ;

32

5 export problemGen , problemGen0 , problemGen2 , problemGen3 , problemGen4 ,6 problemGen5 , answerGen , quest ion , hint , q sp r i n t f ,7 qanswer , subsnames , numProbs , numParts , subsNames ,8 getIndVar , getDepVar , exampleProbIndex , problemGenExample ,

genAnswer , Interna lSeed , In t e rna lSeed2 ;910 #Generates a problem f o r p o s i t i v e data po in t s to do s p l i n e curve f i t t i n g .11 numProbs := 2000 ;12 exampleProbIndex := numProbs+1;13 numParts := 4 ;14 #Change t h i s va lue to have a d i f f e r e n t c o l l e c t i o n o f numProbs problems .15 In t e rna lSeed := 47 ;16 Inte rna lSeed2 := 4747 ;1718 problemGen := proc ( seed )19 l o c a l problems , parts , i n t e rna lS e ed ;20 in t e rna lS e ed := randomize ( seed ) ;21 St r ingToo l s [ Randomize ] ( seed ) ;22 #Pick 3 problems23 problems := combinat [ randperm ] ( numProbs−1) [ 1 . . 3 ] ;24 #Pick 2 out o f 4 par t s25 par t s := combinat [ randperm ] ( numParts ) [ 1 . . 2 ] ;26 re turn [ problems , par t s ] ;27 end proc ;2829 #Return the 1 s t problem30 problemGen2 := proc ( problems )31 re turn Sensor2 :−genAnswer ( problems [ 1 ] ) ;32 end proc ;3334 #Return the 0 th problem ( example )35 problemGen0 := proc ( answer1 )36 re turn Sensor2 :−qanswer [ i n t r o2 ] ( subsNames ( Sensor2 :−genAnswer ( Sensor2 :−

numProbs ) , getIndVar ( answer1 ) ,37 getDepVar ( answer1 ) ) ) ;38 end proc ;3940 #Return the 2nd problem41 problemGen3 := proc ( problems , answer1 )42 re turn subsNames ( Sensor2 :−genAnswer ( problems [ 2 ] ) , getIndVar ( answer1 ) ,43 getDepVar ( answer1 ) ) ;44 end proc ;4546 #Return the 3 rd problem47 problemGen4 := proc ( problems , answer1 )48 re turn subsNames ( Sensor2 :−genAnswer ( problems [ 3 ] ) , getIndVar ( answer1 ) ,49 getDepVar ( answer1 ) ) ;50 end proc ;5152 #Return the example problem53 problemGenExample := proc ( answer1 )54 # return Sensor2 :−qanswer [ i n t r o2 ] ( subsNames ( Sensor2 :−genAnswer ( Sensor2 :−

exampleProbIndex ) , getIndVar ( answer1 ) ,55 # getDepVar ( answer1 ) ) ) ;

33

56 return ( subsNames ( Sensor2 :−genAnswer ( Sensor2 :−exampleProbIndex ) ,getIndVar ( answer1 ) ,

57 getDepVar ( answer1 ) ) ) ;58 end proc ;5960 ques t i on [ i n t r o ] := "You are measuring the p r op e r t i e s o f a c o l l e c t i o n o f

d i s t anc e s en so r s . Because o f impe r f e c t i on s and l im i t a t i o n s in thetechnology , the d i s t anc e a senso r r epo r t s may d i f f e r s i g n i f i c a n t l y fromthe ac tua l d i s t ance . You have performed exper iments on each senso r in thec o l l e c t i o n . For each sensor , you have taken a s e r i e s o f r ead ings

where you measure a d i s t ance from a target , p l ace the senso r there , andrecord what i t s read ing i s . You have recorded the r e s u l t s as a l i s t o fordered pa i r s ( a l i s t o f l i s t s ) . The f i r s t number in each pa i r i s theac tua l d i s t anc e %a , the second number i s the sensor ’ s read ing o f thed i s t anc e %a.For each s e r i e s o f read ings , you wish to compute withMaple the \" trend l i n e \" : the formula o f the gene ra l form %s where c1 andc2 are c a l c u l a t ed to be the l e a s t squares f i t o f the data . You w i l l use

the formula to he lp you understand some p r op e r t i e s o f the s enso r . " ;61 q s p r i n t f [ i n t r o ] := proc ( answer )62 l o c a l63 $ inc lude " answerLocals . mpl"64 ;65 $ inc lude "answerAssignments . mpl"66 return s p r i n t f ( ques t i on [ i n t r o ] , indvar , depvar ,MathML[

ExportPresentat ion ] ( c1 + c2∗ indvar ) ) ;67 end proc ;6869 q s p r i n t f [ i n t r o2 ] := proc ( answer )70 l o c a l71 $ inc lude " answerLocals . mpl"72 ;73 $ inc lude "answerAssignments . mpl"74 re turn s p r i n t f (" For example , f o r exper imenta l measurements %a , we

would have the f o l l ow i n g r e s u l t f o r the trend l i n e : " , r e s u l t s ) ;75 end proc ;7677 qanswer [ i n t r o2 ] := proc ( answer )78 l o c a l79 $ inc lude " answerLocals . mpl"80 ;81 $ inc lude "answerAssignments . mpl"82 re turn83 p l o t s [ d i sp l ay ] ( [ p l o t ( r e s u l t s , s t y l e=point , c o l o r="blue " , symbol s i ze

=10, symbol=c r o s s ) ,84 p l o t ( formula , indvar=loVal . . hiVal , c o l o r="red ") ] ,85 t i t l e=typese t ( eqn ) , l a b e l s =[ indvar , depvar ] ) ;86 end proc ;8788 ques t i on [ part1 ] := "F i l l in the blanks to c r e a t e a Maple exp r e s s i on that

c r e a t e s the l e a s t squares f i t t e d trend l i n e f o r the data with independentva r i ab l e %a.";

89 #CurveFitt ing [W] ( X, Z) ;90 q s p r i n t f [ part1 ] := proc ( answer )91 l o c a l

34

92 $ inc lude " answerLocals . mpl"93 ;94 $ inc lude "answerAssignments . mpl"95 return s p r i n t f ( ques t i on [ part1 ] , indvar ) ;96 end proc ;97 qanswer [ part1 ] := proc ( answer )98 l o c a l99 $ inc lude " answerLocals . mpl"100 ;101 $ inc lude "answerAssignments . mpl"102 return [ ’ LeastSquares ’ , r e s u l t s , indvar ] ;103 end proc ;104105 ques t i on [ part2 ] := "The f i r s t s enso r in your c o l l e c t i o n had these r ead ings

: %a . ";106 q s p r i n t f [ part2 ] := proc ( answer )107 l o c a l108 $ inc lude " answerLocals . mpl"109 ;110 $ inc lude "answerAssignments . mpl"111 return s p r i n t f ( ques t i on [ part2 ] , r e s u l t s ) ;112 end proc ;113 #There i s no answer to t h i s s e c t i o n .114115116 ques t i on [ part3 ] := "The second senso r in your c o l l e c t i o n had these r ead ings

: %a . ";117 q s p r i n t f [ part3 ] := proc ( answer )118 l o c a l119 $ inc lude " answerLocals . mpl"120 ;121 $ inc lude "answerAssignments . mpl"122 return s p r i n t f ( ques t i on [ part3 ] , r e s u l t s ) ;123 end proc ;124 #There i s no answer to t h i s s e c t i o n .125126 ques t i on [ part4 ] := "The th i rd senso r in your c o l l e c t i o n had these r ead ings :

 %a . ";127 q s p r i n t f [ part4 ] := proc ( answer )128 l o c a l129 $ inc lude " answerLocals . mpl"130 ;131 $ inc lude "answerAssignments . mpl"132 return s p r i n t f ( ques t i on [ part4 ] , r e s u l t s ) ;133 end proc ;134 #There i s no answer to t h i s s e c t i o n .135136137 #0 What are the c o e f f i c i e n t s138 ques t i on [ 0 ] := " F i l l in the blanks : the trend l i n e formula i s : ";139 q s p r i n t f [ 0 ] := proc ( answer )140 l o c a l141 $ inc lude " answerLocals . mpl"142 ;

35

143 $ inc lude "answerAssignments . mpl"144 return s p r i n t f ( ques t i on [ 0 ] ) ;145 end proc ;146147 #answer r e qu i r e s c1 , c2 , indvar , depvar148 qanswer [ 0 ] := proc ( answer )149 l o c a l150 $ inc lude " answerLocals . mpl"151 ;152 $ inc lude "answerAssignments . mpl"153 return [ c o e f f ( l s qResu l t s , indvar , 0 ) , c o e f f ( l s qResu l t s , indvar , 1 ) , indvar

, depvar ] ;154 end proc ;155156157 #1 Compute i n t e r s e c t i o n po int158 ques t i on [ 1 ] := "At what d i s t anc e w i l l the measured d i s t ance agree with the

ac tua l d i s t anc e ?" ;159 h int [ 1 ] := "You can s e t up an equat ion based in part on the formula f o r the

f i t t e d l i n e . So lv ing t h i s w i l l f i nd where the measured d i s t anc e agr ee swith the ac tua l d i s t ance . " ;

160 q s p r i n t f [ 1 ] := proc ( answer )161 l o c a l162 $ inc lude " answerLocals . mpl"163 ;164 $ inc lude "answerAssignments . mpl"165 return s p r i n t f ( ques t i on [ 1 ] ) ;166 end proc ;167 qanswer [ 1 ] := proc ( answer )168 l o c a l169 $ inc lude " answerLocals . mpl"170 ;171 $ inc lude "answerAssignments . mpl"172 return eva l ( indvar , op ( i s e c t ) ) ;173 end proc ;174175176 #2 Evaluat ion177 ques t i on [ 2 ] := "What d i s t ance does the senso r read when i t i s p laced %6.4 f

c en t imete r s from the t a r g e t ?" ;178 h int [ 2 ] := "Once you ’ ve computed the formula f o r the f i t t e d l i n e , you can use

the eva l func t i on to eva luate i t f o r any value o f the va r i ab l e . " ;179 q s p r i n t f [ 2 ] := proc ( answer )180 l o c a l181 $ inc lude " answerLocals . mpl"182 ;183 $ inc lude "answerAssignments . mpl"184 return s p r i n t f ( ques t i on [ 2 ] , dataPt1 ) ;185 end proc ;186 #answer f o r 2nd kind o f ques t i on187 qanswer [ 2 ] := proc ( answer )188 l o c a l189 $ inc lude " answerLocals . mpl"190 ;

36

191 $ inc lude "answerAssignments . mpl"192 return dataResult1 ;193 end proc ;194195 #3 Inve r s e s o l u t i o n196 ques t i on [ 3 ] := "How f a r i s the s enso r from the t a r g e t when i t g i v e s a read ing

o f %6.4 f c ent imete r s ?" ;197 h int [ 3 ] := "To determine how f a r the senso r r e a l l y i s when i t g i v e s a

s p e c i f i e d reading , s e t up an equat ion d i s t ance = ( formula f o r l i n e ) andso l v e f o r the va r i ab l e . " ;

198 q s p r i n t f [ 3 ] := proc ( answer )199 l o c a l200 $ inc lude " answerLocals . mpl"201 ;202 $ inc lude "answerAssignments . mpl"203 return s p r i n t f ( ques t i on [ 3 ] , MeasuredValue ) ;204 end proc ;205 qanswer [ 3 ] := proc ( answer )206 l o c a l207 $ inc lude " answerLocals . mpl"208 ;209 $ inc lude "answerAssignments . mpl"210 return ActualValue ;211 end proc ;212213 #4 Maximal dev i a t i on214 ques t i on [ 4 ] := "What i s the l a r g e s t magnitude d i f f e r e n c e between the measured

d i s t anc e as g iven by the f i t t e d l i n e , and the ac tua l d i s t anc e in the range%6.4 f to %6.4 f c en t imete r s ? Enter a non−negat ive f l o a t i n g po int

number . " ;215 h int [ 4 ] := "To f i nd the l a r g e s t magnitude d i f f e r e n c e , f i nd the maximum of the

the abso lu t e value o f the d i f f e r e n c e between the r e s u l t o f eva lua t ing thef i t t e d l i n e formula and the ac tua l d i s tance , in the s p e c i f i e d range . Ap lo t or ca l cu lu s−based mathematical th ink ing w i l l t e l l you where t h i sl i n e a r exp r e s s i on could po s s i b l y reach a maximum on the i n t e r v a l . " ;

216 q s p r i n t f [ 4 ] := proc ( answer )217 l o c a l218 $ inc lude " answerLocals . mpl"219 ;220 $ inc lude "answerAssignments . mpl"221 return s p r i n t f ( ques t i on [ 4 ] , loVal , hiVal ) ;222 end proc ;223 qanswer [ 4 ] := proc ( answer )224 l o c a l225 $ inc lude " answerLocals . mpl"226 ;227 $ inc lude "answerAssignments . mpl"228 return maxDev ;229 end proc ;230231232 subsNames := proc ( answer , indvar1 , depvar1 )233 l o c a l234 $ inc lude " answerLocals . mpl"

37

235 ;236 $ inc lude "answerAssignments . mpl"237 re turn subs ( [ indvar=indvar1 , depvar=depvar1 ] , answer ) ;238 end proc ;239240 getIndVar := proc ( answer )241 l o c a l242 $ inc lude " answerLocals . mpl"243 ;244 $ inc lude "answerAssignments . mpl"245 re turn indvar ;246 end proc ;247248 getDepVar := proc ( answer )249 l o c a l250 $ inc lude " answerLocals . mpl"251 ;252 $ inc lude "answerAssignments . mpl"253 re turn depvar ;254 end proc ;255256 #genAnswer gene ra t e s the problem data from the part number .257 genAnswer := proc (probNum)258259 l o c a l answer , a , b , v a r l i s t , vars , indvar , depvar , c1 , c2 , formula , eqn ,260 numPts , dataPts , r e s u l t s , l s qResu l t s , actua l , maxRange ,261 formatTo3Digits , formatter , i s e c t , dataPt1 , dataPt2 , dataResult1 ,262 dataResult2 , loVal , hiVal , numProbs , i , maxVal , MeasuredValue ,263 ActualValue , maxDev , t1 , t2 , t3264 , seed ;265266 #Set the random number generator seed based on the pre−de f ined s t a r t i n g267 #seed p lus the part Number . This g i v e s a seed in the range268 #Inte rna lSeed +1. . In t e rna lSeed+probNum ;269 seed := Sensor2 :− In te rna lSeed2 +probNum ;270 randomize ( seed ) ; #Set random number genera tor to t h i s va lue .271 St r ingToo l s [ Randomize ] ( seed ) ;272 #Thus the seed does not depend on time o f day , e t c .273274 v a r l i s t := [ [ x , y ] , [ d ,m] , [ a , b ] , [ d1 , d2 ] ] ;275 #i f we are very unlucky and the randomized c o e f f i c i e n t s are the same276 #as the exact answer , then c r e a t e i n t e n t i o n a l l y d i f f e r e n t c o e f f i c i e n t s .277278279280 formatTo3Digits := x −> ss can f ( s p r i n t f ("%.3 f " , x ) ,"% f ") [ 1 ] ;281 formatte r := xy −> map( formatTo3Digits , xy ) ;282283 #pick v a r i a b l e s284285 vars := combinat [ randperm ] ( v a r l i s t ) [ 1 ] ;286 indvar , depvar := vars [ 1 ] , vars [ 2 ] ;287288 #generate data l i s t f o r f i t .

38

289 maxRange := 40 ;290 loVal := rand ( 4 . . 6 ) ( ) ;291 hiVal := rand (maxRange−4. .maxRange) ( ) ;292 numPts := rand ( 5 . . 8 ) ( ) ;293294 #Generate a random l i n e295 c1 := RandomTools [ Generate ] ( f l o a t ( range =2 . 0 . . 8 . 0 ) ) ;296 c2 := RandomTools [ Generate ] ( f l o a t ( range =0 . 5 . . 0 . 9 ) ) ;297 formula := c1 + c2∗ indvar ;298 eqn := depvar = formula ;299 ac tua l := depvar = indvar ;300301 #Generate perturbed data po in t s to go with the l i n e .302 #maxVal := RandomTools [ Generate ] ( f l o a t ( range=c1 / 7 . . c1 /2) ) ;303 maxVal := c1 /2 ;304 dataPts := so r t ( combinat [ randperm ] ( [ seq ( i , i =1. .maxRange) ] ) [ 1 . . numPts ] ) ;305 r e s u l t s := map( va l −> [ val , eva l ( formula , indvar=va l )+RandomTools [ Generate ] (

f l o a t ( range=maxVal / 1 0 . . maxVal ) )−maxVal /2 ] , dataPts ) ;306 r e s u l t s := map( formatter , r e s u l t s ) ;307 #part A Compute l e a s t squares l i n e308 l s qRe su l t s := CurveFitt ing [ LeastSquares ] ( r e s u l t s , indvar ) ;309310 #Generate a dummy r e s u l t i f the l i n e i s ( a c c i d e n t a l l y )311 #the same as the exact r e s u l t312 i f c o e f f ( l s qResu l t s , indvar , 0 )=0 and c o e f f ( l s qResu l t s , indvar , 1 )=1313 then l s qRe su l t s := 3 + 1.047∗ indvar ;314 end i f ;315316 #1317 #Compute i n t e r s e c t i o n po int between est imated and i d e a l l i n e s .318 i s e c t := so l v e ({ l s qRe su l t s=depvar , indvar=depvar } , [ indvar , depvar ] ) ;319320 #2321 #Pick two data po in t s to eva luate the l i n e at .322 dataPt1 , dataPt2 := op ( combinat [ randperm ] ( dataPts ) [ 1 . . 2 ] ) ;323 dataResult1 := eva l ( l s qResu l t s , indvar=dataPt1 ) ;324 dataResult2 := eva l ( l s qResu l t s , indvar=dataPt2 ) ;325326 #3327 #Pick a measurement value , f i nd ac tua l va lue where i t occurs328329 MeasuredValue := formatTo3Digits ( dataResult2 ) ;330331 #Find value o f independent va r i ab l e where that measurement i s332 ActualValue := so l v e (MeasuredValue=l sqResu l t s , indvar ) ;333334 #4335 #Find maximal dev i a t i on336 maxDev := maximize ( abs ( l sqResu l t s−indvar ) , indvar=loVal . . h iVal ) ;337 #Check that t h i s occurs at the extrema .338 t1 := eva l ( abs ( l sqResu l t s−indvar ) , indvar=loVal ) ;339 t2 := eva l ( abs ( l sqResu l t s−indvar ) , indvar=hiVal ) ;340 t3 := i s ( abs (maxDev−t1 )<10e−6 or abs (maxDev−t2 )<10e−6) ;341

39

342 answer := [343 $ inc lude " answerStored . mpl"344 ] ;345346 re turn answer ;347348 end proc :349350351352 end module ;

C.4 Maple program listing for Sensor2 test

12 #3 libname := " . " , l ibname ;45 #t rac e ( Sensor2 :−problemGen ) ;6 r e s u l t s := Sensor2 :−problemGen (47) ;7 problems := r e s u l t s [ 1 ] ;8 par t s := r e s u l t s [ 2 ] ;9 p r i n t ( Sensor2 :− ques t i on ) ;10 p r i n t ( Sensor2 :−hint ) ;11 answer1 := ( Sensor2 :−problemGen2 ( problems ) ) ;12 p r i n t ( answer1 ) ;13 p2A := Sensor2 :−problemGen3 ( problems , answer1 ) ;14 St r ingToo l s [ Randomize ] ( 4 7 ) ;15 p2B := ( Sensor2 :−problemGen3 ( problems , answer1 ) ) ;16 i s (p2A=p2B) ; #Should be same problem .17 p r i n t ( Sensor2 :−problemGen4 ( problems , answer1 ) ) ;18 p r i n t ( Sensor2 :−problemGenExample ( answer1 ) ) ;19 p r i n t ( Sensor2 :− q s p r i n t f [ i n t r o ] ( answer1 ) ) ;20 p r i n t ( Sensor2 :− q s p r i n t f [ i n t r o2 ] ( answer (1 ) ) ) ;21 p r i n t ( Sensor2 :− q s p r i n t f [ part1 ] ( answer1 ) ) ;22 p r i n t ( Sensor2 :− q s p r i n t f [ part2 ] ( answer1 ) ) ;23 p r i n t ( Sensor2 :− q s p r i n t f [ part3 ] ( answer1 ) ) ;24 p r i n t ( Sensor2 :− q s p r i n t f [ part4 ] ( answer1 ) ) ;25 p r i n t ( Sensor2 :− q s p r i n t f [ 0 ] ( answer1 ) ) ;26 p r i n t ( Sensor2 :− q s p r i n t f [ 1 ] ( answer1 ) ) ;27 p r i n t ( Sensor2 :− q s p r i n t f [ 2 ] ( answer1 ) ) ;28 p r i n t ( Sensor2 :− q s p r i n t f [ 3 ] ( answer1 ) ) ;29 p r i n t ( Sensor2 :− q s p r i n t f [ 4 ] ( answer1 ) ) ;3031 p r i n t ( Sensor2 :−qanswer [ i n t r o ] ( answer1 ) ) ;32 p r i n t ( Sensor2 :−qanswer [ i n t r o2 ] ( answer1 ) ) ;33 p r i n t ( Sensor2 :−qanswer [ part1 ] ( answer1 ) ) ;34 p r i n t ( Sensor2 :−qanswer [ 0 ] ( answer1 ) ) ;35 p r i n t ( Sensor2 :−qanswer [ 1 ] ( answer1 ) ) ;36 p r i n t ( Sensor2 :−qanswer [ 2 ] ( answer1 ) ) ;37 p r i n t ( Sensor2 :−qanswer [ 3 ] ( answer1 ) ) ;38 p r i n t ( Sensor2 :−qanswer [ 4 ] ( answer1 ) ) ;3940 p r i n t ( Sensor2 :−getIndVar ( answer1 ) ) ;

40

Figure 23: Initial portion of question for Spline 3 problem

41 pr in t ( Sensor2 :−getDepVar ( answer1 ) ) ;

D Spline3 problem

D.1 Description of problemAs illustrated in Figure 23 , this problem asks the student to fit a spline to data that is dynamically generated.The degree of the spline (quadratic or cubic) is dynamically selected as is the context of the problem (fluidflow, electricity, etc.) . The presentation is also dynamically generated.

D.2 Maple TA script for Spline 3 problem

1 $seed = r i n t (1 ,100000) ;2 $ques t ion=maple (" p r i n t f ( Sp l ine2 :−problemGen3 ( $seed ) ) , l ibname=/mapleta/web/

SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;3 $answer=maple (" Sp l ine2 :−answerGen ( $seed ) , l ibname=/mapleta/web/SanForCs000/

Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;4 $answerAl l=maple (" Sp l ine2 :−answerGen2 ( $seed ) , l ibname=/mapleta/web/SanForCs000

/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;5 $probs1=maple (" Sp l ine2 :−problemGen ( $seed ) , l ibname=/mapleta/web/SanForCs000/

Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;

41

6 $probs2=maple (" Sp l ine2 :−problemGen2 ( $seed ) , l ibname=/mapleta/web/SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;

7 $quest ionb1=maple (" p r i n t f ( Sp l ine2 :−problemGen4 ( $seed , $answerAll , 1 ) ) , l ibname=/mapleta/web/SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;

8 $dataHTML1=maple (" p r i n t f ( Sp l ine2 :−problemGen5 ( $seed , 1 ) ) , l ibname=/mapleta/web/SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;

9 $ x l i s t 1=maple (" $answerAll [ 1 ] [ 1 ] " ) ;10 $ y l i s t 1=maple (" $answerAll [ 1 ] [ 2 ] " ) ;11 $ r e s u l t 1=maple (" $answerAll [ 1 ] [ 4 ] " ) ;12 $unitAnswer=maple ( " [ $answer ] [ 4 ] " ) ;13 $pickAPoint=maple ( " [ $answer ] [ 1 ] " ) ;14 $pickAVal=maple ( " [ $answer ] [ 2 ] " ) ;15 $pickTime=maple ( " [ $probs2 ] [ 2 ] " ) ;16 $pickTimes=maple (" p r i n t f ( convert ( ‘% s%s ‘ , s t r i n g ) , convert ( $pickTime , s t r i n g ) ,

convert ( ‘ s ‘ , s t r i n g ) ) ") ;17 $dataHTML2=maple (" p r i n t f ( Sp l ine2 :−problemGen5 ( $seed , 2 ) ) , l ibname=/mapleta/web/

SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;18 $ x l i s t 2=maple (" $answerAll [ 2 ] [ 1 ] " ) ;19 $ y l i s t 2=maple (" $answerAll [ 2 ] [ 2 ] " ) ;20 $time2=maple (" $answerAl l [ 2 ] [ 3 ] " ) ;21 $ r e s u l t 2=maple (" $answerAll [ 2 ] [ 4 ] " ) ;22 $quest ionb2=maple (" p r i n t f ( Sp l ine2 :−problemGen4 ( $seed , $answerAll , 2 ) ) , l ibname=/

mapleta/web/SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;23 $dataHTML3=maple (" p r i n t f ( Sp l ine2 :−problemGen5 ( $seed , 3 ) ) , l ibname=/mapleta/web/

SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;24 $quest ionb3=maple (" p r i n t f ( Sp l ine2 :−problemGen4 ( $seed , $answerAll , 3 ) ) , l ibname=/

mapleta/web/SanForCs000/Public_Html/ Sp l ine2 / Sp l ine2 . l i b ") ;25 $ x l i s t 3=maple (" $answerAll [ 3 ] [ 1 ] " ) ;26 $ y l i s t 3=maple (" $answerAll [ 3 ] [ 2 ] " ) ;27 $time3=maple (" $answerAl l [ 3 ] [ 3 ] " ) ;28 $ r e s u l t 3=maple (" $answerAl l [ 3 ] [ 4 ] " ) ;29 $answerProc=maple (" $answerAl l [ 7 ] " ) ;30 $ ep s i l on =.01;31 $tr1 = maple (" procTest3 :− t e s t 1 ( $answerProc , $ r e su l t 1 , $answerAl l [ 1 ] ,

$ ep s i l on ) , l ibname = /mapleta/web/SanForCs000/Public_Html/procTest3/ procTest3 . l i b ") ;

32 $tr2 = maple (" procTest3 :− t e s t 1 ( $answerProc , $ r e su l t 2 , $answerAl l [ 2 ] ,$ ep s i l on ) , l ibname = /mapleta/web/SanForCs000/Public_Html/procTest3/ procTest3 . l i b ") ;

33 $tr3 = maple (" procTest3 :− t e s t 1 ( $answerProc , $ r e su l t 3 , $answerAl l [ 3 ] ,$ ep s i l on ) , l ibname = /mapleta/web/SanForCs000/Public_Html/procTest3/ procTest3 . l i b ") ;

34 $ t e s tA l l=maple (" proc ( answerProc , answerAll , r e su l t 1 , r e su l t 2 , r e su l t 3 , e p s i l o n) i s ( ( procTest3 :− t e s t 1 ( answerProc , r e su l t 1 , answerAl l [ 1 ] , e p s i l o n )=true )and ( procTest3 :− t e s t 1 ( answerProc , r e su l t 2 , answerAl l [ 2 ] , e p s i l o n )=true )and ( procTest3 :− t e s t 1 ( answerProc , r e su l t 3 , answerAl l [ 3 ] , e p s i l o n )=true ) ) ;

35 end proc ") ;36 $unitS=maple ( " [ $probs2 ] [ 3 ] " ) ;37 $unitT1=maple ( " [ $probs2 ] [ 2 ] " ) ;38 $unitT2=maple ( " [ $probs2 ] [ 4 ] " ) ;

D.3 Maple code for Spline 3 problemWe broke the Maple computation into pieces in order to make the results returned more easily comprehensible.However, because of the difficulty (both in terms of programmer effort and the run-time expense of pushing

42

a lengthy answer across the Maple TA/Maple interface several times), we regenerated the problem resultsin each phase. We placed the problem generation code in an include file that inserted into each procedurethat needed generated

D.3.1 Spline 3 code

1 Sp l ine3 := module ( )2 export problemGen , problemGen2 , problemGen3 , problemGen4 ,3 problemGen5 , answerGen ;45 #Generates a problem f o r p o s i t i v e data po in t s to do s p l i n e curve f i t t i n g .6 problemGen := proc ( seed )7 $ inc lude " Sp l i n e3 Inc lude . mpl"8 # l o c a l n , numIntervals , x l i s t , y l i s t ;9 # randomize ( seed ) ;10 # n := rand ( 1 . . 8 ) ( ) ;11 # Pick number o f data i n t e r v a l s12 # numIntervals := rand ( 5 . . 9 ) ( ) ;13 # z1 := e v a l f ( Bes se lJZero s (n , 1) ) ;14 # h := 1/n∗ z1 ;15 # x l i s t := [ seq ( ’ RandomTools [ Generate ] ( f l o a t ( range = h∗ i . . h∗( i +1)−0.1e

−1, d i g i t s = 3) ) ’ , i = 0 . . (n−1) ) ] ;16 # y l i s t := map( ( x ) −>Besse lJ (n , x ) , x l i s t ) ;17 re turn n , numIntervals , z1 , x l i s t , y l i s t ;18 end proc ;1920 #gene ra t e s the part o f the ques t i on that asks the student to s o l v e a word21 #problem invo l v i ng a s i t ua t i on , and var i ous t imes and un i t s .22 problemGen2 := proc ( seed )23 $ inc lude " Sp l i n e3 Inc lude . mpl"24 re turn questionType , pickTime , picksUnit , pickdUnit ;25 end proc ;2627 #Generates the ques t i on28 problemGen3 := proc ( seed )29 $ inc lude " Sp l i n e3 Inc lude . mpl"30 ques t i on := framework [ questionType , 1 ] ;31 ques t i on := s p r i n t f ( quest ion , picksUnit , pickTime , pickdUnit ,

sp l ineDesc ) ;32 re turn ques t i on ;33 end proc ;3435 #Generates the part o f the ques t i on that asks the student to i n t e g r a t e the36 #curve and to format the answer in a p a r t i c u l a r kind o f un i t .37 problemGen4 := proc ( seed )38 $ inc lude " Sp l i n e3 Inc lude . mpl"39 ques t ion2 := framework [ questionType , 2 ] ;40 ques t ion2 := s p r i n t f ( quest ion2 , pickdUnit , z1 , pickTime ) ;41 re turn ques t ion2 ;42 end proc ;434445 problemGen5 := proc ( seed )46 $ inc lude " Sp l i n e3 Inc lude . mpl"

43

47 makeTable := proc ( x l i s t , y l i s t )48 l o c a l tableHeader , tableEnd , rowHeader , rowEnd , itemHeader ,49 itemEnd , message ;50 tableHeader := "<tab l e border=\"1\">";51 tableEnd := "</table >";52 rowHeader := "<tr >";53 rowEnd := "</tr >";54 itemHeader := "<td>";55 itemEnd := "</td>";5657 message :=58 cat ( tableHeader ,59 rowHeader , itemHeader , s p r i n t f (" time in %as " , pickTime ) ,

itemEnd ,60 itemHeader , s p r i n t f ("%a" , p icksUni t ) , itemEnd ,61 rowEnd ,62 seq (63 ’ rowHeader , itemHeader , convert ( x l i s t [ i ] , s t r i n g ) , itemEnd ,64 itemHeader ,65 convert ( y l i s t [ i ] , s t r i n g ) , itemEnd , rowEnd ’ , i =1. . nops ( x l i s t ) ) ,66 tableEnd ) ;67 re turn message ;68 end proc ; #end makeTable69 re turn makeTable ( x l i s t , y l i s t ) ;7071 end proc ;7273 answerGen := proc ( seed )74 $ inc lude " Sp l i n e3 Inc lude . mpl"75 curve := CurveFitt ing [ Sp l in e ] ( x l i s t , y l i s t , v , degree=k) ;76 #Pick a random eva lua t i on po int77 pickAPoint := RandomTools [ Generate ] ( f l o a t ( range = z1/2 . . z1−0.1e−1,

d i g i t s = 3) ) ;78 #Calcu la te the area79 valueAtPoint := eva l ( curve , v=pickAPoint ) ;80 area := in t ( curve , v=0. . z1 ) ;81 areaInUnit s := convert ( area , uni t s , p i cksUni t ∗pickTime , pickdUnit ) ;82 re turn pickAPoint , valueAtPoint , area , areaInUnit s ;83 end proc ;8485 end module ;

D.3.2 Spline3Include code

l o c a l n , numIntervals , x l i s t , y l i s t , curve , pickAPoint , z1 , h , k ,valueAtPoint , framework , questionType , pickTime , picksUnit ,pickdUnit , area , quest ion , areaInUnits , quest ion2 , degreeText ,sp l ineDesc , makeTable ;

framework [ 1 , 1 ] := "You are measuring cur rent f low , measured in %as f o r a per iod measured in %as .You wish to es t imate the t o t a l charge c a r r i e d by the f low over the e n t i r e per iod , expres sed in un i t s o f %as . To es t imate the charge , you f i t a curve C to the cur rent f low measurements .C should be an expr e s s i on in the va r i ab l e t ( f o r time ) . I n t e g r a t e C with r e sp e c t to t over the per iod o f time to get the es t imate o f charge . To get the curve , use the CurveFitt ing [ Sp l i ne ] operat i on o f Maple to f i t a %s s p l i n e on the f o l l ow i n g data po in t s : " ;

framework [ 1 , 2 ] := "What i s the t o t a l charge , measured in %as , f o r the per iod t=0 to t=%f %as ?" ;framework [ 1 , 3 ] := [ ampere , b iot , abampere ] ;

44

framework [ 1 , 4 ] := [ second , minute ] ;framework [ 1 , 5 ] := [ coulomb , ampere∗hour , b i o t ∗hour ] ;framework [ 2 , 1 ] := " You are measuring power generated by a device , measured in %a f o r a per iod measured in %as .

You wish to es t imate the t o t a l energy generated over the e n t i r e per iod , expres sed in un i t s o f %as . To es t imate the energy , you f i t a curve C to the power measurements .C should be an expr e s s i on in the va r i ab l e t ( f o r time ) . I n t e g r a t e C with r e sp e c t to t over the per iod o f time to get the es t imate o f energy .To get the curve , use the CurveFitt ing [ Sp l i ne ] operat i on o f Maple , you curve f i t a %s s p l i n e on the f o l l ow i ng data po in t s : " ;

framework [ 2 , 2 ] := "What i s the t o t a l energy , measured in %as , f o r the per iod t=0 to t=%f %as ?" ;framework [ 2 , 3 ] := [ watts , horsepower , k i l owa t t s ] ;framework [ 2 , 4 ] := [ second , minute ] ;framework [ 2 , 5 ] := [ c a l o r i e , k i l o c a l o r i e , BTU, erg , j o u l e ] ;framework [ 3 , 1 ] := " You are measuring the amount o f f low o f a l i q u i d chemical , measured in %a f o r a per iod measured in %as .

You wish to es t imate the t o t a l mass c a r r i e d by the f low over the e n t i r e per iod , expres sed in un i t s o f %as . To es t imate the mass , you f i t a curve C to the f low measurements .C should be an expr e s s i on in the va r i ab l e t ( f o r time ) . I n t e g r a t e C with r e sp e c t to t over the per iod o f time to get the es t imate o f mass.To get the curve , use the CurveFitt ing [ Sp l i ne ] operat i on o f Maple , you curve f i t a %s s p l i n e on the f o l l ow i ng data po in t s : " ;

framework [ 3 , 2 ] := "What i s the t o t a l mass , measured in %as , f o r the per iod t=0 to t=%f %as ?" ;framework [ 3 , 3 ] := [ tons /hour , k i lograms /hour ] ;framework [ 3 , 4 ] := [ second , minute ] ;framework [ 3 , 5 ] := [ pound , gram , ounce ] ;framework [ 4 , 1 ] := "You have a \" black box\" o f an e l e c t r i c v e h i c l e which i s g i v ing you data o f the veh i c l e ’ s v e l o c i t y , measured in %a , f o r a per iod o f time measured in %as .

You wish to es t imate the t o t a l d i s t ance t r a v e l l e d in the per iod , as measured in %a . To do the est imat ion , us ing the CurveFitt ing [ Sp l ine ] operat i on o f Maple , you curve f i t a %s s p l i n e on the f o l l ow i n g data po in t s : " ;framework [ 4 , 2 ] := "What i s the t o t a l d i s tance , measured in %a , f o r the per iod t=0 to t=%f %as ?" ;framework [ 4 , 3 ] := [ mi l e s /hour , k i l ome t e r s /hour ] ;framework [ 4 , 4 ] := [ second , minute ] ;framework [ 4 , 5 ] := [ inches , f e e t , meters ] ;degreeText := [ " l i n e a r " , " quadrat i c " , " cubic " ] ;randomize ( seed ) ;questionType := rand ( 1 . . 4 ) ( ) ; #Pick a ques t i on va r i e t yn := rand ( 1 . . 8 ) ( ) ; #s e l e c t order o f Be s s e l f unc t i onk := rand ( 1 . . 3 ) ( ) ; #s e l e c t i o n o f s p l i n e va r i e t ysp l ineDesc := degreeText [ k ] ;pickTime := framework [ questionType , 4 ] [ rand ( 1 . . nops ( framework [ questionType , 4 ] ) ) ( ) ] ;p i cksUni t := framework [ questionType , 3 ] [ rand ( 1 . . nops ( framework [ questionType , 3 ] ) ) ( ) ] ;p ickdUnit := framework [ questionType , 5 ] [ rand ( 1 . . nops ( framework [ questionType , 5 ] ) ) ( ) ] ;

# Pick number o f data i n t e r v a l snumIntervals := rand ( 5 . . 9 ) ( ) ;

#Pick the end po int ( truncated to 2 decimal p l a c e s ) .z1 := e v a l f ( Bes se lJZeros (n , 1 ) ) ;z1 := e v a l f ( trunc ( z1 ∗100 )/100 . 0 ) ;h := 1/ numIntervals ∗ z1 ;x l i s t := [ seq ( ’ RandomTools [ Generate ] ( f l o a t ( range = h∗ i . . h∗( i +1)−0.1e−1, d i g i t s = 3 ) ) ’ , i = 1 . . ( numIntervals −1 ) ) ] ;y l i s t := map( ( x ) −>Besse lJ (n , x ) , x l i s t ) ;

#Tack on an end po int that ’ s 0 .x l i s t := [ op ( x l i s t ) , z1 ] ;y l i s t := [ op ( y l i s t ) , 0 . 0 ] ;

E Bash script for library and zip file creation and testingThe script makelib2.sh, listed in Figure 24 is a bash script that takes as a single argument the name of amodule, M. The programming follows the convention that the source code for the module is in M.mpl, andthat there is are Maple commands for a test of the module in the file MTest.mpl. It also compresses a copyof the library code, the source (including include files of the form XX Include.mpl), the test script, and anygif files of the form MXX.gif (such as are generated in Figure 18 ) in the directory into a zip file M.zip.

The script, when executed, feeds M.mpl through the Maple processor, along with script-generated prefixand suffix code that will save M ’s definition in the library files M.lib, and M.ind. Currently, the results of

45

Figure 24: makelib2.sh

#Remove a l l e x i s t i n g . l i b and . ind f i l e s in t h i s d i r e c t o r y .rm −f ∗ . l i b ∗ . ind

#Run s c r i p t to c r e a t e . l i b and . ind f i l e s . Assumes module name i s the#same as the name o f the . mpl f i l e with the source .

maple −q <<EOFwith ( LibraryTools ) :Create (" $1 . l i b " ) :$ inc lude "$1 . mpl"Save ( ’ $1 ’ , "$1 . l i b " ) ;EOF

#Remove prev ious z ip f i l e o f t h i s name in parent d i r e c t o r yrm −f . . / $1 . z ip

#Create z ip f i l e conta in ing . l i b , . ind , source code , t e s t and p i c t u r e sz ip . . / $1 . z ip $1 . l i b $1 . ind $1 . mpl $1Test . mpl $1Inc lude . mpl $1 ∗ . g i f

#Run t e s t that . l i b and . ind work c o r r e c t l ymaple < $1Test . mpl

our tests just displays the test results on the standard output, although it could be easily modified to domore elaborate automatic testing.

An obvious last step omitted in this script is to automatically upload the zip file to Maple TA in aspecified directory. Tools such as Selenium RC [16] could automate what are now interactive point-and-clickoperations for uploading through Maple TA’s web interface. However it’s been our experience with otherweb-based applications that the web interface is often revised between versions. The Selenium-based scriptcode would have to be rewritten whenever the revisions of the web interface would change the sequence ofbrowser operations and URLs needed to upload the files.

F Performance experiments with .lib files in Maple TAWe created Maple modules that stored a list of integers. Different modules had varied sizes of the list.Figure 25 shows the question. Figure 26 shows the Maple code that generates the modules (named M10,M100, etc.) that contain the list. Separate library files were generated for each module and uploaded intoseparate locations on the Maple TA server. A question that read in the module was written for the differentmodules (see Figures 27 and 28). An assignment accessing a module was written for each of the modulesM10000, M1000000 and M10000000 was written. The time it took Maple TA to generate the assignment wasmeasured three times on an unloaded system (a dual core 2Ghz Linux system with 4Gb memory with ApacheTomcat as the web service), and averaged. The file sizes and average approximate generation times are listedin Table 1. While performance scales linearly with data size, the overall time to process the question is 20seconds for the list with 10 million items, and and about 3 seconds for the list with 1 million items. In ourproduction situation, we may have over 100 simultaneous users for Maple TA. If they all tried to generate alist concurrently, the expected time to generate all the assignments would be several minutes, even with dualprocessors working in parallel. This conflicts with standard expections of a web server response of severalseconds in the worst case.

We have used these results to establish guidelines limiting library files used in our Maple TA productionenvironment to 100Kb or less.

Zip files containing the .lib and .ind files were created and uploaded to the Maple TA server. Questions

46

Figure 25: A simple question that reads in a list from a .lib file on the Maple TA server

Figure 26: Maple code for defining a module with a list of integers

1 #Generate l i s t s o f va r i ous s i z e s23 seed := 47 ;4 #s i z e L i s t := [ 2 , 4 , 1 0 ] ;5 s i z e L i s t := [ 10^3 , 10^4 , 10^5 , 10^6 , 10^7 ] ;6 makeList := proc (n , seed )7 randomize ( seed ) ;8 RandomTools [ Generate ] ( l i s t ( i n t ege r , n ) ) ;9 end ;1011 with ( LibraryTools ) :1213 f o r x in s i z e L i s t do14 moduleName := M| | x ;15 libName := cat ( convert (moduleName , s t r i n g ) , " . l i b ") ;16 Create ( libName ) ;17 a s s i gn (moduleName , module ( )18 export L , getL ;19 L := makeList (x , seed ) ;20 getL := proc ( ) ; r e turn L ; end ;21 end ) ;22 Save (moduleName , libName ) ;2324 end do ;

47

Figure 27: Question accessing a Maple module with a list with a 1,000,000 integers

accessing the list were created for lists of size 10,000, 1,000,000, and 10,000,000.

48

Figure 28: Presentation and grading of list question

49

Documents

Developing questions for Maple TA using Maple library modules and non-mathematical ... · 2011. 11. 4. · Developing questions for Maple TA using Maple library modules and non-mathematical