Center for Excellence in Learning and Technology

Student Evaluation of Teaching: Creating Reports with Class

ClimateLaura Bestler, Instructional Support Specialist

294-5357 • course-evaluation@iastate.edueLearning:

http://www.celt.iastate.edu/elearning/Class Climate:

https://classclimate.its.iastate.edu/

Objectives

• Introductions• Demonstrating different

reports• Preparing reports• Creating reports• Sending results

Establishing Indicators• Effects Every

Report in Class Climate

• Indicators May be Added/Changed with Running Surveys

• Allows for 2 – 11 Options With or without Abstention

Establishing Indicators• Question Group(s) = Question Categories

• Indicators are an Attribute of the Group

• Indicators May be Added while Questionnaire in use

• Indicators Calculate the Mean Value of all Scaled Questions within the Group (Subtotal)

• Any Group Containing a Scaled Question Should have the Indicator Defined

• All Questions to be Considered in the Sub-Total Calculation MUST Match the Number of Options Set in the Indicator Value for that Group

Questions with a Different Scale (number of options) will be individually calculated but excluded from the Sub-Total Calculation.

Indicators in Questionnaires

Weighted Computation of Indicators

• Total Number of Answers for Each Question used for Computation of the Indicator Value

• Example: Question Group with Two (2) Questions, each with a 5-Point Scale, 4 respondents• Q1=4 Respondents / 5, 5, 4, 3 (Question Average =

4.25)• Q2=3 Respondents / 5, 5, 5 (Question Average =

5.0)• Indicator = 5 + 5 + 4 + 3 + 5 + 5 + 5 = 32 Divided

by # of Responses = 32 / 7 Average of Question Group = 4.57

• Not 4.25 + 5 = 9.25 / 2 = 4.62• Question Group Averages are Averaged into Global

Indicator

Calculation of the Average vs. Median• Total Number of Answers for Each Question used

for Computation of the Median• All question responses are sorted• Example of a Question with 5 respondents (odd

#)• Q1=5 Respondents / 5, 5, 4, 3, 1• Average of Question = 3.6

(5+5+4+3+1=18/5=3.6)• The Median is 4 (the absolute middle number)

• Example of a Question with 4 respondents (even #)• Q2=4 Respondents / 5, 4, 2, 2• Average of Question = 3.75 (5+4+2+2=13/4=

3.75)• The Median is 3 (4+2=6/2=3)

Calculation of the Average vs. Median• Always specify when a Group contains scaled

questions• Question Scales within a Group should have

Identical Options• 2 – 11 Options• With or Without Abstention• Mirrored Scales Required if

Positive on Left• Should be set at Question

Level if ‘Traffic Lights‘ will be used

Report Creator• Change your role to [Report Creator]• Click on [Folders] Think: File Cabinet• Click on [Create new Folder]

• Name folder [Example: F2011]

Report Creator: Single Category

• Include / Exclude questions as desired• Produce Reports by Single Category

• First 3 options in pull-down• Only ONE Choice per Selection

• SubUnit: Pick: 1 Subunit / 1 Period• Program of Study (if Activated and

populated): Pick: 1 POS / 1 Period• Instructor Profile – Pick: 1 Instructor / 1

Period

Report Creator: Any Compilation

Any Compilation (Combination) Last 2 options in pull-down

• Most Flexible• Combine Any

Surveys (except General Surveys)

• Choose multiple criteria from a standard choice box

• Allows Multiple Subunits / Periods / Types / Questionnaires

Report Creator: Profile Line• Create profile line emails (may be printed or saved)

• Compares scaled questions across selected surveys• 2-3 optimal (more becomes difficult to read)

Compare Instructor to their DEPT Peers

Up to 8 (eight)courses taught bysingle Instructor

Up to 8 (eight)Instructors whotaught same/similarCourse(s)

Line 1 Entire Faculty Course 1 Instructor A

Line 2 DEPT Averages Course 2 Instructor B

Line 3 Instructor/Course Averages

Course 3 Instructor C

Line 4 Total Average ofCourses 1,2,3

Total Average ofInstructor A,B,C

Report Creator: Profile Line

Mapping Questions Across Surveys• Used to Combine Surveys with Similar Questions

• Automatic• Use When:

• Questionnaires are Identical• Manual

• Use When:• Questionnaires are Somewhat Similar• Worded Differently, and Re-Sequenced

• You decide which questions are similar enough to average together

• Ability to define the number of character differences

Mapping Questions Across Surveys• Include Answers to Open Questions

• Not Extremely Useful when Combining Different Instructors / Courses

• Weighted Averages• Weighted is Default• Number of Responses will be used to Calculate the

Averages• Un-Weighted Averages

• Number of Surveys will be used to Calculate the Averages

• Not Eligible for Export to CSV nor SPSS• Raw Score Data is not Provided, only the Average• Not Widely Used

Weighted vs. Unweighted• By default, the reports are summarized

weighted, meaning that the calculation is effected according to the number of returns.

• Example: If you summarize a survey with 100 return questionnaires and one with 10, the survey with 100 questionnaires counts for more than the one with 10.

• Mathematically expressed: All individual results are summed for each question (each single value that was selected) and divided by the total number of returns (in this example: 110).

Weighted vs. Unweighted• You must explicitly select unweighted compilation

when creating a report. Unweighted calculation precludes the possibility to adequately undertake further processing in statistics programs.

• With unweighted compilations, each survey counts as 1, regardless of how many questionnaires have been processed for a given survey.

• Mathematically expressed: You add the respective total average of each question and divide by the number of the surveys that you are summarizing (in the example above: by 2).

• Individual response data is lost.

Evaluate Subgroup Data• Used with SINGLE-CHOICE questions

Example: If your survey asked: “Reason for taking this course” and the responses included: “Required” and “Elective”, you could produce a report which included ONLY the responses from those who attended as a required part of their major.

Evaluate Subgroup DataOpens a 2nd screen to allow multiple criteria

Name report as desired

Evaluate Subgroup: Multiple Criteria

Name report as desired

Merging Survey Data• Combine Reports Created by Evaluating a Subgroup

Example: Combine any reports, including those reports created using: Evaluate Subgroup

Single Profile Line Comparison• Combine any 2 Subgroup Reports

Example: Compare the responses from the 2 reports: “Required” and “Elective”, you could see which group evaluated the course higher / lower.

Report for Comparison on Questionnaire

Report for Comparison

Report for Comparison on Questionnaire

Report for Comparison on Questionnaire

The “Hook” Profile line into standard Report

Administrator Reports• Change your role to

[Administrator]• Note: You are unable to do

anything with surveys/questionnaires as a Report Creator

“Printing” Results• Click on [Batch Events] (Left Menu)

Creates PDFs to save on your local CPU or network

For surveys that have already been evaluated, the evaluations of a selection of courses can be requested. There are two possibilities:• The PDF reports of the selected

courses are created together in a common PDF file. This is advantageous for example, when the PDF reports for multiple courses are to be printed at the same time.

• The PDF reports requested are summarized as single PDF files in a ZIP file. This is advantageous for example, when the single PDF files are to be passed on in electronic format, deposited in a public folder or, when one wants to print them individually.

“Sending” Results

• Click on [Batch Events]

Send PDF ReportsYou can determine that an instructor receives all evaluations of his courses accumulated in one email. To do this, please activate the option “Reports of all courses of an instructor collected in one email”. Should this not be the case, the instructor will receive a separate email for each course. Please observe the defined maximum size for email attachments (“System Settings/Class Climate Settings/Network Settings/ Maximum document size”). In “System Settings/Text Templates” the relevant email text may be edited (“EMAIL: Results of a survey” and “E-MAIL: Results of a survey (Summary)”).

IMPORTANT: Discuss with your Department Chair how the reports should be sent to

Instructors

IMPORTANT: Discuss with your Department Chair how the reports should be sent to

Instructors

“Sending” Results

• Click on [Batch Events]

Sending PDF Reports as Instructor Profiles• You can send your instructor profile for surveys that

have already been evaluated as a batch email. By activating the option “Include courses with secondary instructors in instructor profiles”, courses with additional report recipients are also included.

• You can determine whether or not the answers to open questions are to be included in your instructor profile. To do this, please activate the option “Include answers to open questions”.

• If the respective instructor, in addition to his profile, is to receive a detailed report for each of his courses, simply activate the option “Also send single PDF reports”.

• Furthermore, you can determine whether profiles and PDF reports are sent solely to the respective instructor (“Send to instructors”) or only to a specified email address (“Send all to the following email address”).

CELT Training Adapted from:

• Oliver, M. (2011, October 25). Advanced topics in Class Climate: The Report Creator(s) Version 4 & 5 [Scantron webinar].

• Scantron Corporation. (2011). Class Climate Course Evaluation System User Guide v5.0. Eagan, MN: Scantron Corporation. http://classclimate.its.iastate.edu