Reflections on Teaching

Course Improvement

• We can simply use the feedback to pat ourselves on the back, documenting that we are doing a great job

• Or we can use it to capitalize on strengths and improve weak areas

• Either way, we can include some information

Sources of Information

• Gut feelings

• Student performance on the content, in the class or on external tests

• Informal feedback from students

• Peer visits and feedback

• Committee visits and feedback

• IDEA evaluations

Easy to interpret

• Recommendations from colleagues, supervisors and students

• Strong student performance

• High evaluations in all categories

• The latter two don’t really suggest any changes

• The first often comes easy to implement

Harder to Interpret

• Poor student performance

• Gut feelings

• IDEA forms

• The IDEA forms have a lot of information, some of which can be useful

What is reliable anyway?

• This is simply a function of the number of responses; lower numbers are considered less reliable. 25 or more responses are noted as “reliable”, 15?-24 are noted as “fairly reliable”, 10-15 are noted as “marginally reliable”, and 9 and below are noted as “unreliable.”

And what is representative?

• This is the fraction of the class that filled out the form; this is an attempt to ascertain if the response is representative of the class. Response rates of 65%-79% are noted as “probably representative”; response rates of 80% and higher are noted as “representative.”

IDEA Raw vs. Adjusted

• The Raw score is the average of the responses by the students.

• Why adjust this?

• There are factors that affect scores that are independent of the instructor or the class.

• An example of this is the students’ desire to take the course, regardless of teacher

Factors to adjust for

• Class size (larger classes get lower ratings)• Type of course (core requirements get lower

ratings)• Student interest (interested students rate

courses higher, uninterested students rate them lower)

• These are called “extraneous variables” and are discussed in detail in IDEA Research Report #6

How are adjustments made?

• The raw scores are adjusted, based on students’ answers to the questions pertaining to interest and effort and based on class size and level. In short, if most of the students report a high desire to take the course, the raw score will be adjusted down a bit. For a large class, the ratings adjust up a bit.

Why adjust scores at all?

• Once the scores are adjusted, it is possible to make comparisons between classes, without having to worry about class size, or the rest.

• You can compare your own ratings between classes, or from one semester to another.

• We can compare the ratings (which are a reflection of student perceptions, of course) between faculty.

Key General Indicators

• Progress on Course Objectives• Excellent Teacher• Excellent Course

• These are always worth looking at as summary statistics. They do not provide much detail.

• Good for identifying serious issues, or for patting ourselves on the back.

Progress on Course Objectives

• This is a summary evaluation, based on a number of questions in the form.

• It does reflect student perception on their progress.

• The student responses that are used for this indicator are those objectives (items 21-32) that the professor picked as “important” or “essential” (counting double)

Excellent Teacher

• This is a summary evaluation, based on Question #41 on the IDEA form.

• It does reflect student perception of their teacher.

Excellent Course

• This is a summary evaluation, based on a number of questions in the form.

• It does reflect student perception on their progress.

• The question that is used for this indicator simply asks the students if it was an excellent course, question #42

Clear Indicators

• Good ratings in general, 4.0 or above in the three summary numbers would mean that the students are well satisfied with the course.

• Poor ratings in general, 2.5 or below in the three summary numbers would mean that the students are very unhappy with the course.

Mixed Results

• We saw this in FYEx 103. Several faculty received very high ratings as teachers, but the course ratings were very low. This was echoed by the written comments.

• This reflects student satisfaction with the professor, but not with the class. The rest of the evaluation can help us discern what about the class was a problem for them.

Information to modify a class

• The IDEA reports cannot substitute for judgment on our part.

• They can give us information on what students liked or didn’t like about a class.

Why do we care?

• Okay, students can’t tell if we are properly qualified.

• They can tell if we seem competent.

• They do make decisions based on how they respond to us.

What is this “converted” score?

• The converted scores are simply normalized to an average of 50, with 63 being the upper 10%.

• This allows comparison between questions, as well as between classes, semesters, faculty, etc.

What information can be useful?

• Other than identifying student dissatisfaction, some questions can provide specific guidance.

• Specifically, the last page, statistical detail can provide specific guidance

How Can We Use IDEA Results to Improve

Instruction? https://theideaonline.org/idea/cs/index.jsp

Teacher

Old Master

The “Dr. Fox” Lecture

In a well-known 1970 study,

a professional actor (Michael Fox)

was hired to deliver a non-substantive

and contradictory lecture, but in an

enthusiastic and authoritative style.

Naftulin, Donald H., John E. Ware, and Frank A. Donnelly, "The Doctor Fox Lecture: A Paradigm of Educational Seduction," Journal of Medical Education 48 (1973): 630-5. Rice, Lee. "Student Evaluation of Teaching: Problems and Prospects," Teaching Philosophy 11 (1988): 329-44.

X

The “Dr. Fox” Lecturehttp://ecclesiastes911.net/doctor_fox.html

• The lecture was held at the University of Southern California School of Medicine.

• The subject was "Mathematical Game Theory as Applied to Physician Education."

• The speaker was introduced as Dr. Myron L. Fox from Albert Einstein College of Medicine, a pupil of von Neumann and an authority on the application of mathematics to human behavior. The attendees were psychiatrists and psychologists (MDs and PhDs) who were gathered for a training conference. They listened to the lecturer with great interest, asked many questions, and were satisfied with speaker's replies.

Let’s Watch A YouTube Excerpt of Dr. Fox’s Lecture

• http://www.youtube.com/watch?v=RcxW6nrWwtc

Dr. Fox’s Lecture

• Dr. Fox gave his talk on a Scientific American

article on “Game Theory. “ He borrowed from the article some

phrases, but not any sense. He mixed them up with allusions to unrelated subjects, which on purely verbal level may appear relevant. He conducted himself with great confidence and showed such a mastery of the aforementioned allusions that the audience was convinced that a luminary is standing before them.

• 100% had answered in the questionnaire that Dr. Fox had stimulated their thinking, 90% said that he presented material

in a well organized form, and 90% said that Dr. Fox put his material across in an interesting and factual manner.

• Some audience members even commented that they• had read Dr. Fox’s published papers!

Effects of the “Dr. Fox” Study

Can a brilliant delivery technique of a talk so completely “bamboozle” a group of experts that they overlook the fact that the content

was nonsense?

The published study in 1973 cast serious doubts on results obtained from evaluation of teachers and lectures.

More modern studies have affirmed results from the “Dr. Fox” study.

What does “Ms. Mentor” in The Chronicle Say?

The Torment of Teaching Evaluations •By Ms. Mentor•Question: I work twice as hard at teaching as anyone I know, but I still can't get good student evaluations. I've tried outlines and keywords on the board, handouts, individual meetings, midterm evaluations, peer observations, lecturing more, lecturing less. Some student comments are so harsh and demoralizing that I put off reading my evaluations until school breaks, when I have time to be depressed. Is it possible that my low evaluations stem from personality issues that I can't do anything about? Am I alone?•Answer: Certainly you are not solo. Ms. Mentor's mailbag is full of complaints about student evaluations -- none of them from students. Faculty members feel that they are cheated, mistreated, and misunderstood. Often they're right, for most evaluation forms are so vague and perfunctory ("Concerned about students -- rate 1 to 9") that they do nothing to improve teaching. More often, they become weapons to get rid of untenured profs who have made enemies. No one has ever really agreed on what makes a "good teacher," and Ms. Mentor still grieves for Socrates, who taught his students to question everything, and wound up dead instead of tenured.•But your risk is smaller. You just need good evaluations, and Ms. Mentor can tell you how. (Naive and idealistic souls may wish to stop reading at this point.)•Simplest of all, you can give higher grades, which do correlate with student ratings. You can use more hand gestures, modulate your voice more, and walk while you talk. Students give higher evaluations to teachers who are good-looking or very dramatic. This is called "the Dr. Fox effect," named for a hired actor who purported to be "Dr. Fox" and who gave a nonsensical university lecture in a wildly entertaining style, and got outstanding student evaluations for his brilliance.•In one notorious study, those who saw just a 30-second soundless video of a teacher in action gave him virtually the same ratings as the students taking his course -- who'd spent a semester reading, writing, thinking, and talking with him. Smile warmly for the first 30 seconds of the first class in January, and you'll get good scores in May.

Give Higher Grades, Use More Hand Gestures… What to Do?

There are some more effective ways to improve student learning in the classroom.

Mine Your IDEA Results and Other Data

Using IDEA Results to Improve Instruction & Learning: EXAMPLES

1. Use student PERCEPTIONS of general and specific SLO progress.

2. Compare student PERCEPTIONS with

some measure of REALITY.

3. Show PROGRESS in various areas over time in the same course from semester-to-semester.

4. Sift through student COMMENTS to find relevant and useful information.

Student Perceptions of Achieving Progress on SLOs

• General (Progress on Course Objectives)• This is a summary evaluation, based on a number of questions in the form.• It does reflect student perception on their progress.• The student responses that are used for this indicator are those objectives

(items 21-32) that the professor picked as “important” or “essential” (counting double)

• Specific (Supplemental Questions provided by the Instructor)

Indirect Measures-SLOs

• You can use information from your IDEA results (supplemental questions) to examine student perceptions of achieving SLOs in your courses.

  No. students

SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7    

SLO#8 SLO#9 SLO#10

Murphy MWF 8

25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9

Murphy TR 8

21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9

Mean (Murphy)

23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9

INDIRECT MEASURES OF STUDENT LEARNING OUTCOMESChemistry 105: General Chemistry IFall 2010

Even More…

• Once you have a good idea of student perceptions on whether or not they achieved the SLOs, compare their perceptions with REALITY (direct measures).

• Pick questions which directly measure student achievement on a single SLO on a test or examination at the end of the course. Mine the data.

Compare Student Perception with Reality: GEN CHEM I

• SLO #6 “After taking this course, I feel I am able to balance chemical equations.”

  No. students

SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7    

SLO#8 SLO#9 SLO#10

Murphy MWF 8

25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9

Murphy TR 8

21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9

Mean (Murphy)

23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9

From the ACS Gen. Chem. I Examination (questions #15 and #24) given right after the IDEA Evaluations, 23/25 and 20/21 of the students could correctly balance a chemical equation.CHEM 105 Student Perception Direct MeasureMWF 8 4.5/5.0 23/25 (93%)TR 8 4.8/5.0 20/21 (95%)

Compare Student Perception with Reality: GEN. CHEM I

  No. students

SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7    

SLO#8 SLO#9 SLO#10

Murphy MWF 8

25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9

Murphy TR 8

21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9

Mean (Murphy)

23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9

From the ACS Gen. Chem. I Examination (questions #10 and #16) given right after the IDEA Evaluations, 13/25 (52%) and 11/21 (53%) of the students could correctly do this. Student Perception Direct MeasureMWF 8 3.9/5.0 13/25 (52%)TR 8 3.9/5.0 11/21 (53%)

• SLO #10 “After taking this course, I feel I am able to calculate the standard enthalpy of reaction for a given series of reactions by using tabular data.”

SHOW TEMPORAL PROGRESS

CHEMISTRY 105: General Chemistry I Fall Classes

IDEA Results: Must be reliable and representative

Progress on Objectives Excellent Teacher

What If?

Think about the above results:1.Same course objectives selected each year?2.Changes in class size and/or population 3.Change in teaching techniques4.Other5.Supplement with additional measures

How Can Student Comments Be Used

Effectively?

Sample Student Comments• Huntingdon College• IDEA Survey Student Comments• Spring 2011• Murphy, Maureen – CHEM 106 – MWF 8:00

• · Great instructor!• · Great instructor. Made concepts easy to understand.• · Dr. Murphy was an outstanding instructor.• · Left-handed teachers-step away from board so students on the right side of the classroom can see more!• · Great class!!!

• Murphy, Maureen – CHEM 106 – TR 8:00• · The course and approach is fair and straightforward. I believe all courses and• institutions should have a Dr. Murphy.

• -Quit being so positive-heck, it is 8 am in the morning!• · Great teacher!! Little hard but wiling to help!!

• · E for the cell= cathode – anode• · Dr. Murphy has been my favorite teacher in my 2 years at Huntingdon and done• so teaching one of my least favorite subjects! She’s the best!

• · Dr. Murphy puts a lot of her time in to help her students better understand• chemistry. She has the most office hours out of all of my professors!

• Murphy, Maureen – CHEM 446 – MW 19:00• · I really appreciate Dr. Murphy’s concern for her students. She is very open to• helping whomever needs help and she wants the students to understand the

• material. Her classes are never boring and I always learn a lot. Overall I very• much enjoyed the class and Dr. Muphy’s teaching and learned a lot.• · You are an amazing teacher and I appreciate all that you do. If every teacher,• faculty, and staff member were like you.• · Dr. Murphy is the best teacher I’ve had in my entire life. She changed my life

• forever. I couldn’t begin to try to name everything she has done for me so I’m not• going to try. I could never repay her or thank her enough. I love you Dr.• Murphy, thank you.• Gets distracted sometimes talking about her husband teaching chemistry also.

Murphy, Maureen – CHEM 409 – MWF 9:15· Great teacher! Dr. Murphy really cares about seeing us do well in her class aswell as our life. We each share our own personal aademic relationship with her.· Dr. Murphy is the greatest asset at Huntingdon College. She is an incredibleteacher and cares infinitely about her students. Dr. Murphy is the only reason Iam where I am today from an academic perspective. She helped me when Ineeded help, and pushed and dragged me through when I needed dragging. I donot agree with everything that Huntingdon does, but having Dr. Murphy aroundand involved is easily the smartest thing Huntingdon has ever done. Thank youDr. Murphy, I will never be able to repay you for all that you have given me.·

COMMENTS IN BLUE MAY BE HELPFUL IN IMPROVING INSTRUCTION

Student Comments that Deserve Attention

• Tests are too hard and do not reflect the level at which the professor teaches.

• Lets class out early every day-great class!

• Too fast. Slides are read to us in class and we see 56-75 slides per 60 minutes.

• Cannot (or refuses to) answer our questions in class.

• Not available or not there during office hours.

SUMMARY• IDEA Student Ratings of Instruction contain

some useful information for faculty.• Understanding what the ratings mean and how

they are constructed is important.• Student perceptions can be useful, but are

limited.• Supplemental questions can be effectively used

on IDEA forms, especially when paired with direct assessment of progress on SLOs.

• Evaluation of ratings over time can be useful.• “Sifted” comments from students can inform

future instruction.

YOUR QUESTIONS

??????????

Additional Resources

• http://chronicle.com/article/A-Tech-Happy-Professor-Reboots/130741/