Evaluation

Adapted from J. Scott Armstrong

Updated June 2014

Evaluation-R18

Learning Diary

The lectures follow an experiential learning experience. To make this work properly:1.Obtain a learning diary (paper). A 10 x 13 bound diary is suggested.2.Keep it up to date.3.Take the learning diary with you to all class sessions.4.For self-learners, use the diary to track your learning progress for all of your learning activities.

2Adprin.com

Objectives of this session

To understand and apply these principles and techniques (not to convince you). Ask for clarification as needed.

Set a goal for yourself on how many principles and techniques you plan to use by the end of this session. Even a goal of one will help you. Put this in your learning diary now.

Note: We will discuss only some of the slides. When you go through the lecture on your own, view it in “Slide Show” and follow the experiential procedures.

Adprin.com

Procedure

Focus on understanding.

Record questions in your learning diary that will help you to apply the techniques or principles, then, after you decide which ones you want to apply, try to answer these from the readings. If not clear, ask others for help.

4Adprin.com

Evaluation Topics

• How can you select the best ad?

• How can you improve an ad?

5Adprin.com

A Horse Named Sam “A man bought a horse named Sam for $60 and sold him for

$70. Then he bought him back again for $80. Finally, he sold him for $90. How much money did he make in the horse business?”

Write your answer on a small piece of paper.

Usually half the people get this correct. How can you improve upon this?

After individual analysis, form groups, say 3-5 people, and discuss your answers with a focus on understanding different viewpoints. Then seek consensus. This nearly always leads to the correct answer to this problem.Ref: Maier & Solem (1952) Click for answer.

6Adprin.com

Parallel processing for analysis

Groups are not good at analysis.

Use parallel processing – two or more people do the same analysis–

(e.g., structuring a problem, or doing math-- either prior to a meeting or in a “time-out.”)

The answer is $20. Group discussion typically leads to correct solutions for this problem. Write another way to solve this problem. Then click for another approach.

7Adprin.com

Problem storming

Think of another way to write the problem. Then click here for a new statement.

Try this: “A man bought a horse for $60 and sold it for $70. Then he bought a pig for $80 and sold it for $90. How much profit did he make in the animal trading business?”

8Adprin.com

Devil’s Advocate (DA) approach

Group arbitrarily selects a member to find things that are wrong with a proposal.

The DA meets with the proposal team to describe what is wrong.

9Adprin.com

What do you see as the possible defects in the Devil’s Advocate?

Write your ideas in your learning diary.

Then click this slide for the evidence-based findings.

10Adprin.com

Evidence on the value of DA

Devil’s Advocate not effective (review by Nemeth & Nemeth-Brown 2003).

Why? Presenters believe: There is no hope of convincing someone who has been instructed to be negative (the DA). Thus, they reinforce their current solution instead of using the criticisms for change.

11Adprin.com

Alternatives to the Devil’s Advocate

Authentic dissent - people argue for what they believe.

The group realizes that an authentic dissenter might change, whereas there is no hope for this with the DA.

The role does not protect the people playing as Devil’s Advocates – they became unpopular.

Authentic dissenters are also unpopular, but people respect them for their courage.

12Adprin.com

Authentic dissent

Those with authentic beliefs argue their positions better than do those assigned beliefs.

Authentic dissenters were able to get people to focus more on opposing thoughts than supporting ones - thus causing more change. (Experiments by Nemeth et al. 2001)

However, groups are annoyed by dissenters. They are not considered to be team players. Suggest a solution. Use problem storming and write your answers in you learning diary. Then click here.

Ask everyone to act as a dissenter– anonymously.

13Adprin.com

Multiple Anonymous Authentic Dissent (MAAD)

Multiple Anonymous Authentic Dissent (MAAD) involves asking experts (e.g., all project group members) to act as dissenters.

1. Each person independently writes* all perceived defects in a proposal. Everyone sends them to an “assembler” who distributes a written report.

2. Each person assumes that each objection has merit, and writes ways to improve the proposals.

3. The suggested improvements are summarized in writing and appropriate revisions are made in the advertising proposal.

* This procedure cannot be done by simply thinking about the steps.14Adprin.com

Groups often rush to a decision. They then resist changes. How can you avoid this?

Second solution technique

Assume that you cannot use your currently favored solution.– Brainwrite alternatives for five minutes.

– Relax the constraint and use either the first or second solution.

The second solution typically comes in less time than the first, and groups often think it is better.

16Adprin.com

How can you pre-test ads to determine which will be more effective?

Write 3 ways to test ads before they are run. When you complete this, go to the next slide.

Pre-testing ads

17Adprin.com

How can you evaluate the effectiveness of advertisements?

1. Combined expert judgments: Ask experts (5 to 20) to rate the effectiveness of the ads and combine.

2. Adherence to principles: Rate how well the ads use principles for persuasive advertising (Use 5 raters.) See the Persuasion Principles Audit .

3. Researcha. Copy-test the effectiveness of alternative ads. b. Copy-test the effectiveness of components of ads (e.g.,

the illustration, headline, copy, sound track)

Adprin.com

Adherence to Persuasion Principles

Persuasion principles can be used not only for creating, but also for evaluating ads. They can identify problems and suggest how to fix them.

The principles are formulated as checklists. Checklists can improve decision making in complex situations. Such as a reduction of 47% in deaths due to surgery in hospitals (Haynes et al. 2006)

Adprin.com

How could you use consumers to evaluate ads?

Which of the following approaches are useful?

1. Focus groups to discuss their reactions to an ad.2. Surveys to ask people how much they like an ad.3. Surveys that ask people about their intentions to purchase when presented with different ads for the same product.

Click for the evidence-based answer

20Adprin.com

Evidence-based recommendations1. Avoid asking consumers if an ad is effective. They are not

experts in persuasion.In particular, avoid focus groups, an expensive and inappropriate method for evaluation. Focus groups tend to result in bias based on the

• behavior of the group leader• wording of the questions and suggestions• reluctance to put forth unusual ideas

– (Armstrong 2004)

2. Do not use “liking” as a criterion, whether by consumers or experts.1. Liking is not an objective (Persuasive Advertising p 289)2. Focus on persuasiveness

3. Use experiments. Assess how people are affected when presented with different ads (some get ad A while others get ad B)

Adprin.com

Why 195 principles? Why not the top ten?

Advertising is complex-- like medicine. Would you like to be restricted to the top ten medicines?

We use the “Index method” to determine how closely the ads adhere to principles (Armstrong & Graefe 2011).

Regression analysis does not work in such a situation as there are too many variables.

22Adprin.com

Develop action steps for applications

Write a plan in your learning diary for applying at least one technique from this session.

Be explicit about what you will do.

Set a time deadline for completing the application.

Start your applications report for this course. Describe your plan and, later, how it turned out.

23Adprin.com

Techniques Used in this Lecture

• Problem storming• Devil’s advocate approach• Authentic dissent and MAAD• Index method• Second solution

24Adprin.com

Advice on learning

One study found that fewer than 10% of students were successful in applying new knowledge.•This went to 20% if they actively applied what they were taught during a class session.•It went to 90% when they worked with a learning partner and coached each other.

25Adprin.com

Follow-up: Complete prior to next session

1. ___ Go through this lecture on your own (It is on the Educational Materials page)

2.___Study Persuasive Advertising pages 287-292 and record your reading time in your learning diary. Highlight techniques that you want to apply in yellow.

3.___ Complete the End of Chapter Questions for “Evaluating Advertisements” and check your answers against PA.

Adprin.com